May 25 12:12:58.214444 ip-10-0-143-107 systemd[1]: Starting Kubernetes Kubelet...
May 25 12:12:58.741618 ip-10-0-143-107 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 12:12:58.741618 ip-10-0-143-107 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 25 12:12:58.741618 ip-10-0-143-107 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 12:12:58.741618 ip-10-0-143-107 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 25 12:12:58.741618 ip-10-0-143-107 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 12:12:58.742488 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.742399    2578 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 25 12:12:58.746391 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746374    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 12:12:58.746391 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746391    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 12:12:58.746458 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746394    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 12:12:58.746458 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746398    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 12:12:58.746458 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746401    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 12:12:58.746458 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746405    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 12:12:58.746458 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746408    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 12:12:58.746458 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746411    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 12:12:58.746458 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746413    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 12:12:58.746458 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746416    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 12:12:58.746458 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746419    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 12:12:58.746458 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746423    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 12:12:58.746458 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746426    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 12:12:58.746458 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746430    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 12:12:58.746458 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746434    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 12:12:58.746458 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746436    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 12:12:58.746458 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746439    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 12:12:58.746458 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746442    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 12:12:58.746458 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746444    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 12:12:58.746458 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746447    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 12:12:58.746458 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746450    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 12:12:58.746458 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746452    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 12:12:58.746934 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746455    2578 feature_gate.go:328] unrecognized feature gate: Example2
May 25 12:12:58.746934 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746457    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 12:12:58.746934 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746460    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 12:12:58.746934 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746463    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 12:12:58.746934 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746466    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 12:12:58.746934 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746469    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 12:12:58.746934 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746472    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 12:12:58.746934 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746475    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 12:12:58.746934 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746477    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 12:12:58.746934 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746480    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 12:12:58.746934 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746482    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 12:12:58.746934 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746485    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 12:12:58.746934 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746488    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 12:12:58.746934 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746492    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 12:12:58.746934 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746495    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 12:12:58.746934 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746499    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 12:12:58.746934 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746502    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 12:12:58.746934 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746505    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 12:12:58.746934 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746507    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 12:12:58.746934 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746510    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 12:12:58.747435 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746512    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 12:12:58.747435 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746515    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 12:12:58.747435 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746517    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 12:12:58.747435 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746520    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 12:12:58.747435 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746522    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 12:12:58.747435 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746525    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 12:12:58.747435 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746527    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 12:12:58.747435 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746531    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 12:12:58.747435 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746535    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 12:12:58.747435 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746538    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 12:12:58.747435 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746541    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 12:12:58.747435 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746544    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 12:12:58.747435 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746547    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 12:12:58.747435 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746551    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 12:12:58.747435 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746554    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 12:12:58.747435 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746557    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 12:12:58.747435 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746560    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 12:12:58.747435 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746563    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 12:12:58.747435 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746567    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 12:12:58.747435 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746570    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 12:12:58.747965 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746573    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 12:12:58.747965 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746575    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 12:12:58.747965 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746578    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 12:12:58.747965 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746580    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 12:12:58.747965 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746584    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 12:12:58.747965 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746586    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 12:12:58.747965 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746589    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 12:12:58.747965 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746592    2578 feature_gate.go:328] unrecognized feature gate: Example
May 25 12:12:58.747965 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746594    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 12:12:58.747965 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746597    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 12:12:58.747965 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746599    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 12:12:58.747965 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746602    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 12:12:58.747965 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746604    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 12:12:58.747965 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746607    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 12:12:58.747965 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746610    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 12:12:58.747965 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746612    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 12:12:58.747965 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746615    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 12:12:58.747965 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746617    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 12:12:58.747965 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746620    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 12:12:58.748438 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746623    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 12:12:58.748438 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746625    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 12:12:58.748438 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746628    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 12:12:58.748438 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746630    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 12:12:58.748438 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.746633    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 12:12:58.748438 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747019    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 12:12:58.748438 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747025    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 12:12:58.748438 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747028    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 12:12:58.748438 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747031    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 12:12:58.748438 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747033    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 12:12:58.748438 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747036    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 12:12:58.748438 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747039    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 12:12:58.748438 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747041    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 12:12:58.748438 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747044    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 12:12:58.748438 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747046    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 12:12:58.748438 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747049    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 12:12:58.748438 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747052    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 12:12:58.748438 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747070    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 12:12:58.748438 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747073    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 12:12:58.748895 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747075    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 12:12:58.748895 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747078    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 12:12:58.748895 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747080    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 12:12:58.748895 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747083    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 12:12:58.748895 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747086    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 12:12:58.748895 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747088    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 12:12:58.748895 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747091    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 12:12:58.748895 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747093    2578 feature_gate.go:328] unrecognized feature gate: Example2
May 25 12:12:58.748895 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747096    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 12:12:58.748895 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747098    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 12:12:58.748895 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747102    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 12:12:58.748895 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747105    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 12:12:58.748895 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747107    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 12:12:58.748895 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747110    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 12:12:58.748895 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747112    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 12:12:58.748895 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747115    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 12:12:58.748895 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747118    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 12:12:58.748895 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747120    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 12:12:58.748895 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747122    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 12:12:58.748895 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747126    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 12:12:58.749397 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747129    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 12:12:58.749397 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747131    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 12:12:58.749397 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747134    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 12:12:58.749397 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747136    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 12:12:58.749397 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747139    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 12:12:58.749397 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747141    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 12:12:58.749397 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747144    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 12:12:58.749397 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747146    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 12:12:58.749397 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747149    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 12:12:58.749397 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747151    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 12:12:58.749397 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747154    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 12:12:58.749397 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747157    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 12:12:58.749397 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747159    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 12:12:58.749397 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747162    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 12:12:58.749397 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747164    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 12:12:58.749397 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747167    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 12:12:58.749397 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747169    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 12:12:58.749397 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747172    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 12:12:58.749397 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747174    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 12:12:58.749397 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747177    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 12:12:58.749902 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747179    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 12:12:58.749902 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747182    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 12:12:58.749902 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747185    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 12:12:58.749902 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747187    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 12:12:58.749902 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747189    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 12:12:58.749902 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747192    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 12:12:58.749902 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747195    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 12:12:58.749902 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747197    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 12:12:58.749902 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747200    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 12:12:58.749902 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747204    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 12:12:58.749902 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747206    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 12:12:58.749902 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747210    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 12:12:58.749902 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747213    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 12:12:58.749902 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747216    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 12:12:58.749902 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747218    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 12:12:58.749902 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747220    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 12:12:58.749902 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747223    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 12:12:58.749902 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747225    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 12:12:58.749902 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747228    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 12:12:58.750381 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747230    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 12:12:58.750381 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747234    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 12:12:58.750381 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747237    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 12:12:58.750381 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747240    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 12:12:58.750381 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747243    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 12:12:58.750381 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747248    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 12:12:58.750381 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747252    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 12:12:58.750381 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747255    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 12:12:58.750381 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747258    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 12:12:58.750381 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747262    2578 feature_gate.go:328] unrecognized feature gate: Example
May 25 12:12:58.750381 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747264    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 12:12:58.750381 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747267    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 12:12:58.750381 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747270    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 12:12:58.750381 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747342    2578 flags.go:64] FLAG: --address="0.0.0.0"
May 25 12:12:58.750381 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747349    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 25 12:12:58.750381 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747357    2578 flags.go:64] FLAG: --anonymous-auth="true"
May 25 12:12:58.750381 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747361    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 25 12:12:58.750381 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747366    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
May 25 12:12:58.750381 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747369    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 25 12:12:58.750381 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747374    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 25 12:12:58.750877 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747378    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 25 12:12:58.750877 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747382    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 25 12:12:58.750877 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747385    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 25 12:12:58.750877 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747388    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 25 12:12:58.750877 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747392    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 25 12:12:58.750877 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747395    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 25 12:12:58.750877 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747398    2578 flags.go:64] FLAG: --cgroup-root=""
May 25 12:12:58.750877 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747402    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
May 25 12:12:58.750877 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747405    2578 flags.go:64] FLAG: --client-ca-file=""
May 25 12:12:58.750877 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747408    2578 flags.go:64] FLAG: --cloud-config=""
May 25 12:12:58.750877 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747411    2578 flags.go:64] FLAG: --cloud-provider="external"
May 25 12:12:58.750877 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747414    2578 flags.go:64] FLAG: --cluster-dns="[]"
May 25 12:12:58.750877 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747418    2578 flags.go:64] FLAG: --cluster-domain=""
May 25 12:12:58.750877 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747421    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 25 12:12:58.750877 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747424    2578 flags.go:64] FLAG: --config-dir=""
May 25 12:12:58.750877 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747427    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 25 12:12:58.750877 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747431    2578 flags.go:64] FLAG: --container-log-max-files="5"
May 25 12:12:58.750877 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747435    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 25 12:12:58.750877 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747439    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 25 12:12:58.750877 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747442    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 25 12:12:58.750877 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747445    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 25 12:12:58.750877 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747449    2578 flags.go:64] FLAG: --contention-profiling="false"
May 25 12:12:58.750877 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747452    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 25 12:12:58.750877 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747455    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 25 12:12:58.751474 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747458    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
May 25 12:12:58.751474 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747461    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 25 12:12:58.751474 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747465    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 25 12:12:58.751474 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747468    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 25 12:12:58.751474 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747472    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 25 12:12:58.751474 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747475    2578 flags.go:64] FLAG: --enable-load-reader="false"
May 25 12:12:58.751474 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747478    2578 flags.go:64] FLAG: --enable-server="true"
May 25 12:12:58.751474 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747481    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 25 12:12:58.751474 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747486    2578 flags.go:64] FLAG: --event-burst="100"
May 25 12:12:58.751474 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747490    2578 flags.go:64] FLAG: --event-qps="50"
May 25 12:12:58.751474 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747493    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 25 12:12:58.751474 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747496    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 25 12:12:58.751474 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747499    2578 flags.go:64] FLAG: --eviction-hard=""
May 25 12:12:58.751474 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747503    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 25 12:12:58.751474 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747506    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 25 12:12:58.751474 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747509    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 25 12:12:58.751474 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747512    2578 flags.go:64] FLAG: --eviction-soft=""
May 25 12:12:58.751474 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747515    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 25 12:12:58.751474 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747518    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 25 12:12:58.751474 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747521    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 25 12:12:58.751474 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747524    2578 flags.go:64] FLAG: --experimental-mounter-path=""
May 25 12:12:58.751474 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747527    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
May 25 12:12:58.751474 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747530    2578 flags.go:64] FLAG: --fail-swap-on="true"
May 25 12:12:58.751474 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747532    2578 flags.go:64] FLAG: --feature-gates=""
May 25 12:12:58.751474 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747536    2578 flags.go:64] FLAG: --file-check-frequency="20s"
May 25 12:12:58.752103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747539    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 25 12:12:58.752103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747542    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 25 12:12:58.752103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747546    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 25 12:12:58.752103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747549    2578 flags.go:64] FLAG: --healthz-port="10248"
May 25 12:12:58.752103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747553    2578 flags.go:64] FLAG: --help="false"
May 25 12:12:58.752103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747556    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-143-107.ec2.internal"
May 25 12:12:58.752103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747563    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
May 25 12:12:58.752103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747566    2578 flags.go:64] FLAG: --http-check-frequency="20s"
May 25 12:12:58.752103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747568    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 25 12:12:58.752103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747572    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 25 12:12:58.752103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747575    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 25 12:12:58.752103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747578    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 25 12:12:58.752103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747582    2578 flags.go:64] FLAG: --image-service-endpoint=""
May 25 12:12:58.752103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747585    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 25 12:12:58.752103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747588    2578 flags.go:64] FLAG: --kube-api-burst="100"
May 25 12:12:58.752103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747591    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 25 12:12:58.752103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747594    2578 flags.go:64] FLAG: --kube-api-qps="50"
May 25 12:12:58.752103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747597    2578 flags.go:64] FLAG: --kube-reserved=""
May 25 12:12:58.752103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747600    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 25 12:12:58.752103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747603    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 25 12:12:58.752103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747606    2578 flags.go:64] FLAG: --kubelet-cgroups=""
May 25 12:12:58.752103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747609    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 25 12:12:58.752103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747612    2578 flags.go:64] FLAG: --lock-file=""
May 25 12:12:58.752103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747615    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 25 12:12:58.752712 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747618    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
May 25 12:12:58.752712 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747621    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 25 12:12:58.752712 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747626    2578 flags.go:64] FLAG: --log-json-split-stream="false"
May 25 12:12:58.752712 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747629    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 25 12:12:58.752712 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747636    2578 flags.go:64] FLAG: --log-text-split-stream="false"
May 25 12:12:58.752712 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747640    2578 flags.go:64] FLAG: --logging-format="text"
May 25 12:12:58.752712 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747642    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 25 12:12:58.752712 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747646    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 25 12:12:58.752712 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747649    2578 flags.go:64] FLAG: --manifest-url=""
May 25 12:12:58.752712 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747652    2578 flags.go:64] FLAG: --manifest-url-header=""
May 25 12:12:58.752712 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747657    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 25 12:12:58.752712 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747664    2578 flags.go:64] FLAG: --max-open-files="1000000"
May 25 12:12:58.752712 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747669    2578 flags.go:64] FLAG: --max-pods="110"
May 25 12:12:58.752712 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747672    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 25 12:12:58.752712 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747676    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 25 12:12:58.752712 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747679    2578 flags.go:64] FLAG: --memory-manager-policy="None"
May 25 12:12:58.752712 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747682    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 25 12:12:58.752712 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747685    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 25 12:12:58.752712 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747688    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 25 12:12:58.752712 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747691    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 25 12:12:58.752712 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747698    2578 flags.go:64] FLAG: --node-status-max-images="50"
May 25 12:12:58.752712 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747701    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 25 12:12:58.752712 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747705    2578 flags.go:64] FLAG: --oom-score-adj="-999"
May 25 12:12:58.752712 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747708    2578 flags.go:64] FLAG: --pod-cidr=""
May 25 12:12:58.753353 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747711    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 25 12:12:58.753353 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747716    2578 flags.go:64] FLAG: --pod-manifest-path=""
May 25 12:12:58.753353 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747720    2578 flags.go:64] FLAG: --pod-max-pids="-1"
May 25 12:12:58.753353 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747723    2578 flags.go:64] FLAG: --pods-per-core="0"
May 25 12:12:58.753353 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747726    2578 flags.go:64] FLAG: --port="10250"
May 25 12:12:58.753353 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747729    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 25 12:12:58.753353 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747732    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f81332a9c50a2603"
May 25 12:12:58.753353 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747735    2578 flags.go:64] FLAG: --qos-reserved=""
May 25 12:12:58.753353 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747738    2578 flags.go:64] FLAG: --read-only-port="10255"
May 25 12:12:58.753353 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747741    2578 flags.go:64] FLAG: --register-node="true"
May 25 12:12:58.753353 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747744    2578 flags.go:64] FLAG: --register-schedulable="true"
May 25 12:12:58.753353 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747747    2578 flags.go:64] FLAG: --register-with-taints=""
May 25 12:12:58.753353 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747752    2578 flags.go:64] FLAG: --registry-burst="10"
May 25 12:12:58.753353 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747755    2578 flags.go:64] FLAG: --registry-qps="5"
May 25 12:12:58.753353 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747757    2578 flags.go:64] FLAG: --reserved-cpus=""
May 25 12:12:58.753353 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747760    2578 flags.go:64] FLAG: --reserved-memory=""
May 25 12:12:58.753353 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747764    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 25 12:12:58.753353 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747767    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 25 12:12:58.753353 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747770    2578 flags.go:64] FLAG: --rotate-certificates="false"
May 25 12:12:58.753353 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747772    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
May 25 12:12:58.753353 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747781    2578 flags.go:64] FLAG: --runonce="false"
May 25 12:12:58.753353 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747785    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 25 12:12:58.753353 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747788    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 25 12:12:58.753353 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747792    2578 flags.go:64] FLAG: --seccomp-default="false"
May 25 12:12:58.753353 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747795    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
May 25 12:12:58.754309 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747797    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 25 12:12:58.754309 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747801    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 25 12:12:58.754309 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747804    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 25 12:12:58.754309 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747807    2578 flags.go:64] FLAG: --storage-driver-password="root"
May 25 12:12:58.754309 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747810    2578 flags.go:64] FLAG: --storage-driver-secure="false"
May 25 12:12:58.754309 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747813    2578 flags.go:64] FLAG: --storage-driver-table="stats"
May 25 12:12:58.754309 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747816    2578 flags.go:64] FLAG: --storage-driver-user="root"
May 25 12:12:58.754309 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747819    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 25 12:12:58.754309 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747822    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
May 25 12:12:58.754309 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747825    2578 flags.go:64] FLAG: --system-cgroups=""
May 25 12:12:58.754309 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747828    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 25 12:12:58.754309 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747834    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
May 25 12:12:58.754309 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747837    2578 flags.go:64] FLAG: --tls-cert-file=""
May 25 12:12:58.754309 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747840    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 25 12:12:58.754309 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747844    2578 flags.go:64] FLAG: --tls-min-version=""
May 25 12:12:58.754309 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747846    2578 flags.go:64] FLAG: --tls-private-key-file=""
May 25 12:12:58.754309 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747849    2578 flags.go:64] FLAG: --topology-manager-policy="none"
May 25 12:12:58.754309 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747852    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
May 25 12:12:58.754309 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747855    2578 flags.go:64] FLAG: --topology-manager-scope="container"
May 25 12:12:58.754309 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747859    2578 flags.go:64] FLAG: --v="2"
May 25 12:12:58.754309 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747863    2578 flags.go:64] FLAG: --version="false"
May 25 12:12:58.754309 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747867    2578 flags.go:64] FLAG: --vmodule=""
May 25 12:12:58.754309 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747872    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 25 12:12:58.754309 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.747875    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 25 12:12:58.754309 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747994    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 12:12:58.754964 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.747998    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 12:12:58.754964 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748000    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 12:12:58.754964 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748003    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 12:12:58.754964 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748007    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 12:12:58.754964 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748009    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 12:12:58.754964 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748013    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 12:12:58.754964 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748016    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 12:12:58.754964 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748018    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 12:12:58.754964 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748021    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 12:12:58.754964 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748023    2578 feature_gate.go:328] unrecognized feature gate: Example2
May 25 12:12:58.754964 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748026    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 12:12:58.754964 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748028    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 12:12:58.754964 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748031    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 12:12:58.754964 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748034    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 12:12:58.754964 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748037    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 12:12:58.754964 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748040    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 12:12:58.754964 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748042    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 12:12:58.754964 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748045    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 12:12:58.754964 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748047    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 12:12:58.754964 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748050    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 12:12:58.755479 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748066    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 12:12:58.755479 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748069    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 12:12:58.755479 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748072    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 12:12:58.755479 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748074    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 12:12:58.755479 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748076    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 12:12:58.755479 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748079    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 12:12:58.755479 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748083    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 12:12:58.755479 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748085    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 12:12:58.755479 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748088    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 12:12:58.755479 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748090    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 12:12:58.755479 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748094    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 12:12:58.755479 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748098    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 12:12:58.755479 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748101    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 12:12:58.755479 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748104    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 12:12:58.755479 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748107    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 12:12:58.755479 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748110    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 12:12:58.755479 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748113    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 12:12:58.755479 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748117    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 12:12:58.755479 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748125    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 12:12:58.755479 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748128    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 12:12:58.755977 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748130    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 12:12:58.755977 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748133    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 12:12:58.755977 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748135    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 12:12:58.755977 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748138    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 12:12:58.755977 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748141    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 12:12:58.755977 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748143    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 12:12:58.755977 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748146    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 12:12:58.755977 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748149    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 12:12:58.755977 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748151    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 12:12:58.755977 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748154    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 12:12:58.755977 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748156    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 12:12:58.755977 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748159    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 12:12:58.755977 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748161    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 12:12:58.755977 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748164    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 12:12:58.755977 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748167    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 12:12:58.755977 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748169    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 12:12:58.755977 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748172    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 12:12:58.755977 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748174    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 12:12:58.755977 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748178    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 12:12:58.755977 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748181    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 12:12:58.756532 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748183    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 12:12:58.756532 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748186    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 12:12:58.756532 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748188    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 12:12:58.756532 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748191    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 12:12:58.756532 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748193    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 12:12:58.756532 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748196    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 12:12:58.756532 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748199    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 12:12:58.756532 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748201    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 12:12:58.756532 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748204    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 12:12:58.756532 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748209    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 12:12:58.756532 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748213    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 12:12:58.756532 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748216    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 12:12:58.756532 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748219    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 12:12:58.756532 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748222    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 12:12:58.756532 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748225    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 12:12:58.756532 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748227    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 12:12:58.756532 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748230    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 12:12:58.756532 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748232    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 12:12:58.756532 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748235    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 12:12:58.757024 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748237    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 12:12:58.757024 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748240    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 12:12:58.757024 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748243    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 12:12:58.757024 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748245    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 12:12:58.757024 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748248    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 12:12:58.757024 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.748250    2578 feature_gate.go:328] unrecognized feature gate: Example
May 25 12:12:58.757024 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.748256    2578 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 12:12:58.757024 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.756981    2578 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 25 12:12:58.757024 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.757000    2578 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 25 12:12:58.757273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757049    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 12:12:58.757273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757064    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 12:12:58.757273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757068    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 12:12:58.757273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757071    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 12:12:58.757273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757074    2578 feature_gate.go:328] unrecognized feature gate: Example2
May 25 12:12:58.757273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757078    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 12:12:58.757273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757080    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 12:12:58.757273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757083    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 12:12:58.757273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757086    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 12:12:58.757273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757088    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 12:12:58.757273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757091    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 12:12:58.757273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757094    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 12:12:58.757273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757096    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 12:12:58.757273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757099    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 12:12:58.757273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757102    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 12:12:58.757273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757105    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 12:12:58.757273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757107    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 12:12:58.757273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757110    2578 feature_gate.go:328] unrecognized feature gate: Example
May 25 12:12:58.757273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757114    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 12:12:58.757273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757116    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 12:12:58.757767 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757119    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 12:12:58.757767 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757121    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 12:12:58.757767 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757124    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 12:12:58.757767 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757126    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 12:12:58.757767 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757129    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 12:12:58.757767 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757132    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 12:12:58.757767 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757134    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 12:12:58.757767 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757137    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 12:12:58.757767 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757140    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 12:12:58.757767 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757143    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 12:12:58.757767 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757145    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 12:12:58.757767 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757148    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 12:12:58.757767 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757151    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 12:12:58.757767 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757154    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 12:12:58.757767 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757159    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 12:12:58.757767 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757163    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 12:12:58.757767 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757166    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 12:12:58.757767 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757168    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 12:12:58.757767 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757171    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 12:12:58.757767 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757174    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 12:12:58.758291 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757177    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 12:12:58.758291 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757179    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 12:12:58.758291 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757182    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 12:12:58.758291 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757184    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 12:12:58.758291 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757187    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 12:12:58.758291 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757190    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 12:12:58.758291 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757192    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 12:12:58.758291 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757195    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 12:12:58.758291 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757198    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 12:12:58.758291 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757201    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 12:12:58.758291 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757203    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 12:12:58.758291 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757206    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 12:12:58.758291 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757208    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 12:12:58.758291 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757211    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 12:12:58.758291 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757214    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 12:12:58.758291 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757216    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 12:12:58.758291 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757220    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 12:12:58.758291 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757225    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 12:12:58.758291 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757228    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 12:12:58.758761 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757231    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 12:12:58.758761 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757233    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 12:12:58.758761 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757236    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 12:12:58.758761 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757239    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 12:12:58.758761 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757241    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 12:12:58.758761 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757244    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 12:12:58.758761 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757248    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 12:12:58.758761 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757250    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 12:12:58.758761 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757253    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 12:12:58.758761 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757256    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 12:12:58.758761 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757258    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 12:12:58.758761 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757261    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 12:12:58.758761 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757263    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 12:12:58.758761 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757266    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 12:12:58.758761 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757269    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 12:12:58.758761 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757271    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 12:12:58.758761 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757274    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 12:12:58.758761 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757277    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 12:12:58.758761 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757279    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 12:12:58.759316 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757282    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 12:12:58.759316 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757285    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 12:12:58.759316 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757287    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 12:12:58.759316 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757290    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 12:12:58.759316 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757293    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 12:12:58.759316 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757296    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 12:12:58.759316 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757298    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 12:12:58.759316 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757301    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 12:12:58.759316 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.757306    2578 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 12:12:58.759316 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757405    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 12:12:58.759316 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757411    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 12:12:58.759316 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757413    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 12:12:58.759316 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757416    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 12:12:58.759316 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757420    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 12:12:58.759316 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757422    2578 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 12:12:58.759703 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757425    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 12:12:58.759703 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757428    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 12:12:58.759703 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757431    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 12:12:58.759703 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757433    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 12:12:58.759703 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757436    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 12:12:58.759703 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757439    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 12:12:58.759703 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757442    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 12:12:58.759703 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757445    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 12:12:58.759703 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757448    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 12:12:58.759703 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757450    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 12:12:58.759703 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757453    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 12:12:58.759703 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757456    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 12:12:58.759703 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757459    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 12:12:58.759703 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757461    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 12:12:58.759703 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757464    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 12:12:58.759703 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757466    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 12:12:58.759703 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757469    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 12:12:58.759703 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757471    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 12:12:58.759703 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757474    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 12:12:58.759703 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757477    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 12:12:58.760256 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757479    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 12:12:58.760256 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757482    2578 feature_gate.go:328] unrecognized feature gate: Example2
May 25 12:12:58.760256 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757484    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 12:12:58.760256 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757487    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 12:12:58.760256 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757489    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 12:12:58.760256 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757492    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 12:12:58.760256 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757495    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 12:12:58.760256 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757497    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 12:12:58.760256 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757500    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 12:12:58.760256 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757502    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 12:12:58.760256 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757505    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 12:12:58.760256 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757508    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 12:12:58.760256 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757510    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 12:12:58.760256 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757513    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 12:12:58.760256 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757516    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 12:12:58.760256 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757518    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 12:12:58.760256 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757521    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 12:12:58.760256 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757525    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 12:12:58.760256 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757527    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 12:12:58.760256 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757530    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 12:12:58.760736 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757533    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 12:12:58.760736 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757535    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 12:12:58.760736 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757538    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 12:12:58.760736 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757540    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 12:12:58.760736 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757543    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 12:12:58.760736 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757546    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 12:12:58.760736 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757548    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 12:12:58.760736 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757551    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 12:12:58.760736 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757553    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 12:12:58.760736 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757556    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 12:12:58.760736 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757558    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 12:12:58.760736 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757561    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 12:12:58.760736 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757563    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 12:12:58.760736 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757566    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 12:12:58.760736 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757569    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 12:12:58.760736 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757571    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 12:12:58.760736 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757573    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 12:12:58.760736 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757576    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 12:12:58.760736 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757578    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 12:12:58.760736 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757581    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 12:12:58.761273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757583    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 12:12:58.761273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757587    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 12:12:58.761273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757590    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 12:12:58.761273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757592    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 12:12:58.761273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757597    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 12:12:58.761273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757600    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 12:12:58.761273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757603    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 12:12:58.761273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757606    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 12:12:58.761273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757610    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 12:12:58.761273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757613    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 12:12:58.761273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757616    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 12:12:58.761273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757619    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 12:12:58.761273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757622    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 12:12:58.761273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757625    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 12:12:58.761273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757628    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 12:12:58.761273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757631    2578 feature_gate.go:328] unrecognized feature gate: Example
May 25 12:12:58.761273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757633    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 12:12:58.761273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757636    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 12:12:58.761273 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757639    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 12:12:58.761772 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:58.757643    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 12:12:58.761772 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.757648    2578 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 12:12:58.761772 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.758719    2578 server.go:962] "Client rotation is on, will bootstrap in background"
May 25 12:12:58.761772 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.761147    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 25 12:12:58.762175 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.762162    2578 server.go:1019] "Starting client certificate rotation"
May 25 12:12:58.762292 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.762271    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 12:12:58.763182 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.763169    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 12:12:58.792203 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.792176    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 12:12:58.798644 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.798617    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 12:12:58.811344 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.811316    2578 log.go:25] "Validated CRI v1 runtime API"
May 25 12:12:58.817635 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.817621    2578 log.go:25] "Validated CRI v1 image API"
May 25 12:12:58.820717 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.820696    2578 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 25 12:12:58.823738 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.823716    2578 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 12:12:58.826346 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.826327    2578 fs.go:135] Filesystem UUIDs: map[4ed6a41f-08c4-4252-813f-8d9c4e340584:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 97806d4f-eca4-44d9-a98d-788011b54b53:/dev/nvme0n1p3]
May 25 12:12:58.826410 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.826346    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 25 12:12:58.831381 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.831269    2578 manager.go:217] Machine: {Timestamp:2026-05-25 12:12:58.830732525 +0000 UTC m=+0.471927295 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3090585 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26297f5928de88d2c22efe9123cb4e SystemUUID:ec26297f-5928-de88-d2c2-2efe9123cb4e BootID:091817a1-0259-4108-8f43-660387d14fa5 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:77:70:66:01:99 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:77:70:66:01:99 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:c5:d8:e3:79:68 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 25 12:12:58.832110 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.832098    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 25 12:12:58.832206 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.832193    2578 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 25 12:12:58.833459 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.833433    2578 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 25 12:12:58.833614 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.833462    2578 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-107.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 25 12:12:58.833662 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.833625    2578 topology_manager.go:138] "Creating topology manager with none policy"
May 25 12:12:58.833662 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.833633    2578 container_manager_linux.go:306] "Creating device plugin manager"
May 25 12:12:58.833662 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.833640    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 12:12:58.834566 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.834555    2578 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 12:12:58.836426 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.836414    2578 state_mem.go:36] "Initialized new in-memory state store"
May 25 12:12:58.836543 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.836534    2578 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 25 12:12:58.840182 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.840167    2578 kubelet.go:491] "Attempting to sync node with API server"
May 25 12:12:58.840263 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.840187    2578 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 25 12:12:58.840263 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.840207    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 25 12:12:58.840263 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.840226    2578 kubelet.go:397] "Adding apiserver pod source"
May 25 12:12:58.840263 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.840235    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 25 12:12:58.841499 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.841486    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 12:12:58.841549 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.841505    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 12:12:58.845261 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.845241    2578 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 25 12:12:58.847696 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.847678    2578 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 25 12:12:58.851259 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.851232    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 25 12:12:58.851259 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.851259    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 25 12:12:58.851402 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.851269    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 25 12:12:58.851402 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.851285    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 25 12:12:58.851402 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.851298    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 25 12:12:58.851402 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.851307    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 25 12:12:58.851402 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.851315    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 25 12:12:58.851402 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.851324    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 25 12:12:58.851402 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.851336    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 25 12:12:58.851402 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.851346    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 25 12:12:58.851402 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.851359    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 25 12:12:58.851402 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.851369    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 25 12:12:58.853119 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.853099    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 25 12:12:58.853189 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.853133    2578 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 25 12:12:58.856610 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.856587    2578 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-107.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 25 12:12:58.856903 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.856890    2578 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 25 12:12:58.856952 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.856929    2578 server.go:1295] "Started kubelet"
May 25 12:12:58.857360 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.857340    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 25 12:12:58.857741 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:58.857719    2578 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-107.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 25 12:12:58.857741 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:58.857719    2578 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 25 12:12:58.857755 ip-10-0-143-107 systemd[1]: Started Kubernetes Kubelet.
May 25 12:12:58.857896 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.857787    2578 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 25 12:12:58.857937 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.857896    2578 server_v1.go:47] "podresources" method="list" useActivePods=true
May 25 12:12:58.859858 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.859837    2578 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 25 12:12:58.860352 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.860336    2578 server.go:317] "Adding debug handlers to kubelet server"
May 25 12:12:58.864395 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:58.862857    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-107.ec2.internal.18b2cde34a8eb697  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-107.ec2.internal,UID:ip-10-0-143-107.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-107.ec2.internal,},FirstTimestamp:2026-05-25 12:12:58.856904343 +0000 UTC m=+0.498099111,LastTimestamp:2026-05-25 12:12:58.856904343 +0000 UTC m=+0.498099111,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-107.ec2.internal,}"
May 25 12:12:58.867705 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.867688    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 25 12:12:58.867705 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.867693    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 25 12:12:58.867856 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.867753    2578 volume_manager.go:295] "The desired_state_of_world populator starts"
May 25 12:12:58.867856 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.867765    2578 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 25 12:12:58.867856 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.867832    2578 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 25 12:12:58.867994 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.867912    2578 reconstruct.go:97] "Volume reconstruction finished"
May 25 12:12:58.867994 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.867921    2578 reconciler.go:26] "Reconciler: start to sync state"
May 25 12:12:58.868796 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.868777    2578 factory.go:55] Registering systemd factory
May 25 12:12:58.868796 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.868798    2578 factory.go:223] Registration of the systemd container factory successfully
May 25 12:12:58.868925 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:58.868864    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-107.ec2.internal\" not found"
May 25 12:12:58.869355 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.869037    2578 factory.go:153] Registering CRI-O factory
May 25 12:12:58.869355 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.869288    2578 factory.go:223] Registration of the crio container factory successfully
May 25 12:12:58.869478 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.869415    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 25 12:12:58.869478 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.869438    2578 factory.go:103] Registering Raw factory
May 25 12:12:58.869478 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.869451    2578 manager.go:1196] Started watching for new ooms in manager
May 25 12:12:58.870273 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:58.869703    2578 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 25 12:12:58.870273 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.869842    2578 manager.go:319] Starting recovery of all containers
May 25 12:12:58.870736 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:58.870707    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-107.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 25 12:12:58.870854 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:58.870771    2578 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 25 12:12:58.870972 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.870910    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d2csz"
May 25 12:12:58.881414 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.881388    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d2csz"
May 25 12:12:58.881969 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.881950    2578 manager.go:324] Recovery completed
May 25 12:12:58.886704 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.886687    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:12:58.888850 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.888833    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-107.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:12:58.888913 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.888867    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-107.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:12:58.888913 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.888881    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-107.ec2.internal" event="NodeHasSufficientPID"
May 25 12:12:58.889440 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.889425    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 25 12:12:58.889440 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.889437    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 25 12:12:58.889545 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.889452    2578 state_mem.go:36] "Initialized new in-memory state store"
May 25 12:12:58.891234 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:58.891166    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-107.ec2.internal.18b2cde34c763193  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-107.ec2.internal,UID:ip-10-0-143-107.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-107.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-107.ec2.internal,},FirstTimestamp:2026-05-25 12:12:58.888851859 +0000 UTC m=+0.530046627,LastTimestamp:2026-05-25 12:12:58.888851859 +0000 UTC m=+0.530046627,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-107.ec2.internal,}"
May 25 12:12:58.893159 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.893145    2578 policy_none.go:49] "None policy: Start"
May 25 12:12:58.893227 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.893162    2578 memory_manager.go:186] "Starting memorymanager" policy="None"
May 25 12:12:58.893227 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.893172    2578 state_mem.go:35] "Initializing new in-memory state store"
May 25 12:12:58.947287 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.930722    2578 manager.go:341] "Starting Device Plugin manager"
May 25 12:12:58.947287 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:58.930766    2578 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 25 12:12:58.947287 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.930782    2578 server.go:85] "Starting device plugin registration server"
May 25 12:12:58.947287 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.931094    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
May 25 12:12:58.947287 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.931106    2578 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 25 12:12:58.947287 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.931175    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 25 12:12:58.947287 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.931267    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 25 12:12:58.947287 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.931276    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 25 12:12:58.947287 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:58.931950    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 25 12:12:58.947287 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:58.931994    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-107.ec2.internal\" not found"
May 25 12:12:58.947287 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.933296    2578 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 25 12:12:58.947287 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.934620    2578 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 25 12:12:58.947287 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.934639    2578 status_manager.go:230] "Starting to sync pod status with apiserver"
May 25 12:12:58.947287 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.934658    2578 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 25 12:12:58.947287 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.934664    2578 kubelet.go:2451] "Starting kubelet main sync loop"
May 25 12:12:58.947287 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:58.934699    2578 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 25 12:12:58.947287 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:58.937372    2578 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:12:59.032189 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.032105    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:12:59.033301 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.033284    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-107.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:12:59.033418 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.033316    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-107.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:12:59.033418 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.033330    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-107.ec2.internal" event="NodeHasSufficientPID"
May 25 12:12:59.033418 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.033354    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-107.ec2.internal"
May 25 12:12:59.035428 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.035410    2578 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-107.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-107.ec2.internal"]
May 25 12:12:59.035478 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.035474    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:12:59.036370 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.036357    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-107.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:12:59.036418 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.036397    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-107.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:12:59.036418 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.036409    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-107.ec2.internal" event="NodeHasSufficientPID"
May 25 12:12:59.038662 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.038649    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:12:59.038837 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.038822    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-107.ec2.internal"
May 25 12:12:59.038877 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.038865    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:12:59.039337 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.039318    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-107.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:12:59.039402 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.039342    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-107.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:12:59.039402 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.039355    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-107.ec2.internal" event="NodeHasSufficientPID"
May 25 12:12:59.039402 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.039355    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-107.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:12:59.039402 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.039378    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-107.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:12:59.039402 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.039387    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-107.ec2.internal" event="NodeHasSufficientPID"
May 25 12:12:59.041598 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.041583    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-107.ec2.internal"
May 25 12:12:59.041673 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.041607    2578 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 12:12:59.042220 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.042201    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-107.ec2.internal"
May 25 12:12:59.042336 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:59.042223    2578 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-107.ec2.internal\": node \"ip-10-0-143-107.ec2.internal\" not found"
May 25 12:12:59.042402 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.042336    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-107.ec2.internal" event="NodeHasSufficientMemory"
May 25 12:12:59.042402 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.042360    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-107.ec2.internal" event="NodeHasNoDiskPressure"
May 25 12:12:59.042402 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.042376    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-107.ec2.internal" event="NodeHasSufficientPID"
May 25 12:12:59.057300 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:59.057274    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-107.ec2.internal\" not found"
May 25 12:12:59.062688 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:59.062671    2578 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-107.ec2.internal\" not found" node="ip-10-0-143-107.ec2.internal"
May 25 12:12:59.066222 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:59.066202    2578 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-107.ec2.internal\" not found" node="ip-10-0-143-107.ec2.internal"
May 25 12:12:59.069195 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.069173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/decdc30bec8216583ece4597ca83be0b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-107.ec2.internal\" (UID: \"decdc30bec8216583ece4597ca83be0b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-107.ec2.internal"
May 25 12:12:59.069283 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.069201    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/decdc30bec8216583ece4597ca83be0b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-107.ec2.internal\" (UID: \"decdc30bec8216583ece4597ca83be0b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-107.ec2.internal"
May 25 12:12:59.069283 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.069217    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a3617d771b965ababd68b956e9ea4412-config\") pod \"kube-apiserver-proxy-ip-10-0-143-107.ec2.internal\" (UID: \"a3617d771b965ababd68b956e9ea4412\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-107.ec2.internal"
May 25 12:12:59.158317 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:59.158283    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-107.ec2.internal\" not found"
May 25 12:12:59.169971 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.169946    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/decdc30bec8216583ece4597ca83be0b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-107.ec2.internal\" (UID: \"decdc30bec8216583ece4597ca83be0b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-107.ec2.internal"
May 25 12:12:59.170049 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.169982    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a3617d771b965ababd68b956e9ea4412-config\") pod \"kube-apiserver-proxy-ip-10-0-143-107.ec2.internal\" (UID: \"a3617d771b965ababd68b956e9ea4412\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-107.ec2.internal"
May 25 12:12:59.170049 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.170000    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/decdc30bec8216583ece4597ca83be0b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-107.ec2.internal\" (UID: \"decdc30bec8216583ece4597ca83be0b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-107.ec2.internal"
May 25 12:12:59.170163 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.170072    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a3617d771b965ababd68b956e9ea4412-config\") pod \"kube-apiserver-proxy-ip-10-0-143-107.ec2.internal\" (UID: \"a3617d771b965ababd68b956e9ea4412\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-107.ec2.internal"
May 25 12:12:59.170163 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.170080    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/decdc30bec8216583ece4597ca83be0b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-107.ec2.internal\" (UID: \"decdc30bec8216583ece4597ca83be0b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-107.ec2.internal"
May 25 12:12:59.170163 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.170124    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/decdc30bec8216583ece4597ca83be0b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-107.ec2.internal\" (UID: \"decdc30bec8216583ece4597ca83be0b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-107.ec2.internal"
May 25 12:12:59.258832 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:59.258801    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-107.ec2.internal\" not found"
May 25 12:12:59.359478 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:59.359444    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-107.ec2.internal\" not found"
May 25 12:12:59.364639 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.364617    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-107.ec2.internal"
May 25 12:12:59.369954 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.369935    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-107.ec2.internal"
May 25 12:12:59.460132 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:59.460091    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-107.ec2.internal\" not found"
May 25 12:12:59.560624 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:59.560596    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-107.ec2.internal\" not found"
May 25 12:12:59.661198 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:59.661112    2578 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-107.ec2.internal\" not found"
May 25 12:12:59.669798 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.669770    2578 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:12:59.732335 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.732305    2578 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:12:59.762830 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.762808    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 25 12:12:59.763548 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.762964    2578 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 12:12:59.763548 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.762982    2578 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 12:12:59.763548 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.762997    2578 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 12:12:59.769179 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.769162    2578 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-107.ec2.internal"
May 25 12:12:59.792725 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.792698    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 12:12:59.793801 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.793766    2578 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-107.ec2.internal"
May 25 12:12:59.803865 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.803849    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 12:12:59.841103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.841071    2578 apiserver.go:52] "Watching apiserver"
May 25 12:12:59.850603 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.850579    2578 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 25 12:12:59.852146 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.851632    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-107.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6","openshift-cluster-node-tuning-operator/tuned-mpgf7","openshift-dns/node-resolver-sg52n","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-107.ec2.internal","openshift-multus/network-metrics-daemon-n9vrb","openshift-network-diagnostics/network-check-target-zhvk8","openshift-network-operator/iptables-alerter-qkfd7","kube-system/konnectivity-agent-2fqnj","openshift-image-registry/node-ca-j2hhk","openshift-multus/multus-additional-cni-plugins-5xg5h","openshift-multus/multus-zwd5q","openshift-ovn-kubernetes/ovnkube-node-qt8dc"]
May 25 12:12:59.854326 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.854305    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6"
May 25 12:12:59.856388 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.856368    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.856945 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.856920    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 25 12:12:59.857041 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.857018    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 25 12:12:59.857120 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.857025    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mllxb\""
May 25 12:12:59.857120 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.857029    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 25 12:12:59.858486 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.858470    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sg52n"
May 25 12:12:59.859235 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.859220    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 25 12:12:59.859289 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.859251    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 25 12:12:59.859417 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.859391    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gxcf7\""
May 25 12:12:59.860696 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.860674    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 25 12:12:59.860814 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.860708    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2t6cp\""
May 25 12:12:59.860814 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.860781    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:12:59.860967 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:59.860858    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vrb" podUID="02a94135-b85a-4ab1-9dd1-2ae75e77f2de"
May 25 12:12:59.860967 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.860950    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 25 12:12:59.862851 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.862830    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:12:59.862966 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:59.862888    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhvk8" podUID="cae409cd-445f-40b9-9851-c89b09c1ac43"
May 25 12:12:59.864914 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.864895    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qkfd7"
May 25 12:12:59.867364 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.867343    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 25 12:12:59.867364 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.867342    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 25 12:12:59.867528 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.867378    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 25 12:12:59.867528 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.867389    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rgsg9\""
May 25 12:12:59.867877 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.867854    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 25 12:12:59.868789 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.868770    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2fqnj"
May 25 12:12:59.871151 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.871009    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-l9nx4\""
May 25 12:12:59.871151 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.871025    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 25 12:12:59.871151 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.871038    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 25 12:12:59.871387 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.871371    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j2hhk"
May 25 12:12:59.873174 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873151    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-etc-modprobe-d\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.873268 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873183    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4jlw\" (UniqueName: \"kubernetes.io/projected/69c3d413-5409-4322-8e78-1cfa1d96ed00-kube-api-access-c4jlw\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.873268 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873200    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8qmz\" (UniqueName: \"kubernetes.io/projected/02afa42e-73a7-4f9b-a5d7-609a53a507b1-kube-api-access-m8qmz\") pod \"node-resolver-sg52n\" (UID: \"02afa42e-73a7-4f9b-a5d7-609a53a507b1\") " pod="openshift-dns/node-resolver-sg52n"
May 25 12:12:59.873268 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/01b74961-b200-4de0-9d76-e43923f19633-agent-certs\") pod \"konnectivity-agent-2fqnj\" (UID: \"01b74961-b200-4de0-9d76-e43923f19633\") " pod="kube-system/konnectivity-agent-2fqnj"
May 25 12:12:59.873450 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873274    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ddd277f-8d2e-4f5a-b2f9-788afa7f7e33-host-slash\") pod \"iptables-alerter-qkfd7\" (UID: \"6ddd277f-8d2e-4f5a-b2f9-788afa7f7e33\") " pod="openshift-network-operator/iptables-alerter-qkfd7"
May 25 12:12:59.873450 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873291    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft4lg\" (UniqueName: \"kubernetes.io/projected/6ddd277f-8d2e-4f5a-b2f9-788afa7f7e33-kube-api-access-ft4lg\") pod \"iptables-alerter-qkfd7\" (UID: \"6ddd277f-8d2e-4f5a-b2f9-788afa7f7e33\") " pod="openshift-network-operator/iptables-alerter-qkfd7"
May 25 12:12:59.873450 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873311    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48a83491-d11a-4002-8df4-66913cf817f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-68xw6\" (UID: \"48a83491-d11a-4002-8df4-66913cf817f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6"
May 25 12:12:59.873450 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873376    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-etc-sysconfig\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.873450 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873419    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-run\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.873450 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873447    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69c3d413-5409-4322-8e78-1cfa1d96ed00-tmp\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.873701 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-etc-kubernetes\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.873701 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873497    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-host\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.873701 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873521    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48a83491-d11a-4002-8df4-66913cf817f6-socket-dir\") pod \"aws-ebs-csi-driver-node-68xw6\" (UID: \"48a83491-d11a-4002-8df4-66913cf817f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6"
May 25 12:12:59.873701 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873544    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48a83491-d11a-4002-8df4-66913cf817f6-sys-fs\") pod \"aws-ebs-csi-driver-node-68xw6\" (UID: \"48a83491-d11a-4002-8df4-66913cf817f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6"
May 25 12:12:59.873701 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873568    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/69c3d413-5409-4322-8e78-1cfa1d96ed00-etc-tuned\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.873701 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873592    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rfpr\" (UniqueName: \"kubernetes.io/projected/cae409cd-445f-40b9-9851-c89b09c1ac43-kube-api-access-2rfpr\") pod \"network-check-target-zhvk8\" (UID: \"cae409cd-445f-40b9-9851-c89b09c1ac43\") " pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:12:59.873701 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873607    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5xg5h"
May 25 12:12:59.873701 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6ddd277f-8d2e-4f5a-b2f9-788afa7f7e33-iptables-alerter-script\") pod \"iptables-alerter-qkfd7\" (UID: \"6ddd277f-8d2e-4f5a-b2f9-788afa7f7e33\") " pod="openshift-network-operator/iptables-alerter-qkfd7"
May 25 12:12:59.873701 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873639    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 25 12:12:59.873701 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873689    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-lib-modules\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.874179 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873729    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48a83491-d11a-4002-8df4-66913cf817f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-68xw6\" (UID: \"48a83491-d11a-4002-8df4-66913cf817f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6"
May 25 12:12:59.874179 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873761    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-metrics-certs\") pod \"network-metrics-daemon-n9vrb\" (UID: \"02a94135-b85a-4ab1-9dd1-2ae75e77f2de\") " pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:12:59.874179 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873794    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt6l8\" (UniqueName: \"kubernetes.io/projected/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-kube-api-access-vt6l8\") pod \"network-metrics-daemon-n9vrb\" (UID: \"02a94135-b85a-4ab1-9dd1-2ae75e77f2de\") " pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:12:59.874179 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02afa42e-73a7-4f9b-a5d7-609a53a507b1-tmp-dir\") pod \"node-resolver-sg52n\" (UID: \"02afa42e-73a7-4f9b-a5d7-609a53a507b1\") " pod="openshift-dns/node-resolver-sg52n"
May 25 12:12:59.874179 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873844    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-etc-sysctl-conf\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.874179 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873848    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 25 12:12:59.874179 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873873    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 25 12:12:59.874179 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873866    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-etc-systemd\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.874179 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873914    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-var-lib-kubelet\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.874179 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873938    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-sys\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.874179 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873961    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02afa42e-73a7-4f9b-a5d7-609a53a507b1-hosts-file\") pod \"node-resolver-sg52n\" (UID: \"02afa42e-73a7-4f9b-a5d7-609a53a507b1\") " pod="openshift-dns/node-resolver-sg52n"
May 25 12:12:59.874179 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.873984    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/01b74961-b200-4de0-9d76-e43923f19633-konnectivity-ca\") pod \"konnectivity-agent-2fqnj\" (UID: \"01b74961-b200-4de0-9d76-e43923f19633\") " pod="kube-system/konnectivity-agent-2fqnj"
May 25 12:12:59.874179 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.874020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-etc-sysctl-d\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.874179 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.874042    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48a83491-d11a-4002-8df4-66913cf817f6-registration-dir\") pod \"aws-ebs-csi-driver-node-68xw6\" (UID: \"48a83491-d11a-4002-8df4-66913cf817f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6"
May 25 12:12:59.874179 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.874082    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48a83491-d11a-4002-8df4-66913cf817f6-device-dir\") pod \"aws-ebs-csi-driver-node-68xw6\" (UID: \"48a83491-d11a-4002-8df4-66913cf817f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6"
May 25 12:12:59.874179 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.874113    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zntm\" (UniqueName: \"kubernetes.io/projected/48a83491-d11a-4002-8df4-66913cf817f6-kube-api-access-6zntm\") pod \"aws-ebs-csi-driver-node-68xw6\" (UID: \"48a83491-d11a-4002-8df4-66913cf817f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6"
May 25 12:12:59.874631 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.874304    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-l54fv\""
May 25 12:12:59.875844 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.875830    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zwd5q"
May 25 12:12:59.876285 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.876266    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 25 12:12:59.876430 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.876414    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 25 12:12:59.876734 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.876718    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 25 12:12:59.877079 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.877049    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hqs9h\""
May 25 12:12:59.877154 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.877137    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 25 12:12:59.877213 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.877193    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 25 12:12:59.878241 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.878027    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jfqq7\""
May 25 12:12:59.878241 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.878155    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:12:59.878241 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.878186    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 25 12:12:59.880415 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.880397    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 25 12:12:59.880693 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.880678    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 25 12:12:59.880768 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.880718    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 25 12:12:59.880768 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.880730    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 25 12:12:59.880872 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.880760    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 25 12:12:59.881017 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.881002    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-j48lg\""
May 25 12:12:59.881084 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.881069    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 25 12:12:59.881269 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.881251    2578 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 12:12:59.884628 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.884320    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-24 12:07:58 +0000 UTC" deadline="2027-11-17 01:03:58.498898707 +0000 UTC"
May 25 12:12:59.884628 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.884370    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12972h50m58.614535241s"
May 25 12:12:59.904278 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.904255    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mdhlq"
May 25 12:12:59.910796 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.910772    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mdhlq"
May 25 12:12:59.968526 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.968504    2578 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 25 12:12:59.974575 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.974554    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/01b74961-b200-4de0-9d76-e43923f19633-konnectivity-ca\") pod \"konnectivity-agent-2fqnj\" (UID: \"01b74961-b200-4de0-9d76-e43923f19633\") " pod="kube-system/konnectivity-agent-2fqnj"
May 25 12:12:59.974651 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.974588    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48a83491-d11a-4002-8df4-66913cf817f6-registration-dir\") pod \"aws-ebs-csi-driver-node-68xw6\" (UID: \"48a83491-d11a-4002-8df4-66913cf817f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6"
May 25 12:12:59.974651 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.974615    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48a83491-d11a-4002-8df4-66913cf817f6-device-dir\") pod \"aws-ebs-csi-driver-node-68xw6\" (UID: \"48a83491-d11a-4002-8df4-66913cf817f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6"
May 25 12:12:59.974751 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.974651    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6zntm\" (UniqueName: \"kubernetes.io/projected/48a83491-d11a-4002-8df4-66913cf817f6-kube-api-access-6zntm\") pod \"aws-ebs-csi-driver-node-68xw6\" (UID: \"48a83491-d11a-4002-8df4-66913cf817f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6"
May 25 12:12:59.974751 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.974680    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/48a83491-d11a-4002-8df4-66913cf817f6-device-dir\") pod \"aws-ebs-csi-driver-node-68xw6\" (UID: \"48a83491-d11a-4002-8df4-66913cf817f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6"
May 25 12:12:59.974751 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.974682    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a34e5474-3229-4811-8b0e-dee727e7d8b7-host\") pod \"node-ca-j2hhk\" (UID: \"a34e5474-3229-4811-8b0e-dee727e7d8b7\") " pod="openshift-image-registry/node-ca-j2hhk"
May 25 12:12:59.974751 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.974738    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-systemd-units\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:12:59.974931 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.974754    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/48a83491-d11a-4002-8df4-66913cf817f6-registration-dir\") pod \"aws-ebs-csi-driver-node-68xw6\" (UID: \"48a83491-d11a-4002-8df4-66913cf817f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6"
May 25 12:12:59.974931 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.974853    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-etc-modprobe-d\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.974931 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.974886    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/01b74961-b200-4de0-9d76-e43923f19633-agent-certs\") pod \"konnectivity-agent-2fqnj\" (UID: \"01b74961-b200-4de0-9d76-e43923f19633\") " pod="kube-system/konnectivity-agent-2fqnj"
May 25 12:12:59.975084 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.974995    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f97986ac-3ba7-4762-9e76-38ea2ac839b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5xg5h\" (UID: \"f97986ac-3ba7-4762-9e76-38ea2ac839b6\") " pod="openshift-multus/multus-additional-cni-plugins-5xg5h"
May 25 12:12:59.975084 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975015    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-etc-modprobe-d\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.975084 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975042    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-multus-cni-dir\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:12:59.975259 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975089    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f97986ac-3ba7-4762-9e76-38ea2ac839b6-os-release\") pod \"multus-additional-cni-plugins-5xg5h\" (UID: \"f97986ac-3ba7-4762-9e76-38ea2ac839b6\") " pod="openshift-multus/multus-additional-cni-plugins-5xg5h"
May 25 12:12:59.975259 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975102    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/01b74961-b200-4de0-9d76-e43923f19633-konnectivity-ca\") pod \"konnectivity-agent-2fqnj\" (UID: \"01b74961-b200-4de0-9d76-e43923f19633\") " pod="kube-system/konnectivity-agent-2fqnj"
May 25 12:12:59.975259 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975113    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-host-run-netns\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:12:59.975259 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975142    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ft4lg\" (UniqueName: \"kubernetes.io/projected/6ddd277f-8d2e-4f5a-b2f9-788afa7f7e33-kube-api-access-ft4lg\") pod \"iptables-alerter-qkfd7\" (UID: \"6ddd277f-8d2e-4f5a-b2f9-788afa7f7e33\") " pod="openshift-network-operator/iptables-alerter-qkfd7"
May 25 12:12:59.975259 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975168    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48a83491-d11a-4002-8df4-66913cf817f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-68xw6\" (UID: \"48a83491-d11a-4002-8df4-66913cf817f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6"
May 25 12:12:59.975259 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975194    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69c3d413-5409-4322-8e78-1cfa1d96ed00-tmp\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.975259 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975221    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-etc-kubernetes\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:12:59.975510 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975246    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-etc-openvswitch\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:12:59.975510 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975271    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/48a83491-d11a-4002-8df4-66913cf817f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-68xw6\" (UID: \"48a83491-d11a-4002-8df4-66913cf817f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6"
May 25 12:12:59.975510 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975280    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 25 12:12:59.975510 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975308    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48a83491-d11a-4002-8df4-66913cf817f6-socket-dir\") pod \"aws-ebs-csi-driver-node-68xw6\" (UID: \"48a83491-d11a-4002-8df4-66913cf817f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6"
May 25 12:12:59.975510 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975382    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv8fw\" (UniqueName: \"kubernetes.io/projected/f97986ac-3ba7-4762-9e76-38ea2ac839b6-kube-api-access-lv8fw\") pod \"multus-additional-cni-plugins-5xg5h\" (UID: \"f97986ac-3ba7-4762-9e76-38ea2ac839b6\") " pod="openshift-multus/multus-additional-cni-plugins-5xg5h"
May 25 12:12:59.975510 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975432    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-os-release\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:12:59.975510 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975438    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/48a83491-d11a-4002-8df4-66913cf817f6-socket-dir\") pod \"aws-ebs-csi-driver-node-68xw6\" (UID: \"48a83491-d11a-4002-8df4-66913cf817f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6"
May 25 12:12:59.975510 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-hostroot\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:12:59.975510 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975500    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7687fe26-96a3-43ea-9223-0c5d61cd667e-ovnkube-script-lib\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:12:59.975911 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975523    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/69c3d413-5409-4322-8e78-1cfa1d96ed00-etc-tuned\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.975911 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975539    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a34e5474-3229-4811-8b0e-dee727e7d8b7-serviceca\") pod \"node-ca-j2hhk\" (UID: \"a34e5474-3229-4811-8b0e-dee727e7d8b7\") " pod="openshift-image-registry/node-ca-j2hhk"
May 25 12:12:59.975911 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975554    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48a83491-d11a-4002-8df4-66913cf817f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-68xw6\" (UID: \"48a83491-d11a-4002-8df4-66913cf817f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6"
May 25 12:12:59.975911 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975574    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vt6l8\" (UniqueName: \"kubernetes.io/projected/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-kube-api-access-vt6l8\") pod \"network-metrics-daemon-n9vrb\" (UID: \"02a94135-b85a-4ab1-9dd1-2ae75e77f2de\") " pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:12:59.975911 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975617    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-host-kubelet\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:12:59.975911 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975623    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/48a83491-d11a-4002-8df4-66913cf817f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-68xw6\" (UID: \"48a83491-d11a-4002-8df4-66913cf817f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6"
May 25 12:12:59.975911 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975641    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-host-slash\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:12:59.975911 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975666    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-host-cni-netd\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:12:59.975911 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975707    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7687fe26-96a3-43ea-9223-0c5d61cd667e-ovnkube-config\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:12:59.975911 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975748    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02afa42e-73a7-4f9b-a5d7-609a53a507b1-tmp-dir\") pod \"node-resolver-sg52n\" (UID: \"02afa42e-73a7-4f9b-a5d7-609a53a507b1\") " pod="openshift-dns/node-resolver-sg52n"
May 25 12:12:59.975911 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975780    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-etc-sysctl-conf\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.975911 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975806    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4fe8af28-e32b-4375-9cca-803603210b1a-multus-daemon-config\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:12:59.975911 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975830    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-var-lib-openvswitch\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:12:59.975911 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-host-run-ovn-kubernetes\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:12:59.975911 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975878    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-sys\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.975911 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975915    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02afa42e-73a7-4f9b-a5d7-609a53a507b1-hosts-file\") pod \"node-resolver-sg52n\" (UID: \"02afa42e-73a7-4f9b-a5d7-609a53a507b1\") " pod="openshift-dns/node-resolver-sg52n"
May 25 12:12:59.976637 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975937    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-etc-sysctl-conf\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.976637 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.975987    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-sys\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.976637 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976003    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02afa42e-73a7-4f9b-a5d7-609a53a507b1-hosts-file\") pod \"node-resolver-sg52n\" (UID: \"02afa42e-73a7-4f9b-a5d7-609a53a507b1\") " pod="openshift-dns/node-resolver-sg52n"
May 25 12:12:59.976637 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976007    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02afa42e-73a7-4f9b-a5d7-609a53a507b1-tmp-dir\") pod \"node-resolver-sg52n\" (UID: \"02afa42e-73a7-4f9b-a5d7-609a53a507b1\") " pod="openshift-dns/node-resolver-sg52n"
May 25 12:12:59.976637 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976021    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f97986ac-3ba7-4762-9e76-38ea2ac839b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-5xg5h\" (UID: \"f97986ac-3ba7-4762-9e76-38ea2ac839b6\") " pod="openshift-multus/multus-additional-cni-plugins-5xg5h"
May 25 12:12:59.976637 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976049    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f97986ac-3ba7-4762-9e76-38ea2ac839b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5xg5h\" (UID: \"f97986ac-3ba7-4762-9e76-38ea2ac839b6\") " pod="openshift-multus/multus-additional-cni-plugins-5xg5h"
May 25 12:12:59.976637 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976087    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-multus-socket-dir-parent\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:12:59.976637 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976113    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-host-run-multus-certs\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:12:59.976637 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976143    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-etc-sysctl-d\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.976637 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976171    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f97986ac-3ba7-4762-9e76-38ea2ac839b6-system-cni-dir\") pod \"multus-additional-cni-plugins-5xg5h\" (UID: \"f97986ac-3ba7-4762-9e76-38ea2ac839b6\") " pod="openshift-multus/multus-additional-cni-plugins-5xg5h"
May 25 12:12:59.976637 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976193    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4fe8af28-e32b-4375-9cca-803603210b1a-cni-binary-copy\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:12:59.976637 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976212    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-etc-sysctl-d\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.976637 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976230    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-host-var-lib-cni-bin\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:12:59.976637 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976248    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-host-run-netns\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:12:59.976637 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976280    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c4jlw\" (UniqueName: \"kubernetes.io/projected/69c3d413-5409-4322-8e78-1cfa1d96ed00-kube-api-access-c4jlw\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.976637 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976300    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m8qmz\" (UniqueName: \"kubernetes.io/projected/02afa42e-73a7-4f9b-a5d7-609a53a507b1-kube-api-access-m8qmz\") pod \"node-resolver-sg52n\" (UID: \"02afa42e-73a7-4f9b-a5d7-609a53a507b1\") " pod="openshift-dns/node-resolver-sg52n"
May 25 12:12:59.976637 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976321    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-host-run-k8s-cni-cncf-io\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:12:59.977414 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976344    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-run-openvswitch\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:12:59.977414 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976375    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-node-log\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:12:59.977414 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976441    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9xbb\" (UniqueName: \"kubernetes.io/projected/7687fe26-96a3-43ea-9223-0c5d61cd667e-kube-api-access-k9xbb\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:12:59.977414 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976479    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f97986ac-3ba7-4762-9e76-38ea2ac839b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5xg5h\" (UID: \"f97986ac-3ba7-4762-9e76-38ea2ac839b6\") " pod="openshift-multus/multus-additional-cni-plugins-5xg5h"
May 25 12:12:59.977414 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976532    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ddd277f-8d2e-4f5a-b2f9-788afa7f7e33-host-slash\") pod \"iptables-alerter-qkfd7\" (UID: \"6ddd277f-8d2e-4f5a-b2f9-788afa7f7e33\") " pod="openshift-network-operator/iptables-alerter-qkfd7"
May 25 12:12:59.977414 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976575    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-etc-sysconfig\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.977414 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976592    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-run\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.977414 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p7nc\" (UniqueName: \"kubernetes.io/projected/a34e5474-3229-4811-8b0e-dee727e7d8b7-kube-api-access-5p7nc\") pod \"node-ca-j2hhk\" (UID: \"a34e5474-3229-4811-8b0e-dee727e7d8b7\") " pod="openshift-image-registry/node-ca-j2hhk"
May 25 12:12:59.977414 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976645    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-etc-sysconfig\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.977414 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976647    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-run-systemd\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:12:59.977414 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976663    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-run\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.977414 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976680    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-etc-kubernetes\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.977414 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976704    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-host\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.977414 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976712    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6ddd277f-8d2e-4f5a-b2f9-788afa7f7e33-host-slash\") pod \"iptables-alerter-qkfd7\" (UID: \"6ddd277f-8d2e-4f5a-b2f9-788afa7f7e33\") " pod="openshift-network-operator/iptables-alerter-qkfd7"
May 25 12:12:59.977414 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976730    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48a83491-d11a-4002-8df4-66913cf817f6-sys-fs\") pod \"aws-ebs-csi-driver-node-68xw6\" (UID: \"48a83491-d11a-4002-8df4-66913cf817f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6"
May 25 12:12:59.977414 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976760    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-multus-conf-dir\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:12:59.977414 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976785    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-host-cni-bin\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:12:59.978186 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976795    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/48a83491-d11a-4002-8df4-66913cf817f6-sys-fs\") pod \"aws-ebs-csi-driver-node-68xw6\" (UID: \"48a83491-d11a-4002-8df4-66913cf817f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6"
May 25 12:12:59.978186 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976760    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-etc-kubernetes\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.978186 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976764    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-host\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.978186 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976813    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:12:59.978186 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976845    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7687fe26-96a3-43ea-9223-0c5d61cd667e-env-overrides\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:12:59.978186 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976867    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2rfpr\" (UniqueName: \"kubernetes.io/projected/cae409cd-445f-40b9-9851-c89b09c1ac43-kube-api-access-2rfpr\") pod \"network-check-target-zhvk8\" (UID: \"cae409cd-445f-40b9-9851-c89b09c1ac43\") " pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:12:59.978186 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976895    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6ddd277f-8d2e-4f5a-b2f9-788afa7f7e33-iptables-alerter-script\") pod \"iptables-alerter-qkfd7\" (UID: \"6ddd277f-8d2e-4f5a-b2f9-788afa7f7e33\") " pod="openshift-network-operator/iptables-alerter-qkfd7"
May 25 12:12:59.978186 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.976991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f97986ac-3ba7-4762-9e76-38ea2ac839b6-cnibin\") pod \"multus-additional-cni-plugins-5xg5h\" (UID: \"f97986ac-3ba7-4762-9e76-38ea2ac839b6\") " pod="openshift-multus/multus-additional-cni-plugins-5xg5h"
May 25 12:12:59.978186 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.977037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-cnibin\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:12:59.978186 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.977076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-host-var-lib-cni-multus\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:12:59.978186 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.977106    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-host-var-lib-kubelet\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:12:59.978186 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.977130    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llhb7\" (UniqueName: \"kubernetes.io/projected/4fe8af28-e32b-4375-9cca-803603210b1a-kube-api-access-llhb7\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:12:59.978186 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.977167    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-log-socket\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:12:59.978186 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.977196    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-lib-modules\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.978186 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.977215    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-metrics-certs\") pod \"network-metrics-daemon-n9vrb\" (UID: \"02a94135-b85a-4ab1-9dd1-2ae75e77f2de\") " pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:12:59.978186 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.977240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-system-cni-dir\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:12:59.978186 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.977291    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-lib-modules\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.978689 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:59.977307    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:12:59.978689 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.977338    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-etc-systemd\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.978689 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.977362    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-var-lib-kubelet\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.978689 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.977383    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6ddd277f-8d2e-4f5a-b2f9-788afa7f7e33-iptables-alerter-script\") pod \"iptables-alerter-qkfd7\" (UID: \"6ddd277f-8d2e-4f5a-b2f9-788afa7f7e33\") " pod="openshift-network-operator/iptables-alerter-qkfd7"
May 25 12:12:59.978689 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:59.977409    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-metrics-certs podName:02a94135-b85a-4ab1-9dd1-2ae75e77f2de nodeName:}" failed. No retries permitted until 2026-05-25 12:13:00.477366925 +0000 UTC m=+2.118561703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-metrics-certs") pod "network-metrics-daemon-n9vrb" (UID: "02a94135-b85a-4ab1-9dd1-2ae75e77f2de") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:12:59.978689 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.977435    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-etc-systemd\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.978689 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.977409    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69c3d413-5409-4322-8e78-1cfa1d96ed00-var-lib-kubelet\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.978689 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.977478    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-run-ovn\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:12:59.978689 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.977495    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7687fe26-96a3-43ea-9223-0c5d61cd667e-ovn-node-metrics-cert\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:12:59.978689 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.978362    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/69c3d413-5409-4322-8e78-1cfa1d96ed00-etc-tuned\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.978689 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.978395    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69c3d413-5409-4322-8e78-1cfa1d96ed00-tmp\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.978689 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.978626    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/01b74961-b200-4de0-9d76-e43923f19633-agent-certs\") pod \"konnectivity-agent-2fqnj\" (UID: \"01b74961-b200-4de0-9d76-e43923f19633\") " pod="kube-system/konnectivity-agent-2fqnj"
May 25 12:12:59.984175 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.984150    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zntm\" (UniqueName: \"kubernetes.io/projected/48a83491-d11a-4002-8df4-66913cf817f6-kube-api-access-6zntm\") pod \"aws-ebs-csi-driver-node-68xw6\" (UID: \"48a83491-d11a-4002-8df4-66913cf817f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6"
May 25 12:12:59.984296 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:59.984279    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:12:59.984296 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:59.984296    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:12:59.984402 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:59.984308    2578 projected.go:194] Error preparing data for projected volume kube-api-access-2rfpr for pod openshift-network-diagnostics/network-check-target-zhvk8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:12:59.984402 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.984309    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft4lg\" (UniqueName: \"kubernetes.io/projected/6ddd277f-8d2e-4f5a-b2f9-788afa7f7e33-kube-api-access-ft4lg\") pod \"iptables-alerter-qkfd7\" (UID: \"6ddd277f-8d2e-4f5a-b2f9-788afa7f7e33\") " pod="openshift-network-operator/iptables-alerter-qkfd7"
May 25 12:12:59.984402 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:12:59.984379    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cae409cd-445f-40b9-9851-c89b09c1ac43-kube-api-access-2rfpr podName:cae409cd-445f-40b9-9851-c89b09c1ac43 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:00.484359946 +0000 UTC m=+2.125554737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2rfpr" (UniqueName: "kubernetes.io/projected/cae409cd-445f-40b9-9851-c89b09c1ac43-kube-api-access-2rfpr") pod "network-check-target-zhvk8" (UID: "cae409cd-445f-40b9-9851-c89b09c1ac43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:12:59.985129 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.985077    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt6l8\" (UniqueName: \"kubernetes.io/projected/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-kube-api-access-vt6l8\") pod \"network-metrics-daemon-n9vrb\" (UID: \"02a94135-b85a-4ab1-9dd1-2ae75e77f2de\") " pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:12:59.985228 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.985124    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8qmz\" (UniqueName: \"kubernetes.io/projected/02afa42e-73a7-4f9b-a5d7-609a53a507b1-kube-api-access-m8qmz\") pod \"node-resolver-sg52n\" (UID: \"02afa42e-73a7-4f9b-a5d7-609a53a507b1\") " pod="openshift-dns/node-resolver-sg52n"
May 25 12:12:59.985616 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.985593    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4jlw\" (UniqueName: \"kubernetes.io/projected/69c3d413-5409-4322-8e78-1cfa1d96ed00-kube-api-access-c4jlw\") pod \"tuned-mpgf7\" (UID: \"69c3d413-5409-4322-8e78-1cfa1d96ed00\") " pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:12:59.986111 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:59.986077    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3617d771b965ababd68b956e9ea4412.slice/crio-87edb0106c779d934c11ada016c7796a6e2d8e01c5d65c61079c42258dba624e WatchSource:0}: Error finding container 87edb0106c779d934c11ada016c7796a6e2d8e01c5d65c61079c42258dba624e: Status 404 returned error can't find the container with id 87edb0106c779d934c11ada016c7796a6e2d8e01c5d65c61079c42258dba624e
May 25 12:12:59.986319 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:12:59.986301    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddecdc30bec8216583ece4597ca83be0b.slice/crio-efee27abbb66fc9493c0d360cce41e3a8a3a96a496a4e8eb8b6798423c7d7c9a WatchSource:0}: Error finding container efee27abbb66fc9493c0d360cce41e3a8a3a96a496a4e8eb8b6798423c7d7c9a: Status 404 returned error can't find the container with id efee27abbb66fc9493c0d360cce41e3a8a3a96a496a4e8eb8b6798423c7d7c9a
May 25 12:12:59.993309 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.993285    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:12:59.993746 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:12:59.993725    2578 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:13:00.077722 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.077687    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-multus-conf-dir\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.077722 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.077724    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-host-cni-bin\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.077971 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.077752    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.077971 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.077810    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.077971 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.077811    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-host-cni-bin\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.077971 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.077810    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-multus-conf-dir\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.077971 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.077837    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7687fe26-96a3-43ea-9223-0c5d61cd667e-env-overrides\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.077971 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.077878    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f97986ac-3ba7-4762-9e76-38ea2ac839b6-cnibin\") pod \"multus-additional-cni-plugins-5xg5h\" (UID: \"f97986ac-3ba7-4762-9e76-38ea2ac839b6\") " pod="openshift-multus/multus-additional-cni-plugins-5xg5h"
May 25 12:13:00.077971 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.077894    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-cnibin\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.077971 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.077911    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-host-var-lib-cni-multus\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.077971 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.077928    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f97986ac-3ba7-4762-9e76-38ea2ac839b6-cnibin\") pod \"multus-additional-cni-plugins-5xg5h\" (UID: \"f97986ac-3ba7-4762-9e76-38ea2ac839b6\") " pod="openshift-multus/multus-additional-cni-plugins-5xg5h"
May 25 12:13:00.077971 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.077931    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-host-var-lib-kubelet\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.077971 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.077953    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-host-var-lib-kubelet\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.077971 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.077960    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-llhb7\" (UniqueName: \"kubernetes.io/projected/4fe8af28-e32b-4375-9cca-803603210b1a-kube-api-access-llhb7\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.077971 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.077967    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-cnibin\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.077971 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.077982    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-log-socket\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.078569 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.077987    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-host-var-lib-cni-multus\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.078569 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078007    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-system-cni-dir\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.078569 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078041    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-run-ovn\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.078569 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078073    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-system-cni-dir\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.078569 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078076    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-log-socket\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.078569 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078088    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7687fe26-96a3-43ea-9223-0c5d61cd667e-ovn-node-metrics-cert\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.078569 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078114    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-run-ovn\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.078569 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078118    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a34e5474-3229-4811-8b0e-dee727e7d8b7-host\") pod \"node-ca-j2hhk\" (UID: \"a34e5474-3229-4811-8b0e-dee727e7d8b7\") " pod="openshift-image-registry/node-ca-j2hhk"
May 25 12:13:00.078569 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078143    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-systemd-units\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.078569 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078169    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f97986ac-3ba7-4762-9e76-38ea2ac839b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5xg5h\" (UID: \"f97986ac-3ba7-4762-9e76-38ea2ac839b6\") " pod="openshift-multus/multus-additional-cni-plugins-5xg5h"
May 25 12:13:00.078569 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078186    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a34e5474-3229-4811-8b0e-dee727e7d8b7-host\") pod \"node-ca-j2hhk\" (UID: \"a34e5474-3229-4811-8b0e-dee727e7d8b7\") " pod="openshift-image-registry/node-ca-j2hhk"
May 25 12:13:00.078569 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078193    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-multus-cni-dir\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.078569 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078228    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-systemd-units\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.078569 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078253    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f97986ac-3ba7-4762-9e76-38ea2ac839b6-os-release\") pod \"multus-additional-cni-plugins-5xg5h\" (UID: \"f97986ac-3ba7-4762-9e76-38ea2ac839b6\") " pod="openshift-multus/multus-additional-cni-plugins-5xg5h"
May 25 12:13:00.078569 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078277    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-host-run-netns\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.078569 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078307    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-etc-kubernetes\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.078569 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078254    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-multus-cni-dir\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.078569 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078332    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-etc-openvswitch\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.079444 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078346    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f97986ac-3ba7-4762-9e76-38ea2ac839b6-os-release\") pod \"multus-additional-cni-plugins-5xg5h\" (UID: \"f97986ac-3ba7-4762-9e76-38ea2ac839b6\") " pod="openshift-multus/multus-additional-cni-plugins-5xg5h"
May 25 12:13:00.079444 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078355    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f97986ac-3ba7-4762-9e76-38ea2ac839b6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5xg5h\" (UID: \"f97986ac-3ba7-4762-9e76-38ea2ac839b6\") " pod="openshift-multus/multus-additional-cni-plugins-5xg5h"
May 25 12:13:00.079444 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078370    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-host-run-netns\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.079444 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078375    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-etc-kubernetes\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.079444 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078374    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-etc-openvswitch\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.079444 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078394    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lv8fw\" (UniqueName: \"kubernetes.io/projected/f97986ac-3ba7-4762-9e76-38ea2ac839b6-kube-api-access-lv8fw\") pod \"multus-additional-cni-plugins-5xg5h\" (UID: \"f97986ac-3ba7-4762-9e76-38ea2ac839b6\") " pod="openshift-multus/multus-additional-cni-plugins-5xg5h"
May 25 12:13:00.079444 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078420    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-os-release\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.079444 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078445    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-hostroot\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.079444 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078470    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7687fe26-96a3-43ea-9223-0c5d61cd667e-ovnkube-script-lib\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.079444 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078472    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7687fe26-96a3-43ea-9223-0c5d61cd667e-env-overrides\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.079444 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078493    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a34e5474-3229-4811-8b0e-dee727e7d8b7-serviceca\") pod \"node-ca-j2hhk\" (UID: \"a34e5474-3229-4811-8b0e-dee727e7d8b7\") " pod="openshift-image-registry/node-ca-j2hhk"
May 25 12:13:00.079444 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078504    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-os-release\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.079444 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078518    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-host-kubelet\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.079444 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078527    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-hostroot\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.079444 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078535    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-host-slash\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.079444 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078558    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-host-kubelet\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.079444 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078574    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-host-cni-netd\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.079444 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078587    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-host-slash\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.080023 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078605    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7687fe26-96a3-43ea-9223-0c5d61cd667e-ovnkube-config\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.080023 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078619    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-host-cni-netd\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.080023 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078632    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4fe8af28-e32b-4375-9cca-803603210b1a-multus-daemon-config\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.080023 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078655    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-var-lib-openvswitch\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.080023 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078678    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-host-run-ovn-kubernetes\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.080023 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078704    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f97986ac-3ba7-4762-9e76-38ea2ac839b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-5xg5h\" (UID: \"f97986ac-3ba7-4762-9e76-38ea2ac839b6\") " pod="openshift-multus/multus-additional-cni-plugins-5xg5h"
May 25 12:13:00.080023 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078727    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f97986ac-3ba7-4762-9e76-38ea2ac839b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5xg5h\" (UID: \"f97986ac-3ba7-4762-9e76-38ea2ac839b6\") " pod="openshift-multus/multus-additional-cni-plugins-5xg5h"
May 25 12:13:00.080023 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078750    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-multus-socket-dir-parent\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.080023 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078776    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-host-run-multus-certs\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.080023 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078801    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f97986ac-3ba7-4762-9e76-38ea2ac839b6-system-cni-dir\") pod \"multus-additional-cni-plugins-5xg5h\" (UID: \"f97986ac-3ba7-4762-9e76-38ea2ac839b6\") " pod="openshift-multus/multus-additional-cni-plugins-5xg5h"
May 25 12:13:00.080023 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078827    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4fe8af28-e32b-4375-9cca-803603210b1a-cni-binary-copy\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.080023 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078857    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-host-var-lib-cni-bin\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.080023 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078880    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-host-run-netns\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.080023 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078909    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-host-run-k8s-cni-cncf-io\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.080023 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078933    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-run-openvswitch\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.080023 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078957    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-node-log\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.080023 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078966    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a34e5474-3229-4811-8b0e-dee727e7d8b7-serviceca\") pod \"node-ca-j2hhk\" (UID: \"a34e5474-3229-4811-8b0e-dee727e7d8b7\") " pod="openshift-image-registry/node-ca-j2hhk"
May 25 12:13:00.080556 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.078982    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k9xbb\" (UniqueName: \"kubernetes.io/projected/7687fe26-96a3-43ea-9223-0c5d61cd667e-kube-api-access-k9xbb\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.080556 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.079027    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f97986ac-3ba7-4762-9e76-38ea2ac839b6-system-cni-dir\") pod \"multus-additional-cni-plugins-5xg5h\" (UID: \"f97986ac-3ba7-4762-9e76-38ea2ac839b6\") " pod="openshift-multus/multus-additional-cni-plugins-5xg5h"
May 25 12:13:00.080556 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.079028    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7687fe26-96a3-43ea-9223-0c5d61cd667e-ovnkube-script-lib\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.080556 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.079091    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-host-run-netns\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.080556 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.079117    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7687fe26-96a3-43ea-9223-0c5d61cd667e-ovnkube-config\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.080556 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.079170    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-host-var-lib-cni-bin\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.080556 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.079208    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4fe8af28-e32b-4375-9cca-803603210b1a-multus-daemon-config\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.080556 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.079231    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-var-lib-openvswitch\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.080556 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.079265    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-host-run-multus-certs\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.080556 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.079273    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-host-run-k8s-cni-cncf-io\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.080556 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.079285    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-node-log\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.080556 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.079310    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-run-openvswitch\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.080556 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.079315    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4fe8af28-e32b-4375-9cca-803603210b1a-multus-socket-dir-parent\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.080556 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.079332    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-host-run-ovn-kubernetes\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.080556 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.079343    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f97986ac-3ba7-4762-9e76-38ea2ac839b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5xg5h\" (UID: \"f97986ac-3ba7-4762-9e76-38ea2ac839b6\") " pod="openshift-multus/multus-additional-cni-plugins-5xg5h"
May 25 12:13:00.080556 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.079377    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5p7nc\" (UniqueName: \"kubernetes.io/projected/a34e5474-3229-4811-8b0e-dee727e7d8b7-kube-api-access-5p7nc\") pod \"node-ca-j2hhk\" (UID: \"a34e5474-3229-4811-8b0e-dee727e7d8b7\") " pod="openshift-image-registry/node-ca-j2hhk"
May 25 12:13:00.080556 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.079402    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-run-systemd\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.081100 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.079458    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7687fe26-96a3-43ea-9223-0c5d61cd667e-run-systemd\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.081100 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.079592    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4fe8af28-e32b-4375-9cca-803603210b1a-cni-binary-copy\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.081100 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.079612    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f97986ac-3ba7-4762-9e76-38ea2ac839b6-cni-binary-copy\") pod \"multus-additional-cni-plugins-5xg5h\" (UID: \"f97986ac-3ba7-4762-9e76-38ea2ac839b6\") " pod="openshift-multus/multus-additional-cni-plugins-5xg5h"
May 25 12:13:00.081100 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.079640    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f97986ac-3ba7-4762-9e76-38ea2ac839b6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5xg5h\" (UID: \"f97986ac-3ba7-4762-9e76-38ea2ac839b6\") " pod="openshift-multus/multus-additional-cni-plugins-5xg5h"
May 25 12:13:00.081100 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.079786    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f97986ac-3ba7-4762-9e76-38ea2ac839b6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5xg5h\" (UID: \"f97986ac-3ba7-4762-9e76-38ea2ac839b6\") " pod="openshift-multus/multus-additional-cni-plugins-5xg5h"
May 25 12:13:00.081100 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.080652    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7687fe26-96a3-43ea-9223-0c5d61cd667e-ovn-node-metrics-cert\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.087659 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.087634    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llhb7\" (UniqueName: \"kubernetes.io/projected/4fe8af28-e32b-4375-9cca-803603210b1a-kube-api-access-llhb7\") pod \"multus-zwd5q\" (UID: \"4fe8af28-e32b-4375-9cca-803603210b1a\") " pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.087773 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.087713    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv8fw\" (UniqueName: \"kubernetes.io/projected/f97986ac-3ba7-4762-9e76-38ea2ac839b6-kube-api-access-lv8fw\") pod \"multus-additional-cni-plugins-5xg5h\" (UID: \"f97986ac-3ba7-4762-9e76-38ea2ac839b6\") " pod="openshift-multus/multus-additional-cni-plugins-5xg5h"
May 25 12:13:00.087812 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.087778    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9xbb\" (UniqueName: \"kubernetes.io/projected/7687fe26-96a3-43ea-9223-0c5d61cd667e-kube-api-access-k9xbb\") pod \"ovnkube-node-qt8dc\" (UID: \"7687fe26-96a3-43ea-9223-0c5d61cd667e\") " pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.087812 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.087794    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p7nc\" (UniqueName: \"kubernetes.io/projected/a34e5474-3229-4811-8b0e-dee727e7d8b7-kube-api-access-5p7nc\") pod \"node-ca-j2hhk\" (UID: \"a34e5474-3229-4811-8b0e-dee727e7d8b7\") " pod="openshift-image-registry/node-ca-j2hhk"
May 25 12:13:00.185829 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.185744    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6"
May 25 12:13:00.192375 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:13:00.192341    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a83491_d11a_4002_8df4_66913cf817f6.slice/crio-8705c026d4bfdf3c74e6928d7af20c519c064932c6ecef4e7ac79999e2085bea WatchSource:0}: Error finding container 8705c026d4bfdf3c74e6928d7af20c519c064932c6ecef4e7ac79999e2085bea: Status 404 returned error can't find the container with id 8705c026d4bfdf3c74e6928d7af20c519c064932c6ecef4e7ac79999e2085bea
May 25 12:13:00.201339 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.201317    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mpgf7"
May 25 12:13:00.207550 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:13:00.207520    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c3d413_5409_4322_8e78_1cfa1d96ed00.slice/crio-af6481b012acb2053bdbc41bb68f33fbf6b4e3376f7b1e3bb06412f822501a75 WatchSource:0}: Error finding container af6481b012acb2053bdbc41bb68f33fbf6b4e3376f7b1e3bb06412f822501a75: Status 404 returned error can't find the container with id af6481b012acb2053bdbc41bb68f33fbf6b4e3376f7b1e3bb06412f822501a75
May 25 12:13:00.220110 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.220086    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sg52n"
May 25 12:13:00.226231 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:13:00.226201    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02afa42e_73a7_4f9b_a5d7_609a53a507b1.slice/crio-276f4fd4a6e6fdc562c6c05dabef4444619efdeb4078d10241dc05553ede588e WatchSource:0}: Error finding container 276f4fd4a6e6fdc562c6c05dabef4444619efdeb4078d10241dc05553ede588e: Status 404 returned error can't find the container with id 276f4fd4a6e6fdc562c6c05dabef4444619efdeb4078d10241dc05553ede588e
May 25 12:13:00.240510 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.240470    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qkfd7"
May 25 12:13:00.247396 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:13:00.247370    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ddd277f_8d2e_4f5a_b2f9_788afa7f7e33.slice/crio-4a1f3b6f75d5afadd758a2109965c4dc14219b953f5e077a52e2df4acb579d61 WatchSource:0}: Error finding container 4a1f3b6f75d5afadd758a2109965c4dc14219b953f5e077a52e2df4acb579d61: Status 404 returned error can't find the container with id 4a1f3b6f75d5afadd758a2109965c4dc14219b953f5e077a52e2df4acb579d61
May 25 12:13:00.263400 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.263368    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2fqnj"
May 25 12:13:00.271156 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:13:00.271125    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b74961_b200_4de0_9d76_e43923f19633.slice/crio-a3916ba853bb7faa60b6e9a70cf3052f9e5b08b6a9fb8e449d934af5bfac37f9 WatchSource:0}: Error finding container a3916ba853bb7faa60b6e9a70cf3052f9e5b08b6a9fb8e449d934af5bfac37f9: Status 404 returned error can't find the container with id a3916ba853bb7faa60b6e9a70cf3052f9e5b08b6a9fb8e449d934af5bfac37f9
May 25 12:13:00.287923 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.287888    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j2hhk"
May 25 12:13:00.294113 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:13:00.294087    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda34e5474_3229_4811_8b0e_dee727e7d8b7.slice/crio-b883ce8acf6f5271bca889d8c54fec1454dfc4a3bc64feaa9b75a3b015a33bdf WatchSource:0}: Error finding container b883ce8acf6f5271bca889d8c54fec1454dfc4a3bc64feaa9b75a3b015a33bdf: Status 404 returned error can't find the container with id b883ce8acf6f5271bca889d8c54fec1454dfc4a3bc64feaa9b75a3b015a33bdf
May 25 12:13:00.295004 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.294984    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5xg5h"
May 25 12:13:00.301113 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.301092    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zwd5q"
May 25 12:13:00.301198 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:13:00.301176    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf97986ac_3ba7_4762_9e76_38ea2ac839b6.slice/crio-4c8a0618b09c42580b28c846d6c069fdc32d3c2c382dbd9dd809b16fb853bd93 WatchSource:0}: Error finding container 4c8a0618b09c42580b28c846d6c069fdc32d3c2c382dbd9dd809b16fb853bd93: Status 404 returned error can't find the container with id 4c8a0618b09c42580b28c846d6c069fdc32d3c2c382dbd9dd809b16fb853bd93
May 25 12:13:00.306378 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.306361    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:00.306980 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:13:00.306963    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe8af28_e32b_4375_9cca_803603210b1a.slice/crio-51c02e0f9c7ab52fe156475820307d5edac6d4364b52afc7ac47fb7957c9fe14 WatchSource:0}: Error finding container 51c02e0f9c7ab52fe156475820307d5edac6d4364b52afc7ac47fb7957c9fe14: Status 404 returned error can't find the container with id 51c02e0f9c7ab52fe156475820307d5edac6d4364b52afc7ac47fb7957c9fe14
May 25 12:13:00.314090 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:13:00.313879    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7687fe26_96a3_43ea_9223_0c5d61cd667e.slice/crio-256ffc216dedd06b52182d85a42989c031fb0af8cc88de769186f2d449b1ee51 WatchSource:0}: Error finding container 256ffc216dedd06b52182d85a42989c031fb0af8cc88de769186f2d449b1ee51: Status 404 returned error can't find the container with id 256ffc216dedd06b52182d85a42989c031fb0af8cc88de769186f2d449b1ee51
May 25 12:13:00.482115 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.482018    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-metrics-certs\") pod \"network-metrics-daemon-n9vrb\" (UID: \"02a94135-b85a-4ab1-9dd1-2ae75e77f2de\") " pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:13:00.482274 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:00.482193    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:00.482274 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:00.482253    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-metrics-certs podName:02a94135-b85a-4ab1-9dd1-2ae75e77f2de nodeName:}" failed. No retries permitted until 2026-05-25 12:13:01.48223503 +0000 UTC m=+3.123429789 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-metrics-certs") pod "network-metrics-daemon-n9vrb" (UID: "02a94135-b85a-4ab1-9dd1-2ae75e77f2de") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:00.583287 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.583250    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2rfpr\" (UniqueName: \"kubernetes.io/projected/cae409cd-445f-40b9-9851-c89b09c1ac43-kube-api-access-2rfpr\") pod \"network-check-target-zhvk8\" (UID: \"cae409cd-445f-40b9-9851-c89b09c1ac43\") " pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:13:00.583470 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:00.583451    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:13:00.583470 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:00.583468    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:13:00.583573 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:00.583480    2578 projected.go:194] Error preparing data for projected volume kube-api-access-2rfpr for pod openshift-network-diagnostics/network-check-target-zhvk8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:00.583573 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:00.583538    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cae409cd-445f-40b9-9851-c89b09c1ac43-kube-api-access-2rfpr podName:cae409cd-445f-40b9-9851-c89b09c1ac43 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:01.583518906 +0000 UTC m=+3.224713666 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2rfpr" (UniqueName: "kubernetes.io/projected/cae409cd-445f-40b9-9851-c89b09c1ac43-kube-api-access-2rfpr") pod "network-check-target-zhvk8" (UID: "cae409cd-445f-40b9-9851-c89b09c1ac43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:00.707105 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.707073    2578 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:13:00.912307 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.912266    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 12:07:59 +0000 UTC" deadline="2028-02-14 05:25:14.947019818 +0000 UTC"
May 25 12:13:00.912307 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.912305    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15113h12m14.034719114s"
May 25 12:13:00.938742 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.938226    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:13:00.938742 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:00.938362    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vrb" podUID="02a94135-b85a-4ab1-9dd1-2ae75e77f2de"
May 25 12:13:00.962599 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.962493    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mpgf7" event={"ID":"69c3d413-5409-4322-8e78-1cfa1d96ed00","Type":"ContainerStarted","Data":"af6481b012acb2053bdbc41bb68f33fbf6b4e3376f7b1e3bb06412f822501a75"}
May 25 12:13:00.967463 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.967427    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6" event={"ID":"48a83491-d11a-4002-8df4-66913cf817f6","Type":"ContainerStarted","Data":"8705c026d4bfdf3c74e6928d7af20c519c064932c6ecef4e7ac79999e2085bea"}
May 25 12:13:00.987988 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.987943    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-107.ec2.internal" event={"ID":"decdc30bec8216583ece4597ca83be0b","Type":"ContainerStarted","Data":"efee27abbb66fc9493c0d360cce41e3a8a3a96a496a4e8eb8b6798423c7d7c9a"}
May 25 12:13:00.995644 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:00.995607    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc" event={"ID":"7687fe26-96a3-43ea-9223-0c5d61cd667e","Type":"ContainerStarted","Data":"256ffc216dedd06b52182d85a42989c031fb0af8cc88de769186f2d449b1ee51"}
May 25 12:13:01.008298 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:01.008260    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j2hhk" event={"ID":"a34e5474-3229-4811-8b0e-dee727e7d8b7","Type":"ContainerStarted","Data":"b883ce8acf6f5271bca889d8c54fec1454dfc4a3bc64feaa9b75a3b015a33bdf"}
May 25 12:13:01.012587 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:01.012543    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2fqnj" event={"ID":"01b74961-b200-4de0-9d76-e43923f19633","Type":"ContainerStarted","Data":"a3916ba853bb7faa60b6e9a70cf3052f9e5b08b6a9fb8e449d934af5bfac37f9"}
May 25 12:13:01.023488 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:01.023451    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-107.ec2.internal" event={"ID":"a3617d771b965ababd68b956e9ea4412","Type":"ContainerStarted","Data":"87edb0106c779d934c11ada016c7796a6e2d8e01c5d65c61079c42258dba624e"}
May 25 12:13:01.049484 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:01.049443    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zwd5q" event={"ID":"4fe8af28-e32b-4375-9cca-803603210b1a","Type":"ContainerStarted","Data":"51c02e0f9c7ab52fe156475820307d5edac6d4364b52afc7ac47fb7957c9fe14"}
May 25 12:13:01.060984 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:01.060941    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xg5h" event={"ID":"f97986ac-3ba7-4762-9e76-38ea2ac839b6","Type":"ContainerStarted","Data":"4c8a0618b09c42580b28c846d6c069fdc32d3c2c382dbd9dd809b16fb853bd93"}
May 25 12:13:01.073382 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:01.073342    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qkfd7" event={"ID":"6ddd277f-8d2e-4f5a-b2f9-788afa7f7e33","Type":"ContainerStarted","Data":"4a1f3b6f75d5afadd758a2109965c4dc14219b953f5e077a52e2df4acb579d61"}
May 25 12:13:01.085723 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:01.085687    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sg52n" event={"ID":"02afa42e-73a7-4f9b-a5d7-609a53a507b1","Type":"ContainerStarted","Data":"276f4fd4a6e6fdc562c6c05dabef4444619efdeb4078d10241dc05553ede588e"}
May 25 12:13:01.492956 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:01.492882    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-metrics-certs\") pod \"network-metrics-daemon-n9vrb\" (UID: \"02a94135-b85a-4ab1-9dd1-2ae75e77f2de\") " pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:13:01.493160 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:01.493105    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:01.493233 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:01.493171    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-metrics-certs podName:02a94135-b85a-4ab1-9dd1-2ae75e77f2de nodeName:}" failed. No retries permitted until 2026-05-25 12:13:03.493151629 +0000 UTC m=+5.134346387 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-metrics-certs") pod "network-metrics-daemon-n9vrb" (UID: "02a94135-b85a-4ab1-9dd1-2ae75e77f2de") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:01.593735 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:01.593604    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2rfpr\" (UniqueName: \"kubernetes.io/projected/cae409cd-445f-40b9-9851-c89b09c1ac43-kube-api-access-2rfpr\") pod \"network-check-target-zhvk8\" (UID: \"cae409cd-445f-40b9-9851-c89b09c1ac43\") " pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:13:01.593896 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:01.593827    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:13:01.593896 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:01.593849    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:13:01.593896 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:01.593862    2578 projected.go:194] Error preparing data for projected volume kube-api-access-2rfpr for pod openshift-network-diagnostics/network-check-target-zhvk8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:01.594049 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:01.593923    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cae409cd-445f-40b9-9851-c89b09c1ac43-kube-api-access-2rfpr podName:cae409cd-445f-40b9-9851-c89b09c1ac43 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:03.593903276 +0000 UTC m=+5.235098036 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2rfpr" (UniqueName: "kubernetes.io/projected/cae409cd-445f-40b9-9851-c89b09c1ac43-kube-api-access-2rfpr") pod "network-check-target-zhvk8" (UID: "cae409cd-445f-40b9-9851-c89b09c1ac43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:01.913036 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:01.912939    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 12:07:59 +0000 UTC" deadline="2027-12-05 15:26:42.774953416 +0000 UTC"
May 25 12:13:01.913036 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:01.912982    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13419h13m40.861974403s"
May 25 12:13:01.936081 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:01.935523    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:13:01.936081 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:01.935667    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhvk8" podUID="cae409cd-445f-40b9-9851-c89b09c1ac43"
May 25 12:13:02.028788 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:02.028758    2578 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:13:02.101172 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:02.101131    2578 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 12:13:02.937889 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:02.937859    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:13:02.938356 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:02.937986    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vrb" podUID="02a94135-b85a-4ab1-9dd1-2ae75e77f2de"
May 25 12:13:03.511926 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:03.511891    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-metrics-certs\") pod \"network-metrics-daemon-n9vrb\" (UID: \"02a94135-b85a-4ab1-9dd1-2ae75e77f2de\") " pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:13:03.512155 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:03.512077    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:03.512225 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:03.512162    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-metrics-certs podName:02a94135-b85a-4ab1-9dd1-2ae75e77f2de nodeName:}" failed. No retries permitted until 2026-05-25 12:13:07.512140724 +0000 UTC m=+9.153335484 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-metrics-certs") pod "network-metrics-daemon-n9vrb" (UID: "02a94135-b85a-4ab1-9dd1-2ae75e77f2de") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:03.613913 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:03.613297    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2rfpr\" (UniqueName: \"kubernetes.io/projected/cae409cd-445f-40b9-9851-c89b09c1ac43-kube-api-access-2rfpr\") pod \"network-check-target-zhvk8\" (UID: \"cae409cd-445f-40b9-9851-c89b09c1ac43\") " pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:13:03.613913 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:03.613484    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:13:03.613913 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:03.613505    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:13:03.613913 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:03.613517    2578 projected.go:194] Error preparing data for projected volume kube-api-access-2rfpr for pod openshift-network-diagnostics/network-check-target-zhvk8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:03.613913 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:03.613577    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cae409cd-445f-40b9-9851-c89b09c1ac43-kube-api-access-2rfpr podName:cae409cd-445f-40b9-9851-c89b09c1ac43 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:07.613558555 +0000 UTC m=+9.254753323 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2rfpr" (UniqueName: "kubernetes.io/projected/cae409cd-445f-40b9-9851-c89b09c1ac43-kube-api-access-2rfpr") pod "network-check-target-zhvk8" (UID: "cae409cd-445f-40b9-9851-c89b09c1ac43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:03.935174 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:03.935142    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:13:03.935353 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:03.935274    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhvk8" podUID="cae409cd-445f-40b9-9851-c89b09c1ac43"
May 25 12:13:04.935217 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:04.935186    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:13:04.935652 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:04.935337    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vrb" podUID="02a94135-b85a-4ab1-9dd1-2ae75e77f2de"
May 25 12:13:05.935461 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:05.935426    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:13:05.935922 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:05.935572    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhvk8" podUID="cae409cd-445f-40b9-9851-c89b09c1ac43"
May 25 12:13:06.935935 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:06.935900    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:13:06.936425 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:06.936079    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vrb" podUID="02a94135-b85a-4ab1-9dd1-2ae75e77f2de"
May 25 12:13:07.548656 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:07.548618    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-metrics-certs\") pod \"network-metrics-daemon-n9vrb\" (UID: \"02a94135-b85a-4ab1-9dd1-2ae75e77f2de\") " pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:13:07.548842 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:07.548798    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:07.548920 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:07.548884    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-metrics-certs podName:02a94135-b85a-4ab1-9dd1-2ae75e77f2de nodeName:}" failed. No retries permitted until 2026-05-25 12:13:15.548861001 +0000 UTC m=+17.190055770 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-metrics-certs") pod "network-metrics-daemon-n9vrb" (UID: "02a94135-b85a-4ab1-9dd1-2ae75e77f2de") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:07.650217 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:07.649593    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2rfpr\" (UniqueName: \"kubernetes.io/projected/cae409cd-445f-40b9-9851-c89b09c1ac43-kube-api-access-2rfpr\") pod \"network-check-target-zhvk8\" (UID: \"cae409cd-445f-40b9-9851-c89b09c1ac43\") " pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:13:07.650217 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:07.649778    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:13:07.650217 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:07.649797    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:13:07.650217 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:07.649809    2578 projected.go:194] Error preparing data for projected volume kube-api-access-2rfpr for pod openshift-network-diagnostics/network-check-target-zhvk8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:07.650217 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:07.649867    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cae409cd-445f-40b9-9851-c89b09c1ac43-kube-api-access-2rfpr podName:cae409cd-445f-40b9-9851-c89b09c1ac43 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:15.649848853 +0000 UTC m=+17.291043614 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2rfpr" (UniqueName: "kubernetes.io/projected/cae409cd-445f-40b9-9851-c89b09c1ac43-kube-api-access-2rfpr") pod "network-check-target-zhvk8" (UID: "cae409cd-445f-40b9-9851-c89b09c1ac43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:07.935957 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:07.935872    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:13:07.936341 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:07.936023    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhvk8" podUID="cae409cd-445f-40b9-9851-c89b09c1ac43"
May 25 12:13:08.935787 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:08.935747    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:13:08.935977 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:08.935865    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vrb" podUID="02a94135-b85a-4ab1-9dd1-2ae75e77f2de"
May 25 12:13:09.935037 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:09.935009    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:13:09.935192 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:09.935119    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhvk8" podUID="cae409cd-445f-40b9-9851-c89b09c1ac43"
May 25 12:13:10.600075 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:10.600023    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-8zd5f"]
May 25 12:13:10.616998 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:10.616966    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:10.617195 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:10.617051    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zd5f" podUID="131acddf-521f-481d-84a3-b4e9ce65b58e"
May 25 12:13:10.671910 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:10.671872    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/131acddf-521f-481d-84a3-b4e9ce65b58e-kubelet-config\") pod \"global-pull-secret-syncer-8zd5f\" (UID: \"131acddf-521f-481d-84a3-b4e9ce65b58e\") " pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:10.672068 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:10.671976    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/131acddf-521f-481d-84a3-b4e9ce65b58e-original-pull-secret\") pod \"global-pull-secret-syncer-8zd5f\" (UID: \"131acddf-521f-481d-84a3-b4e9ce65b58e\") " pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:10.672068 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:10.672021    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/131acddf-521f-481d-84a3-b4e9ce65b58e-dbus\") pod \"global-pull-secret-syncer-8zd5f\" (UID: \"131acddf-521f-481d-84a3-b4e9ce65b58e\") " pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:10.773265 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:10.773230    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/131acddf-521f-481d-84a3-b4e9ce65b58e-dbus\") pod \"global-pull-secret-syncer-8zd5f\" (UID: \"131acddf-521f-481d-84a3-b4e9ce65b58e\") " pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:10.773265 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:10.773275    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/131acddf-521f-481d-84a3-b4e9ce65b58e-kubelet-config\") pod \"global-pull-secret-syncer-8zd5f\" (UID: \"131acddf-521f-481d-84a3-b4e9ce65b58e\") " pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:10.773481 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:10.773373    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/131acddf-521f-481d-84a3-b4e9ce65b58e-kubelet-config\") pod \"global-pull-secret-syncer-8zd5f\" (UID: \"131acddf-521f-481d-84a3-b4e9ce65b58e\") " pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:10.773481 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:10.773449    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/131acddf-521f-481d-84a3-b4e9ce65b58e-dbus\") pod \"global-pull-secret-syncer-8zd5f\" (UID: \"131acddf-521f-481d-84a3-b4e9ce65b58e\") " pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:10.773556 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:10.773477    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/131acddf-521f-481d-84a3-b4e9ce65b58e-original-pull-secret\") pod \"global-pull-secret-syncer-8zd5f\" (UID: \"131acddf-521f-481d-84a3-b4e9ce65b58e\") " pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:10.773625 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:10.773609    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 12:13:10.773699 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:10.773688    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/131acddf-521f-481d-84a3-b4e9ce65b58e-original-pull-secret podName:131acddf-521f-481d-84a3-b4e9ce65b58e nodeName:}" failed. No retries permitted until 2026-05-25 12:13:11.273669532 +0000 UTC m=+12.914864291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/131acddf-521f-481d-84a3-b4e9ce65b58e-original-pull-secret") pod "global-pull-secret-syncer-8zd5f" (UID: "131acddf-521f-481d-84a3-b4e9ce65b58e") : object "kube-system"/"original-pull-secret" not registered
May 25 12:13:10.935000 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:10.934924    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:13:10.935169 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:10.935048    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vrb" podUID="02a94135-b85a-4ab1-9dd1-2ae75e77f2de"
May 25 12:13:11.277390 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:11.277311    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/131acddf-521f-481d-84a3-b4e9ce65b58e-original-pull-secret\") pod \"global-pull-secret-syncer-8zd5f\" (UID: \"131acddf-521f-481d-84a3-b4e9ce65b58e\") " pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:11.277544 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:11.277428    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 12:13:11.277544 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:11.277500    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/131acddf-521f-481d-84a3-b4e9ce65b58e-original-pull-secret podName:131acddf-521f-481d-84a3-b4e9ce65b58e nodeName:}" failed. No retries permitted until 2026-05-25 12:13:12.277480677 +0000 UTC m=+13.918675435 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/131acddf-521f-481d-84a3-b4e9ce65b58e-original-pull-secret") pod "global-pull-secret-syncer-8zd5f" (UID: "131acddf-521f-481d-84a3-b4e9ce65b58e") : object "kube-system"/"original-pull-secret" not registered
May 25 12:13:11.935100 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:11.935045    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:13:11.935503 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:11.935045    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:11.935503 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:11.935175    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhvk8" podUID="cae409cd-445f-40b9-9851-c89b09c1ac43"
May 25 12:13:11.935503 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:11.935301    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zd5f" podUID="131acddf-521f-481d-84a3-b4e9ce65b58e"
May 25 12:13:12.284187 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:12.284095    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/131acddf-521f-481d-84a3-b4e9ce65b58e-original-pull-secret\") pod \"global-pull-secret-syncer-8zd5f\" (UID: \"131acddf-521f-481d-84a3-b4e9ce65b58e\") " pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:12.284362 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:12.284220    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 12:13:12.284362 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:12.284282    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/131acddf-521f-481d-84a3-b4e9ce65b58e-original-pull-secret podName:131acddf-521f-481d-84a3-b4e9ce65b58e nodeName:}" failed. No retries permitted until 2026-05-25 12:13:14.284267257 +0000 UTC m=+15.925462012 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/131acddf-521f-481d-84a3-b4e9ce65b58e-original-pull-secret") pod "global-pull-secret-syncer-8zd5f" (UID: "131acddf-521f-481d-84a3-b4e9ce65b58e") : object "kube-system"/"original-pull-secret" not registered
May 25 12:13:12.935563 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:12.935525    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:13:12.936089 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:12.935663    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vrb" podUID="02a94135-b85a-4ab1-9dd1-2ae75e77f2de"
May 25 12:13:13.934897 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:13.934859    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:13.935117 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:13.934859    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:13:13.935117 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:13.934985    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zd5f" podUID="131acddf-521f-481d-84a3-b4e9ce65b58e"
May 25 12:13:13.935117 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:13.935095    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhvk8" podUID="cae409cd-445f-40b9-9851-c89b09c1ac43"
May 25 12:13:14.296384 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:14.296352    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/131acddf-521f-481d-84a3-b4e9ce65b58e-original-pull-secret\") pod \"global-pull-secret-syncer-8zd5f\" (UID: \"131acddf-521f-481d-84a3-b4e9ce65b58e\") " pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:14.296793 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:14.296517    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 12:13:14.296793 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:14.296595    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/131acddf-521f-481d-84a3-b4e9ce65b58e-original-pull-secret podName:131acddf-521f-481d-84a3-b4e9ce65b58e nodeName:}" failed. No retries permitted until 2026-05-25 12:13:18.296574415 +0000 UTC m=+19.937769175 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/131acddf-521f-481d-84a3-b4e9ce65b58e-original-pull-secret") pod "global-pull-secret-syncer-8zd5f" (UID: "131acddf-521f-481d-84a3-b4e9ce65b58e") : object "kube-system"/"original-pull-secret" not registered
May 25 12:13:14.935917 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:14.935880    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:13:14.936115 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:14.936024    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vrb" podUID="02a94135-b85a-4ab1-9dd1-2ae75e77f2de"
May 25 12:13:15.609044 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:15.609006    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-metrics-certs\") pod \"network-metrics-daemon-n9vrb\" (UID: \"02a94135-b85a-4ab1-9dd1-2ae75e77f2de\") " pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:13:15.609566 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:15.609204    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:15.609566 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:15.609301    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-metrics-certs podName:02a94135-b85a-4ab1-9dd1-2ae75e77f2de nodeName:}" failed. No retries permitted until 2026-05-25 12:13:31.609278362 +0000 UTC m=+33.250473132 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-metrics-certs") pod "network-metrics-daemon-n9vrb" (UID: "02a94135-b85a-4ab1-9dd1-2ae75e77f2de") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:15.709397 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:15.709361    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2rfpr\" (UniqueName: \"kubernetes.io/projected/cae409cd-445f-40b9-9851-c89b09c1ac43-kube-api-access-2rfpr\") pod \"network-check-target-zhvk8\" (UID: \"cae409cd-445f-40b9-9851-c89b09c1ac43\") " pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:13:15.709574 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:15.709510    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:13:15.709574 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:15.709529    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:13:15.709574 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:15.709567    2578 projected.go:194] Error preparing data for projected volume kube-api-access-2rfpr for pod openshift-network-diagnostics/network-check-target-zhvk8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:15.709691 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:15.709634    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cae409cd-445f-40b9-9851-c89b09c1ac43-kube-api-access-2rfpr podName:cae409cd-445f-40b9-9851-c89b09c1ac43 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:31.709612845 +0000 UTC m=+33.350807602 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2rfpr" (UniqueName: "kubernetes.io/projected/cae409cd-445f-40b9-9851-c89b09c1ac43-kube-api-access-2rfpr") pod "network-check-target-zhvk8" (UID: "cae409cd-445f-40b9-9851-c89b09c1ac43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:15.935583 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:15.935507    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:13:15.935790 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:15.935507    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:15.935790 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:15.935621    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhvk8" podUID="cae409cd-445f-40b9-9851-c89b09c1ac43"
May 25 12:13:15.935790 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:15.935727    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zd5f" podUID="131acddf-521f-481d-84a3-b4e9ce65b58e"
May 25 12:13:16.935885 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:16.935848    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:13:16.936438 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:16.935979    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vrb" podUID="02a94135-b85a-4ab1-9dd1-2ae75e77f2de"
May 25 12:13:17.935542 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:17.935507    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:13:17.935712 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:17.935508    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:17.935712 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:17.935613    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhvk8" podUID="cae409cd-445f-40b9-9851-c89b09c1ac43"
May 25 12:13:17.935799 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:17.935709    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zd5f" podUID="131acddf-521f-481d-84a3-b4e9ce65b58e"
May 25 12:13:18.327171 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:18.327137    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/131acddf-521f-481d-84a3-b4e9ce65b58e-original-pull-secret\") pod \"global-pull-secret-syncer-8zd5f\" (UID: \"131acddf-521f-481d-84a3-b4e9ce65b58e\") " pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:18.327522 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:18.327300    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 12:13:18.327522 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:18.327368    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/131acddf-521f-481d-84a3-b4e9ce65b58e-original-pull-secret podName:131acddf-521f-481d-84a3-b4e9ce65b58e nodeName:}" failed. No retries permitted until 2026-05-25 12:13:26.327353913 +0000 UTC m=+27.968548668 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/131acddf-521f-481d-84a3-b4e9ce65b58e-original-pull-secret") pod "global-pull-secret-syncer-8zd5f" (UID: "131acddf-521f-481d-84a3-b4e9ce65b58e") : object "kube-system"/"original-pull-secret" not registered
May 25 12:13:18.943825 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:18.943809    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:13:18.943916 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:18.943900    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vrb" podUID="02a94135-b85a-4ab1-9dd1-2ae75e77f2de"
May 25 12:13:19.123828 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:19.123796    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mpgf7" event={"ID":"69c3d413-5409-4322-8e78-1cfa1d96ed00","Type":"ContainerStarted","Data":"7d397b785381beb1297e9ed294439fdee05985fde3697066021eace43cc90f01"}
May 25 12:13:19.125020 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:19.124996    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc" event={"ID":"7687fe26-96a3-43ea-9223-0c5d61cd667e","Type":"ContainerStarted","Data":"5be7c3773df7c733b683dc32aa1c8810dbf7b55d95cd9e8c83ee248c593dc05b"}
May 25 12:13:19.126231 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:19.126207    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-107.ec2.internal" event={"ID":"a3617d771b965ababd68b956e9ea4412","Type":"ContainerStarted","Data":"efed5d4b81ea40f49ea659702a5d09b29a6ab41f5e985d5a17bbfe6b8ec6ca39"}
May 25 12:13:19.144086 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:19.144012    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mpgf7" podStartSLOduration=1.523585364 podStartE2EDuration="20.143995396s" podCreationTimestamp="2026-05-25 12:12:59 +0000 UTC" firstStartedPulling="2026-05-25 12:13:00.209049263 +0000 UTC m=+1.850244017" lastFinishedPulling="2026-05-25 12:13:18.829459283 +0000 UTC m=+20.470654049" observedRunningTime="2026-05-25 12:13:19.14276177 +0000 UTC m=+20.783956547" watchObservedRunningTime="2026-05-25 12:13:19.143995396 +0000 UTC m=+20.785190174"
May 25 12:13:19.159430 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:19.159374    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-107.ec2.internal" podStartSLOduration=20.159362177 podStartE2EDuration="20.159362177s" podCreationTimestamp="2026-05-25 12:12:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:13:19.15885742 +0000 UTC m=+20.800052198" watchObservedRunningTime="2026-05-25 12:13:19.159362177 +0000 UTC m=+20.800556954"
May 25 12:13:19.935695 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:19.935658    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:19.936404 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:19.935658    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:13:19.936404 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:19.935771    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zd5f" podUID="131acddf-521f-481d-84a3-b4e9ce65b58e"
May 25 12:13:19.936404 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:19.935829    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhvk8" podUID="cae409cd-445f-40b9-9851-c89b09c1ac43"
May 25 12:13:20.129813 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:20.129737    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zwd5q" event={"ID":"4fe8af28-e32b-4375-9cca-803603210b1a","Type":"ContainerStarted","Data":"6a7fe781361636b2ba2558b80f3e18aa5a57925c09eea751c15b42a8d95c5831"}
May 25 12:13:20.131249 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:20.131217    2578 generic.go:358] "Generic (PLEG): container finished" podID="f97986ac-3ba7-4762-9e76-38ea2ac839b6" containerID="daf9f30fee9563a7138d66c33612ed43bc4633f49a9258d2c1d8fd8e0fb4959b" exitCode=0
May 25 12:13:20.131393 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:20.131304    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xg5h" event={"ID":"f97986ac-3ba7-4762-9e76-38ea2ac839b6","Type":"ContainerDied","Data":"daf9f30fee9563a7138d66c33612ed43bc4633f49a9258d2c1d8fd8e0fb4959b"}
May 25 12:13:20.132897 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:20.132608    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sg52n" event={"ID":"02afa42e-73a7-4f9b-a5d7-609a53a507b1","Type":"ContainerStarted","Data":"ad4b58ec98605896264a7853b1f595844e817b5d0b1d6b7259ad2f7d9bfb2a28"}
May 25 12:13:20.134233 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:20.134211    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6" event={"ID":"48a83491-d11a-4002-8df4-66913cf817f6","Type":"ContainerStarted","Data":"6d1568d72f24636700b37fe6fef1c0e462c3382bc434cdb42c12ab0579e02289"}
May 25 12:13:20.135697 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:20.135674    2578 generic.go:358] "Generic (PLEG): container finished" podID="decdc30bec8216583ece4597ca83be0b" containerID="a9d6a747be53c335a2ba1a7c26f41bdce56f284ae641cdeab3395db80e6ffcff" exitCode=0
May 25 12:13:20.135800 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:20.135702    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-107.ec2.internal" event={"ID":"decdc30bec8216583ece4597ca83be0b","Type":"ContainerDied","Data":"a9d6a747be53c335a2ba1a7c26f41bdce56f284ae641cdeab3395db80e6ffcff"}
May 25 12:13:20.138629 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:20.138614    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/ovn-acl-logging/0.log"
May 25 12:13:20.138934 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:20.138912    2578 generic.go:358] "Generic (PLEG): container finished" podID="7687fe26-96a3-43ea-9223-0c5d61cd667e" containerID="d15c4c452e037e5c72f60c995e9f1bbfde8b6720a7410f933a5968e9066bb7e5" exitCode=1
May 25 12:13:20.139042 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:20.138969    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc" event={"ID":"7687fe26-96a3-43ea-9223-0c5d61cd667e","Type":"ContainerStarted","Data":"895fd4692a5a6289a378d6ca9e09c804175240aec743b589e6bee335b4a77942"}
May 25 12:13:20.139042 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:20.138995    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc" event={"ID":"7687fe26-96a3-43ea-9223-0c5d61cd667e","Type":"ContainerStarted","Data":"777c7d6612db499af24832fb00a3f1a4d20f11c4cd2b119463092ddec627f0dc"}
May 25 12:13:20.139042 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:20.139011    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc" event={"ID":"7687fe26-96a3-43ea-9223-0c5d61cd667e","Type":"ContainerStarted","Data":"8ab049afb7d7471e6a2bf1ec2b36ee796532206dd36a2c79aa9ae813a218cabb"}
May 25 12:13:20.139042 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:20.139024    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc" event={"ID":"7687fe26-96a3-43ea-9223-0c5d61cd667e","Type":"ContainerStarted","Data":"b126149638713e0f03b94f48ac90d13c9d3683e1cbe1c6c5f1770582cb02e1c4"}
May 25 12:13:20.139042 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:20.139036    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc" event={"ID":"7687fe26-96a3-43ea-9223-0c5d61cd667e","Type":"ContainerDied","Data":"d15c4c452e037e5c72f60c995e9f1bbfde8b6720a7410f933a5968e9066bb7e5"}
May 25 12:13:20.140322 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:20.140298    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j2hhk" event={"ID":"a34e5474-3229-4811-8b0e-dee727e7d8b7","Type":"ContainerStarted","Data":"b8e50a4cffc5a39dc62a4c3415ab9d513ffbd6b1f6108b0b3faed1b2917bd1c2"}
May 25 12:13:20.141775 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:20.141741    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2fqnj" event={"ID":"01b74961-b200-4de0-9d76-e43923f19633","Type":"ContainerStarted","Data":"8a2b6654f267a46c30382bfc4c2a1f7f1bb4df87696c5118ba5920b481929330"}
May 25 12:13:20.149002 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:20.148960    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-zwd5q" podStartSLOduration=2.288957421 podStartE2EDuration="21.148948105s" podCreationTimestamp="2026-05-25 12:12:59 +0000 UTC" firstStartedPulling="2026-05-25 12:13:00.309572733 +0000 UTC m=+1.950767488" lastFinishedPulling="2026-05-25 12:13:19.169563403 +0000 UTC m=+20.810758172" observedRunningTime="2026-05-25 12:13:20.148500874 +0000 UTC m=+21.789695651" watchObservedRunningTime="2026-05-25 12:13:20.148948105 +0000 UTC m=+21.790142883"
May 25 12:13:20.200889 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:20.200789    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-sg52n" podStartSLOduration=2.600645303 podStartE2EDuration="21.200771622s" podCreationTimestamp="2026-05-25 12:12:59 +0000 UTC" firstStartedPulling="2026-05-25 12:13:00.22769139 +0000 UTC m=+1.868886145" lastFinishedPulling="2026-05-25 12:13:18.827817709 +0000 UTC m=+20.469012464" observedRunningTime="2026-05-25 12:13:20.183240216 +0000 UTC m=+21.824434994" watchObservedRunningTime="2026-05-25 12:13:20.200771622 +0000 UTC m=+21.841966401"
May 25 12:13:20.201159 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:20.201121    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2fqnj" podStartSLOduration=2.692583832 podStartE2EDuration="21.201111864s" podCreationTimestamp="2026-05-25 12:12:59 +0000 UTC" firstStartedPulling="2026-05-25 12:13:00.272625717 +0000 UTC m=+1.913820471" lastFinishedPulling="2026-05-25 12:13:18.78115373 +0000 UTC m=+20.422348503" observedRunningTime="2026-05-25 12:13:20.200052183 +0000 UTC m=+21.841246959" watchObservedRunningTime="2026-05-25 12:13:20.201111864 +0000 UTC m=+21.842306643"
May 25 12:13:20.220748 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:20.220695    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-j2hhk" podStartSLOduration=2.735360832 podStartE2EDuration="21.220679949s" podCreationTimestamp="2026-05-25 12:12:59 +0000 UTC" firstStartedPulling="2026-05-25 12:13:00.295849663 +0000 UTC m=+1.937044419" lastFinishedPulling="2026-05-25 12:13:18.781168779 +0000 UTC m=+20.422363536" observedRunningTime="2026-05-25 12:13:20.220673549 +0000 UTC m=+21.861868327" watchObservedRunningTime="2026-05-25 12:13:20.220679949 +0000 UTC m=+21.861874725"
May 25 12:13:20.918026 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:20.917996    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 25 12:13:20.938706 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:20.938684    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:13:20.939156 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:20.938815    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vrb" podUID="02a94135-b85a-4ab1-9dd1-2ae75e77f2de"
May 25 12:13:20.943605 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:20.943521    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-25T12:13:20.918020357Z","UUID":"ec14cb48-392b-4282-b9f3-a2bd7919eacd","Handler":null,"Name":"","Endpoint":""}
May 25 12:13:20.945485 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:20.945440    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 25 12:13:20.945485 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:20.945469    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 25 12:13:21.145738 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:21.145631    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qkfd7" event={"ID":"6ddd277f-8d2e-4f5a-b2f9-788afa7f7e33","Type":"ContainerStarted","Data":"f84baf189c71b5c5adf9135a43ec6c8adee3ba0c016803b0dfb96a999013cff8"}
May 25 12:13:21.147464 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:21.147433    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6" event={"ID":"48a83491-d11a-4002-8df4-66913cf817f6","Type":"ContainerStarted","Data":"8319067bed33132c5f4f72e06658fcdad9010bc7d8955e130e64cb076dc4543b"}
May 25 12:13:21.149264 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:21.149237    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-107.ec2.internal" event={"ID":"decdc30bec8216583ece4597ca83be0b","Type":"ContainerStarted","Data":"44f54e3a7b06bdff95bb89e31cc8b6e06b421bbe3c836d6be425fdef039413a3"}
May 25 12:13:21.163322 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:21.163267    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qkfd7" podStartSLOduration=3.630943068 podStartE2EDuration="22.163251389s" podCreationTimestamp="2026-05-25 12:12:59 +0000 UTC" firstStartedPulling="2026-05-25 12:13:00.248886923 +0000 UTC m=+1.890081679" lastFinishedPulling="2026-05-25 12:13:18.781195228 +0000 UTC m=+20.422390000" observedRunningTime="2026-05-25 12:13:21.162959312 +0000 UTC m=+22.804154134" watchObservedRunningTime="2026-05-25 12:13:21.163251389 +0000 UTC m=+22.804446169"
May 25 12:13:21.888160 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:21.887830    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2fqnj"
May 25 12:13:21.888584 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:21.888565    2578 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2fqnj"
May 25 12:13:21.905334 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:21.905285    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-107.ec2.internal" podStartSLOduration=22.905273036 podStartE2EDuration="22.905273036s" podCreationTimestamp="2026-05-25 12:12:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:13:21.181899636 +0000 UTC m=+22.823094456" watchObservedRunningTime="2026-05-25 12:13:21.905273036 +0000 UTC m=+23.546467812"
May 25 12:13:21.935397 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:21.935375    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:13:21.935513 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:21.935382    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:21.935513 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:21.935494    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhvk8" podUID="cae409cd-445f-40b9-9851-c89b09c1ac43"
May 25 12:13:21.935626 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:21.935553    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zd5f" podUID="131acddf-521f-481d-84a3-b4e9ce65b58e"
May 25 12:13:22.154240 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:22.154200    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/ovn-acl-logging/0.log"
May 25 12:13:22.154706 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:22.154645    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc" event={"ID":"7687fe26-96a3-43ea-9223-0c5d61cd667e","Type":"ContainerStarted","Data":"7dbc9aebc995be3f4b57d1f5ad4096107980b27c133bab253ae1c894f19090bb"}
May 25 12:13:22.154976 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:22.154952    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2fqnj"
May 25 12:13:22.155432 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:22.155414    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2fqnj"
May 25 12:13:22.935857 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:22.935819    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:13:22.936049 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:22.935967    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vrb" podUID="02a94135-b85a-4ab1-9dd1-2ae75e77f2de"
May 25 12:13:23.158351 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:23.158313    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6" event={"ID":"48a83491-d11a-4002-8df4-66913cf817f6","Type":"ContainerStarted","Data":"f722fcd58863183b90e2a071f7910b9f0ec23b42aac17eafb813d2cadee0fd02"}
May 25 12:13:23.935768 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:23.935728    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:13:23.935958 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:23.935742    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:23.935958 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:23.935865    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhvk8" podUID="cae409cd-445f-40b9-9851-c89b09c1ac43"
May 25 12:13:23.935958 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:23.935936    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zd5f" podUID="131acddf-521f-481d-84a3-b4e9ce65b58e"
May 25 12:13:24.935295 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:24.935043    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:13:24.935722 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:24.935325    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vrb" podUID="02a94135-b85a-4ab1-9dd1-2ae75e77f2de"
May 25 12:13:25.164688 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:25.164665    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/ovn-acl-logging/0.log"
May 25 12:13:25.165039 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:25.165016    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc" event={"ID":"7687fe26-96a3-43ea-9223-0c5d61cd667e","Type":"ContainerStarted","Data":"4125b3e764861b9a57a3fdaba00302e8212cc4714f4e86ff475d11786ef4c351"}
May 25 12:13:25.165465 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:25.165395    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:25.165592 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:25.165544    2578 scope.go:117] "RemoveContainer" containerID="d15c4c452e037e5c72f60c995e9f1bbfde8b6720a7410f933a5968e9066bb7e5"
May 25 12:13:25.166791 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:25.166725    2578 generic.go:358] "Generic (PLEG): container finished" podID="f97986ac-3ba7-4762-9e76-38ea2ac839b6" containerID="28adbfbe5cdca7b395db38712ba09372a2cc7ee8bcd15d14305d987d7bd6e437" exitCode=0
May 25 12:13:25.166791 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:25.166763    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xg5h" event={"ID":"f97986ac-3ba7-4762-9e76-38ea2ac839b6","Type":"ContainerDied","Data":"28adbfbe5cdca7b395db38712ba09372a2cc7ee8bcd15d14305d987d7bd6e437"}
May 25 12:13:25.181612 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:25.181592    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:25.216611 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:25.216566    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68xw6" podStartSLOduration=4.160042149 podStartE2EDuration="26.216551055s" podCreationTimestamp="2026-05-25 12:12:59 +0000 UTC" firstStartedPulling="2026-05-25 12:13:00.194111416 +0000 UTC m=+1.835306175" lastFinishedPulling="2026-05-25 12:13:22.250620321 +0000 UTC m=+23.891815081" observedRunningTime="2026-05-25 12:13:23.180629014 +0000 UTC m=+24.821823804" watchObservedRunningTime="2026-05-25 12:13:25.216551055 +0000 UTC m=+26.857745858"
May 25 12:13:25.935807 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:25.935767    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:13:25.936461 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:25.935767    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:25.936461 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:25.935905    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhvk8" podUID="cae409cd-445f-40b9-9851-c89b09c1ac43"
May 25 12:13:25.936461 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:25.935987    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zd5f" podUID="131acddf-521f-481d-84a3-b4e9ce65b58e"
May 25 12:13:26.171533 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:26.171507    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/ovn-acl-logging/0.log"
May 25 12:13:26.171821 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:26.171800    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc" event={"ID":"7687fe26-96a3-43ea-9223-0c5d61cd667e","Type":"ContainerStarted","Data":"7af958798e77ae60d9720cb0cb24adab21084f2d25a69c25d6df8c9f3fb7ddfe"}
May 25 12:13:26.172163 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:26.172138    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:26.172245 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:26.172174    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:26.187430 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:26.187365    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:26.214863 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:26.214818    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc" podStartSLOduration=8.662714793 podStartE2EDuration="27.214803777s" podCreationTimestamp="2026-05-25 12:12:59 +0000 UTC" firstStartedPulling="2026-05-25 12:13:00.31541829 +0000 UTC m=+1.956613045" lastFinishedPulling="2026-05-25 12:13:18.867507259 +0000 UTC m=+20.508702029" observedRunningTime="2026-05-25 12:13:26.213248282 +0000 UTC m=+27.854443062" watchObservedRunningTime="2026-05-25 12:13:26.214803777 +0000 UTC m=+27.855998553"
May 25 12:13:26.282667 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:26.282639    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8zd5f"]
May 25 12:13:26.282784 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:26.282734    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:26.282825 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:26.282813    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zd5f" podUID="131acddf-521f-481d-84a3-b4e9ce65b58e"
May 25 12:13:26.285397 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:26.285357    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n9vrb"]
May 25 12:13:26.285508 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:26.285491    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:13:26.285636 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:26.285605    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vrb" podUID="02a94135-b85a-4ab1-9dd1-2ae75e77f2de"
May 25 12:13:26.288146 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:26.288126    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zhvk8"]
May 25 12:13:26.288229 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:26.288195    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:13:26.288312 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:26.288290    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhvk8" podUID="cae409cd-445f-40b9-9851-c89b09c1ac43"
May 25 12:13:26.386727 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:26.386696    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/131acddf-521f-481d-84a3-b4e9ce65b58e-original-pull-secret\") pod \"global-pull-secret-syncer-8zd5f\" (UID: \"131acddf-521f-481d-84a3-b4e9ce65b58e\") " pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:26.386866 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:26.386806    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 25 12:13:26.386866 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:26.386857    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/131acddf-521f-481d-84a3-b4e9ce65b58e-original-pull-secret podName:131acddf-521f-481d-84a3-b4e9ce65b58e nodeName:}" failed. No retries permitted until 2026-05-25 12:13:42.386843842 +0000 UTC m=+44.028038597 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/131acddf-521f-481d-84a3-b4e9ce65b58e-original-pull-secret") pod "global-pull-secret-syncer-8zd5f" (UID: "131acddf-521f-481d-84a3-b4e9ce65b58e") : object "kube-system"/"original-pull-secret" not registered
May 25 12:13:27.175650 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:27.175617    2578 generic.go:358] "Generic (PLEG): container finished" podID="f97986ac-3ba7-4762-9e76-38ea2ac839b6" containerID="5ac6c5f7ac5332f1d78a36feb7e5cfdf1a9cb04496eca47d1a4373d94a548a7d" exitCode=0
May 25 12:13:27.176035 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:27.175701    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xg5h" event={"ID":"f97986ac-3ba7-4762-9e76-38ea2ac839b6","Type":"ContainerDied","Data":"5ac6c5f7ac5332f1d78a36feb7e5cfdf1a9cb04496eca47d1a4373d94a548a7d"}
May 25 12:13:27.934835 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:27.934800    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:13:27.935027 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:27.934806    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:13:27.935027 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:27.934910    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vrb" podUID="02a94135-b85a-4ab1-9dd1-2ae75e77f2de"
May 25 12:13:27.935027 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:27.934996    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhvk8" podUID="cae409cd-445f-40b9-9851-c89b09c1ac43"
May 25 12:13:27.935027 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:27.934806    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:27.935239 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:27.935118    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zd5f" podUID="131acddf-521f-481d-84a3-b4e9ce65b58e"
May 25 12:13:29.180740 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:29.180706    2578 generic.go:358] "Generic (PLEG): container finished" podID="f97986ac-3ba7-4762-9e76-38ea2ac839b6" containerID="cc49907bfd3cdee193d69b31da94ff9234f60b711fb5495b3ebd534552f3a6b3" exitCode=0
May 25 12:13:29.181397 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:29.180763    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xg5h" event={"ID":"f97986ac-3ba7-4762-9e76-38ea2ac839b6","Type":"ContainerDied","Data":"cc49907bfd3cdee193d69b31da94ff9234f60b711fb5495b3ebd534552f3a6b3"}
May 25 12:13:29.935044 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:29.934977    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:13:29.935044 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:29.935004    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:13:29.935287 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:29.934977    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:29.935287 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:29.935141    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhvk8" podUID="cae409cd-445f-40b9-9851-c89b09c1ac43"
May 25 12:13:29.935287 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:29.935196    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zd5f" podUID="131acddf-521f-481d-84a3-b4e9ce65b58e"
May 25 12:13:29.935427 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:29.935284    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vrb" podUID="02a94135-b85a-4ab1-9dd1-2ae75e77f2de"
May 25 12:13:31.628081 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:31.628035    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-metrics-certs\") pod \"network-metrics-daemon-n9vrb\" (UID: \"02a94135-b85a-4ab1-9dd1-2ae75e77f2de\") " pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:13:31.628569 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:31.628198    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:31.628569 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:31.628287    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-metrics-certs podName:02a94135-b85a-4ab1-9dd1-2ae75e77f2de nodeName:}" failed. No retries permitted until 2026-05-25 12:14:03.628267064 +0000 UTC m=+65.269461832 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-metrics-certs") pod "network-metrics-daemon-n9vrb" (UID: "02a94135-b85a-4ab1-9dd1-2ae75e77f2de") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 12:13:31.729378 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:31.729343    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2rfpr\" (UniqueName: \"kubernetes.io/projected/cae409cd-445f-40b9-9851-c89b09c1ac43-kube-api-access-2rfpr\") pod \"network-check-target-zhvk8\" (UID: \"cae409cd-445f-40b9-9851-c89b09c1ac43\") " pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:13:31.729565 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:31.729528    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 12:13:31.729565 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:31.729555    2578 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 12:13:31.729676 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:31.729568    2578 projected.go:194] Error preparing data for projected volume kube-api-access-2rfpr for pod openshift-network-diagnostics/network-check-target-zhvk8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:31.729676 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:31.729636    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cae409cd-445f-40b9-9851-c89b09c1ac43-kube-api-access-2rfpr podName:cae409cd-445f-40b9-9851-c89b09c1ac43 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:03.729616432 +0000 UTC m=+65.370811201 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2rfpr" (UniqueName: "kubernetes.io/projected/cae409cd-445f-40b9-9851-c89b09c1ac43-kube-api-access-2rfpr") pod "network-check-target-zhvk8" (UID: "cae409cd-445f-40b9-9851-c89b09c1ac43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 12:13:31.935111 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:31.935009    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:13:31.935111 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:31.935033    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:31.935111 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:31.935013    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:13:31.935382 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:31.935161    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhvk8" podUID="cae409cd-445f-40b9-9851-c89b09c1ac43"
May 25 12:13:31.935382 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:31.935235    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8zd5f" podUID="131acddf-521f-481d-84a3-b4e9ce65b58e"
May 25 12:13:31.935382 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:31.935311    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9vrb" podUID="02a94135-b85a-4ab1-9dd1-2ae75e77f2de"
May 25 12:13:32.178761 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.178733    2578 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-107.ec2.internal" event="NodeReady"
May 25 12:13:32.178919 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.178865    2578 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 25 12:13:32.217935 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.217846    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567"]
May 25 12:13:32.219724 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.219674    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567"
May 25 12:13:32.222150 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.222086    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d77d4df8-q8k4f"]
May 25 12:13:32.223389 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.223357    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
May 25 12:13:32.223520 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.223357    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
May 25 12:13:32.223775 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.223753    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
May 25 12:13:32.224404 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.224194    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f577cd989-k9p4z"]
May 25 12:13:32.224404 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.224270    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
May 25 12:13:32.224644 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.224624    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
May 25 12:13:32.224774 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.224740    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
May 25 12:13:32.224923 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.224906    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
May 25 12:13:32.225827 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.225808    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d44874984-xhvxn"]
May 25 12:13:32.225928 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.225867    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d77d4df8-q8k4f"
May 25 12:13:32.225928 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.225893    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:32.227550 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.227532    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d44874984-xhvxn"
May 25 12:13:32.228116 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.228097    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
May 25 12:13:32.228837 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.228696    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 25 12:13:32.229095 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.229078    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 25 12:13:32.229323 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.229301    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s7xbv\""
May 25 12:13:32.229323 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.229302    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-s952h\""
May 25 12:13:32.229622 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.229605    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 25 12:13:32.230374 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.230354    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
May 25 12:13:32.233237 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.233220    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567"]
May 25 12:13:32.238732 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.238542    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 25 12:13:32.239611 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.239406    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d77d4df8-q8k4f"]
May 25 12:13:32.240958 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.240908    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d44874984-xhvxn"]
May 25 12:13:32.241847 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.241820    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f577cd989-k9p4z"]
May 25 12:13:32.242683 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.242667    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xk9w8"]
May 25 12:13:32.245046 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.245026    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mzmd6"]
May 25 12:13:32.245306 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.245169    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xk9w8"
May 25 12:13:32.247244 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.247224    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mzmd6"
May 25 12:13:32.247978 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.247959    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 25 12:13:32.248186 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.247965    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 25 12:13:32.248280 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.248048    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c8dc6\""
May 25 12:13:32.249485 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.249470    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 25 12:13:32.250273 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.250249    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 25 12:13:32.250464 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.250256    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 25 12:13:32.250464 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.250403    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mzmd6"]
May 25 12:13:32.250464 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.250438    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-sw9bq\""
May 25 12:13:32.253235 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.253217    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xk9w8"]
May 25 12:13:32.333115 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.333082    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmszn\" (UniqueName: \"kubernetes.io/projected/a651c313-a83a-44ed-9ee1-e86abaf28f6b-kube-api-access-mmszn\") pod \"cluster-proxy-proxy-agent-65455b8849-fx567\" (UID: \"a651c313-a83a-44ed-9ee1-e86abaf28f6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567"
May 25 12:13:32.333293 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.333123    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb75886-2392-4014-b64b-e61e092ffbd9-tmp-dir\") pod \"dns-default-xk9w8\" (UID: \"6bb75886-2392-4014-b64b-e61e092ffbd9\") " pod="openshift-dns/dns-default-xk9w8"
May 25 12:13:32.333293 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.333153    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f49b1b6-b6fb-4d59-9e56-163789263c0e-tmp\") pod \"klusterlet-addon-workmgr-d44874984-xhvxn\" (UID: \"5f49b1b6-b6fb-4d59-9e56-163789263c0e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d44874984-xhvxn"
May 25 12:13:32.333293 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.333202    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:32.333293 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.333251    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a42b2c14-3506-4a13-9c37-e917e26a9307-image-registry-private-configuration\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:32.333293 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.333275    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-certificates\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:32.333503 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.333299    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a42b2c14-3506-4a13-9c37-e917e26a9307-installation-pull-secrets\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:32.333503 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.333319    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a651c313-a83a-44ed-9ee1-e86abaf28f6b-hub\") pod \"cluster-proxy-proxy-agent-65455b8849-fx567\" (UID: \"a651c313-a83a-44ed-9ee1-e86abaf28f6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567"
May 25 12:13:32.333503 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.333360    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb75886-2392-4014-b64b-e61e092ffbd9-config-volume\") pod \"dns-default-xk9w8\" (UID: \"6bb75886-2392-4014-b64b-e61e092ffbd9\") " pod="openshift-dns/dns-default-xk9w8"
May 25 12:13:32.333503 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.333427    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a42b2c14-3506-4a13-9c37-e917e26a9307-ca-trust-extracted\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:32.333503 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.333468    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6bb75886-2392-4014-b64b-e61e092ffbd9-metrics-tls\") pod \"dns-default-xk9w8\" (UID: \"6bb75886-2392-4014-b64b-e61e092ffbd9\") " pod="openshift-dns/dns-default-xk9w8"
May 25 12:13:32.333503 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.333499    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5f49b1b6-b6fb-4d59-9e56-163789263c0e-klusterlet-config\") pod \"klusterlet-addon-workmgr-d44874984-xhvxn\" (UID: \"5f49b1b6-b6fb-4d59-9e56-163789263c0e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d44874984-xhvxn"
May 25 12:13:32.333844 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.333523    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzhc8\" (UniqueName: \"kubernetes.io/projected/5f49b1b6-b6fb-4d59-9e56-163789263c0e-kube-api-access-qzhc8\") pod \"klusterlet-addon-workmgr-d44874984-xhvxn\" (UID: \"5f49b1b6-b6fb-4d59-9e56-163789263c0e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d44874984-xhvxn"
May 25 12:13:32.333844 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.333546    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-bound-sa-token\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:32.333844 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.333637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/587db4ef-f6a5-48c3-9e78-957d815002a4-cert\") pod \"ingress-canary-mzmd6\" (UID: \"587db4ef-f6a5-48c3-9e78-957d815002a4\") " pod="openshift-ingress-canary/ingress-canary-mzmd6"
May 25 12:13:32.333844 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.333676    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds8ng\" (UniqueName: \"kubernetes.io/projected/587db4ef-f6a5-48c3-9e78-957d815002a4-kube-api-access-ds8ng\") pod \"ingress-canary-mzmd6\" (UID: \"587db4ef-f6a5-48c3-9e78-957d815002a4\") " pod="openshift-ingress-canary/ingress-canary-mzmd6"
May 25 12:13:32.333844 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.333708    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwp9z\" (UniqueName: \"kubernetes.io/projected/00a70888-8d3b-47a2-b012-fa50c61fd91d-kube-api-access-bwp9z\") pod \"managed-serviceaccount-addon-agent-67d77d4df8-q8k4f\" (UID: \"00a70888-8d3b-47a2-b012-fa50c61fd91d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d77d4df8-q8k4f"
May 25 12:13:32.333844 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.333736    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svjn5\" (UniqueName: \"kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-kube-api-access-svjn5\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:32.333844 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.333770    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a651c313-a83a-44ed-9ee1-e86abaf28f6b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-65455b8849-fx567\" (UID: \"a651c313-a83a-44ed-9ee1-e86abaf28f6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567"
May 25 12:13:32.333844 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.333815    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a42b2c14-3506-4a13-9c37-e917e26a9307-trusted-ca\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:32.333844 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.333840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a651c313-a83a-44ed-9ee1-e86abaf28f6b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-65455b8849-fx567\" (UID: \"a651c313-a83a-44ed-9ee1-e86abaf28f6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567"
May 25 12:13:32.334225 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.333873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/a651c313-a83a-44ed-9ee1-e86abaf28f6b-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-65455b8849-fx567\" (UID: \"a651c313-a83a-44ed-9ee1-e86abaf28f6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567"
May 25 12:13:32.334225 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.333904    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00a70888-8d3b-47a2-b012-fa50c61fd91d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-67d77d4df8-q8k4f\" (UID: \"00a70888-8d3b-47a2-b012-fa50c61fd91d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d77d4df8-q8k4f"
May 25 12:13:32.334225 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.333950    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a651c313-a83a-44ed-9ee1-e86abaf28f6b-ca\") pod \"cluster-proxy-proxy-agent-65455b8849-fx567\" (UID: \"a651c313-a83a-44ed-9ee1-e86abaf28f6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567"
May 25 12:13:32.334225 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.333985    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn599\" (UniqueName: \"kubernetes.io/projected/6bb75886-2392-4014-b64b-e61e092ffbd9-kube-api-access-gn599\") pod \"dns-default-xk9w8\" (UID: \"6bb75886-2392-4014-b64b-e61e092ffbd9\") " pod="openshift-dns/dns-default-xk9w8"
May 25 12:13:32.434685 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.434647    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00a70888-8d3b-47a2-b012-fa50c61fd91d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-67d77d4df8-q8k4f\" (UID: \"00a70888-8d3b-47a2-b012-fa50c61fd91d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d77d4df8-q8k4f"
May 25 12:13:32.434685 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.434690    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a651c313-a83a-44ed-9ee1-e86abaf28f6b-ca\") pod \"cluster-proxy-proxy-agent-65455b8849-fx567\" (UID: \"a651c313-a83a-44ed-9ee1-e86abaf28f6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567"
May 25 12:13:32.434938 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.434718    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gn599\" (UniqueName: \"kubernetes.io/projected/6bb75886-2392-4014-b64b-e61e092ffbd9-kube-api-access-gn599\") pod \"dns-default-xk9w8\" (UID: \"6bb75886-2392-4014-b64b-e61e092ffbd9\") " pod="openshift-dns/dns-default-xk9w8"
May 25 12:13:32.434938 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.434746    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mmszn\" (UniqueName: \"kubernetes.io/projected/a651c313-a83a-44ed-9ee1-e86abaf28f6b-kube-api-access-mmszn\") pod \"cluster-proxy-proxy-agent-65455b8849-fx567\" (UID: \"a651c313-a83a-44ed-9ee1-e86abaf28f6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567"
May 25 12:13:32.434938 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.434774    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb75886-2392-4014-b64b-e61e092ffbd9-tmp-dir\") pod \"dns-default-xk9w8\" (UID: \"6bb75886-2392-4014-b64b-e61e092ffbd9\") " pod="openshift-dns/dns-default-xk9w8"
May 25 12:13:32.434938 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.434798    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f49b1b6-b6fb-4d59-9e56-163789263c0e-tmp\") pod \"klusterlet-addon-workmgr-d44874984-xhvxn\" (UID: \"5f49b1b6-b6fb-4d59-9e56-163789263c0e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d44874984-xhvxn"
May 25 12:13:32.434938 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.434841    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:32.434938 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.434897    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a42b2c14-3506-4a13-9c37-e917e26a9307-image-registry-private-configuration\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:32.434938 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.434925    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-certificates\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:32.435301 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.434946    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a42b2c14-3506-4a13-9c37-e917e26a9307-installation-pull-secrets\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:32.435301 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.434963    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a651c313-a83a-44ed-9ee1-e86abaf28f6b-hub\") pod \"cluster-proxy-proxy-agent-65455b8849-fx567\" (UID: \"a651c313-a83a-44ed-9ee1-e86abaf28f6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567"
May 25 12:13:32.435301 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.434978    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb75886-2392-4014-b64b-e61e092ffbd9-config-volume\") pod \"dns-default-xk9w8\" (UID: \"6bb75886-2392-4014-b64b-e61e092ffbd9\") " pod="openshift-dns/dns-default-xk9w8"
May 25 12:13:32.435301 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.434997    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a42b2c14-3506-4a13-9c37-e917e26a9307-ca-trust-extracted\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:32.435301 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.435012    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6bb75886-2392-4014-b64b-e61e092ffbd9-metrics-tls\") pod \"dns-default-xk9w8\" (UID: \"6bb75886-2392-4014-b64b-e61e092ffbd9\") " pod="openshift-dns/dns-default-xk9w8"
May 25 12:13:32.435301 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.435029    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5f49b1b6-b6fb-4d59-9e56-163789263c0e-klusterlet-config\") pod \"klusterlet-addon-workmgr-d44874984-xhvxn\" (UID: \"5f49b1b6-b6fb-4d59-9e56-163789263c0e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d44874984-xhvxn"
May 25 12:13:32.435301 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.435046    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qzhc8\" (UniqueName: \"kubernetes.io/projected/5f49b1b6-b6fb-4d59-9e56-163789263c0e-kube-api-access-qzhc8\") pod \"klusterlet-addon-workmgr-d44874984-xhvxn\" (UID: \"5f49b1b6-b6fb-4d59-9e56-163789263c0e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d44874984-xhvxn"
May 25 12:13:32.435301 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.435095    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-bound-sa-token\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:32.435301 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.435121    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/587db4ef-f6a5-48c3-9e78-957d815002a4-cert\") pod \"ingress-canary-mzmd6\" (UID: \"587db4ef-f6a5-48c3-9e78-957d815002a4\") " pod="openshift-ingress-canary/ingress-canary-mzmd6"
May 25 12:13:32.435301 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.435136    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ds8ng\" (UniqueName: \"kubernetes.io/projected/587db4ef-f6a5-48c3-9e78-957d815002a4-kube-api-access-ds8ng\") pod \"ingress-canary-mzmd6\" (UID: \"587db4ef-f6a5-48c3-9e78-957d815002a4\") " pod="openshift-ingress-canary/ingress-canary-mzmd6"
May 25 12:13:32.435301 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.435156    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bwp9z\" (UniqueName: \"kubernetes.io/projected/00a70888-8d3b-47a2-b012-fa50c61fd91d-kube-api-access-bwp9z\") pod \"managed-serviceaccount-addon-agent-67d77d4df8-q8k4f\" (UID: \"00a70888-8d3b-47a2-b012-fa50c61fd91d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d77d4df8-q8k4f"
May 25 12:13:32.435301 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.435178    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-svjn5\" (UniqueName: \"kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-kube-api-access-svjn5\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:32.435301 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.435195    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a651c313-a83a-44ed-9ee1-e86abaf28f6b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-65455b8849-fx567\" (UID: \"a651c313-a83a-44ed-9ee1-e86abaf28f6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567"
May 25 12:13:32.435301 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.435219    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a42b2c14-3506-4a13-9c37-e917e26a9307-trusted-ca\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:32.435301 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.435235    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a651c313-a83a-44ed-9ee1-e86abaf28f6b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-65455b8849-fx567\" (UID: \"a651c313-a83a-44ed-9ee1-e86abaf28f6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567"
May 25 12:13:32.435301 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.435259    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/a651c313-a83a-44ed-9ee1-e86abaf28f6b-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-65455b8849-fx567\" (UID: \"a651c313-a83a-44ed-9ee1-e86abaf28f6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567"
May 25 12:13:32.436075 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.435459    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a42b2c14-3506-4a13-9c37-e917e26a9307-ca-trust-extracted\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:32.436075 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.435692    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5f49b1b6-b6fb-4d59-9e56-163789263c0e-tmp\") pod \"klusterlet-addon-workmgr-d44874984-xhvxn\" (UID: \"5f49b1b6-b6fb-4d59-9e56-163789263c0e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d44874984-xhvxn"
May 25 12:13:32.436075 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.435870    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb75886-2392-4014-b64b-e61e092ffbd9-tmp-dir\") pod \"dns-default-xk9w8\" (UID: \"6bb75886-2392-4014-b64b-e61e092ffbd9\") " pod="openshift-dns/dns-default-xk9w8"
May 25 12:13:32.436075 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:32.435939    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:13:32.436075 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.435972    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-certificates\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:32.436075 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.435996    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/a651c313-a83a-44ed-9ee1-e86abaf28f6b-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-65455b8849-fx567\" (UID: \"a651c313-a83a-44ed-9ee1-e86abaf28f6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567"
May 25 12:13:32.436075 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:32.436023    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bb75886-2392-4014-b64b-e61e092ffbd9-metrics-tls podName:6bb75886-2392-4014-b64b-e61e092ffbd9 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:32.936004991 +0000 UTC m=+34.577199755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6bb75886-2392-4014-b64b-e61e092ffbd9-metrics-tls") pod "dns-default-xk9w8" (UID: "6bb75886-2392-4014-b64b-e61e092ffbd9") : secret "dns-default-metrics-tls" not found
May 25 12:13:32.436075 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:32.436034    2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:13:32.436433 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:32.436118    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587db4ef-f6a5-48c3-9e78-957d815002a4-cert podName:587db4ef-f6a5-48c3-9e78-957d815002a4 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:32.936101474 +0000 UTC m=+34.577296253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/587db4ef-f6a5-48c3-9e78-957d815002a4-cert") pod "ingress-canary-mzmd6" (UID: "587db4ef-f6a5-48c3-9e78-957d815002a4") : secret "canary-serving-cert" not found
May 25 12:13:32.436433 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:32.436433    2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 12:13:32.436524 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.436439    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6bb75886-2392-4014-b64b-e61e092ffbd9-config-volume\") pod \"dns-default-xk9w8\" (UID: \"6bb75886-2392-4014-b64b-e61e092ffbd9\") " pod="openshift-dns/dns-default-xk9w8"
May 25 12:13:32.436524 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:32.436448    2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f577cd989-k9p4z: secret "image-registry-tls" not found
May 25 12:13:32.436607 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:32.436525    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls podName:a42b2c14-3506-4a13-9c37-e917e26a9307 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:32.93651175 +0000 UTC m=+34.577706507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls") pod "image-registry-6f577cd989-k9p4z" (UID: "a42b2c14-3506-4a13-9c37-e917e26a9307") : secret "image-registry-tls" not found
May 25 12:13:32.437406 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.437362    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a42b2c14-3506-4a13-9c37-e917e26a9307-trusted-ca\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:32.441103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.440618    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a651c313-a83a-44ed-9ee1-e86abaf28f6b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-65455b8849-fx567\" (UID: \"a651c313-a83a-44ed-9ee1-e86abaf28f6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567"
May 25 12:13:32.441103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.440735    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a651c313-a83a-44ed-9ee1-e86abaf28f6b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-65455b8849-fx567\" (UID: \"a651c313-a83a-44ed-9ee1-e86abaf28f6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567"
May 25 12:13:32.441103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.440800    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/5f49b1b6-b6fb-4d59-9e56-163789263c0e-klusterlet-config\") pod \"klusterlet-addon-workmgr-d44874984-xhvxn\" (UID: \"5f49b1b6-b6fb-4d59-9e56-163789263c0e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d44874984-xhvxn"
May 25 12:13:32.441103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.440998    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a42b2c14-3506-4a13-9c37-e917e26a9307-image-registry-private-configuration\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:32.441356 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.441106    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a651c313-a83a-44ed-9ee1-e86abaf28f6b-hub\") pod \"cluster-proxy-proxy-agent-65455b8849-fx567\" (UID: \"a651c313-a83a-44ed-9ee1-e86abaf28f6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567"
May 25 12:13:32.441356 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.441161    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a651c313-a83a-44ed-9ee1-e86abaf28f6b-ca\") pod \"cluster-proxy-proxy-agent-65455b8849-fx567\" (UID: \"a651c313-a83a-44ed-9ee1-e86abaf28f6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567"
May 25 12:13:32.441457 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.441433    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00a70888-8d3b-47a2-b012-fa50c61fd91d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-67d77d4df8-q8k4f\" (UID: \"00a70888-8d3b-47a2-b012-fa50c61fd91d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d77d4df8-q8k4f"
May 25 12:13:32.443013 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.442987    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a42b2c14-3506-4a13-9c37-e917e26a9307-installation-pull-secrets\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:32.443807 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.443785    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn599\" (UniqueName: \"kubernetes.io/projected/6bb75886-2392-4014-b64b-e61e092ffbd9-kube-api-access-gn599\") pod \"dns-default-xk9w8\" (UID: \"6bb75886-2392-4014-b64b-e61e092ffbd9\") " pod="openshift-dns/dns-default-xk9w8"
May 25 12:13:32.444172 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.444152    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds8ng\" (UniqueName: \"kubernetes.io/projected/587db4ef-f6a5-48c3-9e78-957d815002a4-kube-api-access-ds8ng\") pod \"ingress-canary-mzmd6\" (UID: \"587db4ef-f6a5-48c3-9e78-957d815002a4\") " pod="openshift-ingress-canary/ingress-canary-mzmd6"
May 25 12:13:32.445358 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.445327    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmszn\" (UniqueName: \"kubernetes.io/projected/a651c313-a83a-44ed-9ee1-e86abaf28f6b-kube-api-access-mmszn\") pod \"cluster-proxy-proxy-agent-65455b8849-fx567\" (UID: \"a651c313-a83a-44ed-9ee1-e86abaf28f6b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567"
May 25 12:13:32.447830 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.447803    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svjn5\" (UniqueName: \"kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-kube-api-access-svjn5\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:32.448074 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.448004    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwp9z\" (UniqueName: \"kubernetes.io/projected/00a70888-8d3b-47a2-b012-fa50c61fd91d-kube-api-access-bwp9z\") pod \"managed-serviceaccount-addon-agent-67d77d4df8-q8k4f\" (UID: \"00a70888-8d3b-47a2-b012-fa50c61fd91d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d77d4df8-q8k4f"
May 25 12:13:32.448164 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.448072    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzhc8\" (UniqueName: \"kubernetes.io/projected/5f49b1b6-b6fb-4d59-9e56-163789263c0e-kube-api-access-qzhc8\") pod \"klusterlet-addon-workmgr-d44874984-xhvxn\" (UID: \"5f49b1b6-b6fb-4d59-9e56-163789263c0e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d44874984-xhvxn"
May 25 12:13:32.448333 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.448315    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-bound-sa-token\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:32.542431 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.542401    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567"
May 25 12:13:32.550226 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.550204    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d77d4df8-q8k4f"
May 25 12:13:32.576132 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.576093    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d44874984-xhvxn"
May 25 12:13:32.725764 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.725710    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d77d4df8-q8k4f"]
May 25 12:13:32.729895 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:13:32.729868    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00a70888_8d3b_47a2_b012_fa50c61fd91d.slice/crio-18efe5b15b20af55ad5a5ba503d2bbf5e3d0ca842409660f52094f08b1c4fe56 WatchSource:0}: Error finding container 18efe5b15b20af55ad5a5ba503d2bbf5e3d0ca842409660f52094f08b1c4fe56: Status 404 returned error can't find the container with id 18efe5b15b20af55ad5a5ba503d2bbf5e3d0ca842409660f52094f08b1c4fe56
May 25 12:13:32.736764 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.736736    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567"]
May 25 12:13:32.740541 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:13:32.740511    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda651c313_a83a_44ed_9ee1_e86abaf28f6b.slice/crio-c817c3a74080059583234a627ede57566b278c420fa70c04109adfa063b4a032 WatchSource:0}: Error finding container c817c3a74080059583234a627ede57566b278c420fa70c04109adfa063b4a032: Status 404 returned error can't find the container with id c817c3a74080059583234a627ede57566b278c420fa70c04109adfa063b4a032
May 25 12:13:32.768204 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.768166    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d44874984-xhvxn"]
May 25 12:13:32.773483 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:13:32.773448    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f49b1b6_b6fb_4d59_9e56_163789263c0e.slice/crio-090afb9da428a51ecf317fd511e4d8ca87f7ffb5bfa192778650a6df127fd86a WatchSource:0}: Error finding container 090afb9da428a51ecf317fd511e4d8ca87f7ffb5bfa192778650a6df127fd86a: Status 404 returned error can't find the container with id 090afb9da428a51ecf317fd511e4d8ca87f7ffb5bfa192778650a6df127fd86a
May 25 12:13:32.940488 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.940374    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:32.940488 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.940435    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6bb75886-2392-4014-b64b-e61e092ffbd9-metrics-tls\") pod \"dns-default-xk9w8\" (UID: \"6bb75886-2392-4014-b64b-e61e092ffbd9\") " pod="openshift-dns/dns-default-xk9w8"
May 25 12:13:32.940488 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:32.940468    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/587db4ef-f6a5-48c3-9e78-957d815002a4-cert\") pod \"ingress-canary-mzmd6\" (UID: \"587db4ef-f6a5-48c3-9e78-957d815002a4\") " pod="openshift-ingress-canary/ingress-canary-mzmd6"
May 25 12:13:32.940761 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:32.940537    2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 12:13:32.940761 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:32.940560    2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f577cd989-k9p4z: secret "image-registry-tls" not found
May 25 12:13:32.940761 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:32.940560    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:13:32.940761 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:32.940568    2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:13:32.940761 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:32.940616    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls podName:a42b2c14-3506-4a13-9c37-e917e26a9307 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:33.940598023 +0000 UTC m=+35.581792782 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls") pod "image-registry-6f577cd989-k9p4z" (UID: "a42b2c14-3506-4a13-9c37-e917e26a9307") : secret "image-registry-tls" not found
May 25 12:13:32.940761 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:32.940630    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bb75886-2392-4014-b64b-e61e092ffbd9-metrics-tls podName:6bb75886-2392-4014-b64b-e61e092ffbd9 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:33.940623877 +0000 UTC m=+35.581818631 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6bb75886-2392-4014-b64b-e61e092ffbd9-metrics-tls") pod "dns-default-xk9w8" (UID: "6bb75886-2392-4014-b64b-e61e092ffbd9") : secret "dns-default-metrics-tls" not found
May 25 12:13:32.940761 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:32.940640    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587db4ef-f6a5-48c3-9e78-957d815002a4-cert podName:587db4ef-f6a5-48c3-9e78-957d815002a4 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:33.940634738 +0000 UTC m=+35.581829494 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/587db4ef-f6a5-48c3-9e78-957d815002a4-cert") pod "ingress-canary-mzmd6" (UID: "587db4ef-f6a5-48c3-9e78-957d815002a4") : secret "canary-serving-cert" not found
May 25 12:13:33.189703 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:33.189660    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d77d4df8-q8k4f" event={"ID":"00a70888-8d3b-47a2-b012-fa50c61fd91d","Type":"ContainerStarted","Data":"18efe5b15b20af55ad5a5ba503d2bbf5e3d0ca842409660f52094f08b1c4fe56"}
May 25 12:13:33.191185 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:33.191105    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d44874984-xhvxn" event={"ID":"5f49b1b6-b6fb-4d59-9e56-163789263c0e","Type":"ContainerStarted","Data":"090afb9da428a51ecf317fd511e4d8ca87f7ffb5bfa192778650a6df127fd86a"}
May 25 12:13:33.192330 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:33.192282    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567" event={"ID":"a651c313-a83a-44ed-9ee1-e86abaf28f6b","Type":"ContainerStarted","Data":"c817c3a74080059583234a627ede57566b278c420fa70c04109adfa063b4a032"}
May 25 12:13:33.935356 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:33.935103    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:33.935356 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:33.935105    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:13:33.935847 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:33.935507    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:13:33.940357 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:33.939848    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 25 12:13:33.942779 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:33.942087    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 12:13:33.942779 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:33.942325    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nmk2h\""
May 25 12:13:33.942779 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:33.942389    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 12:13:33.942779 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:33.942502    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rrs9z\""
May 25 12:13:33.942779 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:33.942612    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 12:13:33.951647 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:33.950695    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:33.951647 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:33.950773    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6bb75886-2392-4014-b64b-e61e092ffbd9-metrics-tls\") pod \"dns-default-xk9w8\" (UID: \"6bb75886-2392-4014-b64b-e61e092ffbd9\") " pod="openshift-dns/dns-default-xk9w8"
May 25 12:13:33.951647 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:33.950825    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/587db4ef-f6a5-48c3-9e78-957d815002a4-cert\") pod \"ingress-canary-mzmd6\" (UID: \"587db4ef-f6a5-48c3-9e78-957d815002a4\") " pod="openshift-ingress-canary/ingress-canary-mzmd6"
May 25 12:13:33.951647 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:33.950961    2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:13:33.951647 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:33.951018    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587db4ef-f6a5-48c3-9e78-957d815002a4-cert podName:587db4ef-f6a5-48c3-9e78-957d815002a4 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:35.950999852 +0000 UTC m=+37.592194607 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/587db4ef-f6a5-48c3-9e78-957d815002a4-cert") pod "ingress-canary-mzmd6" (UID: "587db4ef-f6a5-48c3-9e78-957d815002a4") : secret "canary-serving-cert" not found
May 25 12:13:33.951647 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:33.951450    2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 12:13:33.951647 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:33.951467    2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f577cd989-k9p4z: secret "image-registry-tls" not found
May 25 12:13:33.951647 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:33.951513    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls podName:a42b2c14-3506-4a13-9c37-e917e26a9307 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:35.951498128 +0000 UTC m=+37.592692883 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls") pod "image-registry-6f577cd989-k9p4z" (UID: "a42b2c14-3506-4a13-9c37-e917e26a9307") : secret "image-registry-tls" not found
May 25 12:13:33.951647 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:33.951579    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:13:33.951647 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:33.951609    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bb75886-2392-4014-b64b-e61e092ffbd9-metrics-tls podName:6bb75886-2392-4014-b64b-e61e092ffbd9 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:35.951598588 +0000 UTC m=+37.592793357 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6bb75886-2392-4014-b64b-e61e092ffbd9-metrics-tls") pod "dns-default-xk9w8" (UID: "6bb75886-2392-4014-b64b-e61e092ffbd9") : secret "dns-default-metrics-tls" not found
May 25 12:13:35.968692 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:35.968653    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:35.969160 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:35.968736    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6bb75886-2392-4014-b64b-e61e092ffbd9-metrics-tls\") pod \"dns-default-xk9w8\" (UID: \"6bb75886-2392-4014-b64b-e61e092ffbd9\") " pod="openshift-dns/dns-default-xk9w8"
May 25 12:13:35.969160 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:35.968780    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/587db4ef-f6a5-48c3-9e78-957d815002a4-cert\") pod \"ingress-canary-mzmd6\" (UID: \"587db4ef-f6a5-48c3-9e78-957d815002a4\") " pod="openshift-ingress-canary/ingress-canary-mzmd6"
May 25 12:13:35.969160 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:35.968821    2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 12:13:35.969160 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:35.968841    2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f577cd989-k9p4z: secret "image-registry-tls" not found
May 25 12:13:35.969160 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:35.968904    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls podName:a42b2c14-3506-4a13-9c37-e917e26a9307 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:39.968882746 +0000 UTC m=+41.610077523 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls") pod "image-registry-6f577cd989-k9p4z" (UID: "a42b2c14-3506-4a13-9c37-e917e26a9307") : secret "image-registry-tls" not found
May 25 12:13:35.969160 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:35.968908    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:13:35.969160 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:35.968933    2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:13:35.969160 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:35.968967    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bb75886-2392-4014-b64b-e61e092ffbd9-metrics-tls podName:6bb75886-2392-4014-b64b-e61e092ffbd9 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:39.968949326 +0000 UTC m=+41.610144084 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6bb75886-2392-4014-b64b-e61e092ffbd9-metrics-tls") pod "dns-default-xk9w8" (UID: "6bb75886-2392-4014-b64b-e61e092ffbd9") : secret "dns-default-metrics-tls" not found
May 25 12:13:35.969160 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:35.968987    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587db4ef-f6a5-48c3-9e78-957d815002a4-cert podName:587db4ef-f6a5-48c3-9e78-957d815002a4 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:39.968977005 +0000 UTC m=+41.610171767 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/587db4ef-f6a5-48c3-9e78-957d815002a4-cert") pod "ingress-canary-mzmd6" (UID: "587db4ef-f6a5-48c3-9e78-957d815002a4") : secret "canary-serving-cert" not found
May 25 12:13:40.007449 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:40.007412    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:40.007963 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:40.007485    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6bb75886-2392-4014-b64b-e61e092ffbd9-metrics-tls\") pod \"dns-default-xk9w8\" (UID: \"6bb75886-2392-4014-b64b-e61e092ffbd9\") " pod="openshift-dns/dns-default-xk9w8"
May 25 12:13:40.007963 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:40.007527    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/587db4ef-f6a5-48c3-9e78-957d815002a4-cert\") pod \"ingress-canary-mzmd6\" (UID: \"587db4ef-f6a5-48c3-9e78-957d815002a4\") " pod="openshift-ingress-canary/ingress-canary-mzmd6"
May 25 12:13:40.007963 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:40.007573    2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 12:13:40.007963 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:40.007592    2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f577cd989-k9p4z: secret "image-registry-tls" not found
May 25 12:13:40.007963 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:40.007643    2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:13:40.007963 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:40.007653    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls podName:a42b2c14-3506-4a13-9c37-e917e26a9307 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:48.007633122 +0000 UTC m=+49.648827883 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls") pod "image-registry-6f577cd989-k9p4z" (UID: "a42b2c14-3506-4a13-9c37-e917e26a9307") : secret "image-registry-tls" not found
May 25 12:13:40.007963 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:40.007654    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:13:40.007963 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:40.007675    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587db4ef-f6a5-48c3-9e78-957d815002a4-cert podName:587db4ef-f6a5-48c3-9e78-957d815002a4 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:48.007664636 +0000 UTC m=+49.648859394 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/587db4ef-f6a5-48c3-9e78-957d815002a4-cert") pod "ingress-canary-mzmd6" (UID: "587db4ef-f6a5-48c3-9e78-957d815002a4") : secret "canary-serving-cert" not found
May 25 12:13:40.007963 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:40.007738    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bb75886-2392-4014-b64b-e61e092ffbd9-metrics-tls podName:6bb75886-2392-4014-b64b-e61e092ffbd9 nodeName:}" failed. No retries permitted until 2026-05-25 12:13:48.007717813 +0000 UTC m=+49.648912585 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6bb75886-2392-4014-b64b-e61e092ffbd9-metrics-tls") pod "dns-default-xk9w8" (UID: "6bb75886-2392-4014-b64b-e61e092ffbd9") : secret "dns-default-metrics-tls" not found
May 25 12:13:42.212327 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:42.212281    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d77d4df8-q8k4f" event={"ID":"00a70888-8d3b-47a2-b012-fa50c61fd91d","Type":"ContainerStarted","Data":"752e4c68d6ecbfa1b74abafc14cc1946de4d946291af2a0645c986618a5ec462"}
May 25 12:13:42.213541 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:42.213520    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567" event={"ID":"a651c313-a83a-44ed-9ee1-e86abaf28f6b","Type":"ContainerStarted","Data":"b456e78c21331a5970ae3ffa679678ba2bd6bf2201d251e565aa74a0d55cef69"}
May 25 12:13:42.215765 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:42.215738    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xg5h" event={"ID":"f97986ac-3ba7-4762-9e76-38ea2ac839b6","Type":"ContainerStarted","Data":"482ad531cfb052e1221e4d870c3a34e8c880c336d1408fea9d6b4ebb169ccce4"}
May 25 12:13:42.238604 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:42.238558    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d77d4df8-q8k4f" podStartSLOduration=25.151222633 podStartE2EDuration="34.238546492s" podCreationTimestamp="2026-05-25 12:13:08 +0000 UTC" firstStartedPulling="2026-05-25 12:13:32.732214138 +0000 UTC m=+34.373408893" lastFinishedPulling="2026-05-25 12:13:41.819537997 +0000 UTC m=+43.460732752" observedRunningTime="2026-05-25 12:13:42.237788359 +0000 UTC m=+43.878983137" watchObservedRunningTime="2026-05-25 12:13:42.238546492 +0000 UTC m=+43.879741269"
May 25 12:13:42.427082 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:42.427038    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/131acddf-521f-481d-84a3-b4e9ce65b58e-original-pull-secret\") pod \"global-pull-secret-syncer-8zd5f\" (UID: \"131acddf-521f-481d-84a3-b4e9ce65b58e\") " pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:42.430265 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:42.430241    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/131acddf-521f-481d-84a3-b4e9ce65b58e-original-pull-secret\") pod \"global-pull-secret-syncer-8zd5f\" (UID: \"131acddf-521f-481d-84a3-b4e9ce65b58e\") " pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:42.654135 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:42.654006    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8zd5f"
May 25 12:13:42.776004 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:42.775972    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8zd5f"]
May 25 12:13:42.779185 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:13:42.779153    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131acddf_521f_481d_84a3_b4e9ce65b58e.slice/crio-1eefff2d5e3624abf50fd08c742b205beb07fdc560d88a3dfa6f2d5fb09318db WatchSource:0}: Error finding container 1eefff2d5e3624abf50fd08c742b205beb07fdc560d88a3dfa6f2d5fb09318db: Status 404 returned error can't find the container with id 1eefff2d5e3624abf50fd08c742b205beb07fdc560d88a3dfa6f2d5fb09318db
May 25 12:13:43.219263 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:43.219224    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d44874984-xhvxn" event={"ID":"5f49b1b6-b6fb-4d59-9e56-163789263c0e","Type":"ContainerStarted","Data":"8614720c0194789be0a6896d36c6c28e1edf3ae18d98bff26702b95cd0dc3cdd"}
May 25 12:13:43.219746 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:43.219399    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d44874984-xhvxn"
May 25 12:13:43.220320 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:43.220289    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8zd5f" event={"ID":"131acddf-521f-481d-84a3-b4e9ce65b58e","Type":"ContainerStarted","Data":"1eefff2d5e3624abf50fd08c742b205beb07fdc560d88a3dfa6f2d5fb09318db"}
May 25 12:13:43.221234 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:43.221217    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d44874984-xhvxn"
May 25 12:13:43.222689 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:43.222661    2578 generic.go:358] "Generic (PLEG): container finished" podID="f97986ac-3ba7-4762-9e76-38ea2ac839b6" containerID="482ad531cfb052e1221e4d870c3a34e8c880c336d1408fea9d6b4ebb169ccce4" exitCode=0
May 25 12:13:43.222771 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:43.222695    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xg5h" event={"ID":"f97986ac-3ba7-4762-9e76-38ea2ac839b6","Type":"ContainerDied","Data":"482ad531cfb052e1221e4d870c3a34e8c880c336d1408fea9d6b4ebb169ccce4"}
May 25 12:13:43.237355 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:43.237310    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d44874984-xhvxn" podStartSLOduration=25.67735937 podStartE2EDuration="35.237296168s" podCreationTimestamp="2026-05-25 12:13:08 +0000 UTC" firstStartedPulling="2026-05-25 12:13:32.775507089 +0000 UTC m=+34.416701844" lastFinishedPulling="2026-05-25 12:13:42.335443881 +0000 UTC m=+43.976638642" observedRunningTime="2026-05-25 12:13:43.236725654 +0000 UTC m=+44.877920432" watchObservedRunningTime="2026-05-25 12:13:43.237296168 +0000 UTC m=+44.878490944"
May 25 12:13:44.228636 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:44.228543    2578 generic.go:358] "Generic (PLEG): container finished" podID="f97986ac-3ba7-4762-9e76-38ea2ac839b6" containerID="d4fad190898d88d9d332171b9b46d2e93c416bf86243e532dc8b45478a06ae8c" exitCode=0
May 25 12:13:44.229087 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:44.228638    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xg5h" event={"ID":"f97986ac-3ba7-4762-9e76-38ea2ac839b6","Type":"ContainerDied","Data":"d4fad190898d88d9d332171b9b46d2e93c416bf86243e532dc8b45478a06ae8c"}
May 25 12:13:45.234426 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:45.234385    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xg5h" event={"ID":"f97986ac-3ba7-4762-9e76-38ea2ac839b6","Type":"ContainerStarted","Data":"7d15c647f96d2f9ca4365d669d3f7ef09fd1e06487f159c03f3f501750cd2b19"}
May 25 12:13:45.261531 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:45.261471    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5xg5h" podStartSLOduration=4.746383534 podStartE2EDuration="46.261451616s" podCreationTimestamp="2026-05-25 12:12:59 +0000 UTC" firstStartedPulling="2026-05-25 12:13:00.303858763 +0000 UTC m=+1.945053522" lastFinishedPulling="2026-05-25 12:13:41.818926833 +0000 UTC m=+43.460121604" observedRunningTime="2026-05-25 12:13:45.259672941 +0000 UTC m=+46.900867728" watchObservedRunningTime="2026-05-25 12:13:45.261451616 +0000 UTC m=+46.902646394"
May 25 12:13:48.075658 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:48.075626    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:13:48.076003 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:48.075685    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6bb75886-2392-4014-b64b-e61e092ffbd9-metrics-tls\") pod \"dns-default-xk9w8\" (UID: \"6bb75886-2392-4014-b64b-e61e092ffbd9\") " pod="openshift-dns/dns-default-xk9w8"
May 25 12:13:48.076003 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:48.075771    2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 12:13:48.076003 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:48.075791    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/587db4ef-f6a5-48c3-9e78-957d815002a4-cert\") pod \"ingress-canary-mzmd6\" (UID: \"587db4ef-f6a5-48c3-9e78-957d815002a4\") " pod="openshift-ingress-canary/ingress-canary-mzmd6"
May 25 12:13:48.076003 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:48.075793    2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f577cd989-k9p4z: secret "image-registry-tls" not found
May 25 12:13:48.076003 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:48.075894    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls podName:a42b2c14-3506-4a13-9c37-e917e26a9307 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:04.075879493 +0000 UTC m=+65.717074248 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls") pod "image-registry-6f577cd989-k9p4z" (UID: "a42b2c14-3506-4a13-9c37-e917e26a9307") : secret "image-registry-tls" not found
May 25 12:13:48.076003 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:48.075775    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:13:48.076003 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:48.075941    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bb75886-2392-4014-b64b-e61e092ffbd9-metrics-tls podName:6bb75886-2392-4014-b64b-e61e092ffbd9 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:04.075927711 +0000 UTC m=+65.717122472 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6bb75886-2392-4014-b64b-e61e092ffbd9-metrics-tls") pod "dns-default-xk9w8" (UID: "6bb75886-2392-4014-b64b-e61e092ffbd9") : secret "dns-default-metrics-tls" not found
May 25 12:13:48.076003 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:48.075837    2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:13:48.076003 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:13:48.075971    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587db4ef-f6a5-48c3-9e78-957d815002a4-cert podName:587db4ef-f6a5-48c3-9e78-957d815002a4 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:04.075963119 +0000 UTC m=+65.717157879 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/587db4ef-f6a5-48c3-9e78-957d815002a4-cert") pod "ingress-canary-mzmd6" (UID: "587db4ef-f6a5-48c3-9e78-957d815002a4") : secret "canary-serving-cert" not found
May 25 12:13:48.242587 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:48.242547    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8zd5f" event={"ID":"131acddf-521f-481d-84a3-b4e9ce65b58e","Type":"ContainerStarted","Data":"500ee981e093c895333b8768157cf496e0e8db0eba06c18129c6cefaaedb70bc"}
May 25 12:13:48.244340 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:48.244311    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567" event={"ID":"a651c313-a83a-44ed-9ee1-e86abaf28f6b","Type":"ContainerStarted","Data":"b815d54da91ef55f7a84e65ceec1e88be073e9871455002a8652e6afbe4c0fbe"}
May 25 12:13:48.244340 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:48.244342    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567" event={"ID":"a651c313-a83a-44ed-9ee1-e86abaf28f6b","Type":"ContainerStarted","Data":"5df326b1a6792d8ae0d0a02dc24b0c3f97e11c021e0e7470124a2457ffe85c75"}
May 25 12:13:48.265659 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:48.265617    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8zd5f" podStartSLOduration=33.466670169 podStartE2EDuration="38.265603151s" podCreationTimestamp="2026-05-25 12:13:10 +0000 UTC" firstStartedPulling="2026-05-25 12:13:42.780826017 +0000 UTC m=+44.422020776" lastFinishedPulling="2026-05-25 12:13:47.579759 +0000 UTC m=+49.220953758" observedRunningTime="2026-05-25 12:13:48.26520241 +0000 UTC m=+49.906397189" watchObservedRunningTime="2026-05-25 12:13:48.265603151 +0000 UTC m=+49.906797929"
May 25 12:13:58.196558 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:58.196517    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qt8dc"
May 25 12:13:58.231006 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:13:58.230948    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567" podStartSLOduration=35.404062793 podStartE2EDuration="50.230914159s" podCreationTimestamp="2026-05-25 12:13:08 +0000 UTC" firstStartedPulling="2026-05-25 12:13:32.7425854 +0000 UTC m=+34.383780170" lastFinishedPulling="2026-05-25 12:13:47.569436766 +0000 UTC m=+49.210631536" observedRunningTime="2026-05-25 12:13:48.293975396 +0000 UTC m=+49.935170173" watchObservedRunningTime="2026-05-25 12:13:58.230914159 +0000 UTC m=+59.872108933"
May 25 12:14:03.692106 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:14:03.692025    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-metrics-certs\") pod \"network-metrics-daemon-n9vrb\" (UID: \"02a94135-b85a-4ab1-9dd1-2ae75e77f2de\") " pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:14:03.703310 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:14:03.703286    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 12:14:03.713095 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:14:03.713052    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 25 12:14:03.713211 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:14:03.713138    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-metrics-certs podName:02a94135-b85a-4ab1-9dd1-2ae75e77f2de nodeName:}" failed. No retries permitted until 2026-05-25 12:15:07.713116156 +0000 UTC m=+129.354310913 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-metrics-certs") pod "network-metrics-daemon-n9vrb" (UID: "02a94135-b85a-4ab1-9dd1-2ae75e77f2de") : secret "metrics-daemon-secret" not found
May 25 12:14:03.793392 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:14:03.793351    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2rfpr\" (UniqueName: \"kubernetes.io/projected/cae409cd-445f-40b9-9851-c89b09c1ac43-kube-api-access-2rfpr\") pod \"network-check-target-zhvk8\" (UID: \"cae409cd-445f-40b9-9851-c89b09c1ac43\") " pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:14:03.796877 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:14:03.796854    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 12:14:03.807604 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:14:03.807582    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 12:14:03.817746 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:14:03.817722    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rfpr\" (UniqueName: \"kubernetes.io/projected/cae409cd-445f-40b9-9851-c89b09c1ac43-kube-api-access-2rfpr\") pod \"network-check-target-zhvk8\" (UID: \"cae409cd-445f-40b9-9851-c89b09c1ac43\") " pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:14:03.965854 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:14:03.965777    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nmk2h\""
May 25 12:14:03.973424 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:14:03.973405    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:14:04.096642 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:14:04.096603    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6bb75886-2392-4014-b64b-e61e092ffbd9-metrics-tls\") pod \"dns-default-xk9w8\" (UID: \"6bb75886-2392-4014-b64b-e61e092ffbd9\") " pod="openshift-dns/dns-default-xk9w8"
May 25 12:14:04.096642 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:14:04.096642    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/587db4ef-f6a5-48c3-9e78-957d815002a4-cert\") pod \"ingress-canary-mzmd6\" (UID: \"587db4ef-f6a5-48c3-9e78-957d815002a4\") " pod="openshift-ingress-canary/ingress-canary-mzmd6"
May 25 12:14:04.096812 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:14:04.096711    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:14:04.096812 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:14:04.096754    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:14:04.096812 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:14:04.096806    2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 12:14:04.096912 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:14:04.096817    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bb75886-2392-4014-b64b-e61e092ffbd9-metrics-tls podName:6bb75886-2392-4014-b64b-e61e092ffbd9 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:36.096796741 +0000 UTC m=+97.737991510 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6bb75886-2392-4014-b64b-e61e092ffbd9-metrics-tls") pod "dns-default-xk9w8" (UID: "6bb75886-2392-4014-b64b-e61e092ffbd9") : secret "dns-default-metrics-tls" not found
May 25 12:14:04.096912 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:14:04.096818    2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:14:04.096912 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:14:04.096818    2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f577cd989-k9p4z: secret "image-registry-tls" not found
May 25 12:14:04.096912 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:14:04.096882    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587db4ef-f6a5-48c3-9e78-957d815002a4-cert podName:587db4ef-f6a5-48c3-9e78-957d815002a4 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:36.096863711 +0000 UTC m=+97.738058480 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/587db4ef-f6a5-48c3-9e78-957d815002a4-cert") pod "ingress-canary-mzmd6" (UID: "587db4ef-f6a5-48c3-9e78-957d815002a4") : secret "canary-serving-cert" not found
May 25 12:14:04.097077 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:14:04.096922    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls podName:a42b2c14-3506-4a13-9c37-e917e26a9307 nodeName:}" failed. No retries permitted until 2026-05-25 12:14:36.096902897 +0000 UTC m=+97.738097657 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls") pod "image-registry-6f577cd989-k9p4z" (UID: "a42b2c14-3506-4a13-9c37-e917e26a9307") : secret "image-registry-tls" not found
May 25 12:14:04.099302 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:14:04.099278    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zhvk8"]
May 25 12:14:04.103260 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:14:04.103239    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcae409cd_445f_40b9_9851_c89b09c1ac43.slice/crio-fdd80cd71f246f1b634b501a7f971007fb2bb25e25480f0d64bf688f4a7a68bd WatchSource:0}: Error finding container fdd80cd71f246f1b634b501a7f971007fb2bb25e25480f0d64bf688f4a7a68bd: Status 404 returned error can't find the container with id fdd80cd71f246f1b634b501a7f971007fb2bb25e25480f0d64bf688f4a7a68bd
May 25 12:14:04.287575 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:14:04.287503    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zhvk8" event={"ID":"cae409cd-445f-40b9-9851-c89b09c1ac43","Type":"ContainerStarted","Data":"fdd80cd71f246f1b634b501a7f971007fb2bb25e25480f0d64bf688f4a7a68bd"}
May 25 12:14:07.297305 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:14:07.297269    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zhvk8" event={"ID":"cae409cd-445f-40b9-9851-c89b09c1ac43","Type":"ContainerStarted","Data":"45afbb3d514b14bbec18598be2c6d10d9e972915fc475615dd999645bd6b6439"}
May 25 12:14:07.297683 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:14:07.297413    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:14:07.316599 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:14:07.316556    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zhvk8" podStartSLOduration=65.644878233 podStartE2EDuration="1m8.316544674s" podCreationTimestamp="2026-05-25 12:12:59 +0000 UTC" firstStartedPulling="2026-05-25 12:14:04.105199325 +0000 UTC m=+65.746394081" lastFinishedPulling="2026-05-25 12:14:06.776865758 +0000 UTC m=+68.418060522" observedRunningTime="2026-05-25 12:14:07.31630936 +0000 UTC m=+68.957504138" watchObservedRunningTime="2026-05-25 12:14:07.316544674 +0000 UTC m=+68.957739445"
May 25 12:14:36.139362 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:14:36.139257    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:14:36.139787 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:14:36.139397    2578 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 25 12:14:36.139787 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:14:36.139417    2578 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f577cd989-k9p4z: secret "image-registry-tls" not found
May 25 12:14:36.139787 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:14:36.139427    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6bb75886-2392-4014-b64b-e61e092ffbd9-metrics-tls\") pod \"dns-default-xk9w8\" (UID: \"6bb75886-2392-4014-b64b-e61e092ffbd9\") " pod="openshift-dns/dns-default-xk9w8"
May 25 12:14:36.139787 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:14:36.139475    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls podName:a42b2c14-3506-4a13-9c37-e917e26a9307 nodeName:}" failed. No retries permitted until 2026-05-25 12:15:40.139460429 +0000 UTC m=+161.780655183 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls") pod "image-registry-6f577cd989-k9p4z" (UID: "a42b2c14-3506-4a13-9c37-e917e26a9307") : secret "image-registry-tls" not found
May 25 12:14:36.139787 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:14:36.139491    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/587db4ef-f6a5-48c3-9e78-957d815002a4-cert\") pod \"ingress-canary-mzmd6\" (UID: \"587db4ef-f6a5-48c3-9e78-957d815002a4\") " pod="openshift-ingress-canary/ingress-canary-mzmd6"
May 25 12:14:36.139787 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:14:36.139521    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 25 12:14:36.139787 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:14:36.139553    2578 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 25 12:14:36.139787 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:14:36.139579    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bb75886-2392-4014-b64b-e61e092ffbd9-metrics-tls podName:6bb75886-2392-4014-b64b-e61e092ffbd9 nodeName:}" failed. No retries permitted until 2026-05-25 12:15:40.139564139 +0000 UTC m=+161.780758895 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6bb75886-2392-4014-b64b-e61e092ffbd9-metrics-tls") pod "dns-default-xk9w8" (UID: "6bb75886-2392-4014-b64b-e61e092ffbd9") : secret "dns-default-metrics-tls" not found
May 25 12:14:36.139787 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:14:36.139596    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/587db4ef-f6a5-48c3-9e78-957d815002a4-cert podName:587db4ef-f6a5-48c3-9e78-957d815002a4 nodeName:}" failed. No retries permitted until 2026-05-25 12:15:40.139589415 +0000 UTC m=+161.780784170 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/587db4ef-f6a5-48c3-9e78-957d815002a4-cert") pod "ingress-canary-mzmd6" (UID: "587db4ef-f6a5-48c3-9e78-957d815002a4") : secret "canary-serving-cert" not found
May 25 12:14:38.302344 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:14:38.302312    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zhvk8"
May 25 12:14:53.008771 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:14:53.008742    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sg52n_02afa42e-73a7-4f9b-a5d7-609a53a507b1/dns-node-resolver/0.log"
May 25 12:14:53.800865 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:14:53.800837    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j2hhk_a34e5474-3229-4811-8b0e-dee727e7d8b7/node-ca/0.log"
May 25 12:15:02.544474 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:02.544427    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567" podUID="a651c313-a83a-44ed-9ee1-e86abaf28f6b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 25 12:15:07.766081 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:07.766023    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-metrics-certs\") pod \"network-metrics-daemon-n9vrb\" (UID: \"02a94135-b85a-4ab1-9dd1-2ae75e77f2de\") " pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:15:07.768368 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:07.768341    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02a94135-b85a-4ab1-9dd1-2ae75e77f2de-metrics-certs\") pod \"network-metrics-daemon-n9vrb\" (UID: \"02a94135-b85a-4ab1-9dd1-2ae75e77f2de\") " pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:15:07.873775 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:07.873741    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rrs9z\""
May 25 12:15:07.881131 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:07.881105    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9vrb"
May 25 12:15:08.005518 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:08.005486    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n9vrb"]
May 25 12:15:08.008249 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:15:08.008225    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a94135_b85a_4ab1_9dd1_2ae75e77f2de.slice/crio-344ab2498fa1fb142371b933088b6244a8de631c1cd1a6578288adc6ceb3fe55 WatchSource:0}: Error finding container 344ab2498fa1fb142371b933088b6244a8de631c1cd1a6578288adc6ceb3fe55: Status 404 returned error can't find the container with id 344ab2498fa1fb142371b933088b6244a8de631c1cd1a6578288adc6ceb3fe55
May 25 12:15:08.439461 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:08.439425    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n9vrb" event={"ID":"02a94135-b85a-4ab1-9dd1-2ae75e77f2de","Type":"ContainerStarted","Data":"344ab2498fa1fb142371b933088b6244a8de631c1cd1a6578288adc6ceb3fe55"}
May 25 12:15:10.444715 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:10.444673    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n9vrb" event={"ID":"02a94135-b85a-4ab1-9dd1-2ae75e77f2de","Type":"ContainerStarted","Data":"d4e256ff516ec9866cc522167e304eee1b3feb3396a53573ba1ffc969e6fc73e"}
May 25 12:15:10.444715 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:10.444712    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n9vrb" event={"ID":"02a94135-b85a-4ab1-9dd1-2ae75e77f2de","Type":"ContainerStarted","Data":"bd86990bf98136b2198f90b8bcfc75a66934a9398f1147a3026391ef523d48a6"}
May 25 12:15:10.463295 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:10.463248    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-n9vrb" podStartSLOduration=130.033527483 podStartE2EDuration="2m11.463234397s" podCreationTimestamp="2026-05-25 12:12:59 +0000 UTC" firstStartedPulling="2026-05-25 12:15:08.010440432 +0000 UTC m=+129.651635186" lastFinishedPulling="2026-05-25 12:15:09.440147346 +0000 UTC m=+131.081342100" observedRunningTime="2026-05-25 12:15:10.462754996 +0000 UTC m=+132.103949774" watchObservedRunningTime="2026-05-25 12:15:10.463234397 +0000 UTC m=+132.104429173"
May 25 12:15:12.544135 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:12.544092    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567" podUID="a651c313-a83a-44ed-9ee1-e86abaf28f6b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 25 12:15:13.940713 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:13.940681    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-49t8t"]
May 25 12:15:13.942646 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:13.942630    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-49t8t"
May 25 12:15:13.945048 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:13.945024    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 25 12:15:13.946214 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:13.946195    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 25 12:15:13.946283 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:13.946214    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-82qnj\""
May 25 12:15:13.946283 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:13.946229    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 25 12:15:13.946283 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:13.946269    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 25 12:15:13.951773 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:13.951752    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-49t8t"]
May 25 12:15:14.013425 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:14.013389    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a07290b-c095-4e6e-b86f-6d9abbbd9ce0-data-volume\") pod \"insights-runtime-extractor-49t8t\" (UID: \"0a07290b-c095-4e6e-b86f-6d9abbbd9ce0\") " pod="openshift-insights/insights-runtime-extractor-49t8t"
May 25 12:15:14.013425 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:14.013429    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnv8h\" (UniqueName: \"kubernetes.io/projected/0a07290b-c095-4e6e-b86f-6d9abbbd9ce0-kube-api-access-xnv8h\") pod \"insights-runtime-extractor-49t8t\" (UID: \"0a07290b-c095-4e6e-b86f-6d9abbbd9ce0\") " pod="openshift-insights/insights-runtime-extractor-49t8t"
May 25 12:15:14.013643 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:14.013461    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a07290b-c095-4e6e-b86f-6d9abbbd9ce0-crio-socket\") pod \"insights-runtime-extractor-49t8t\" (UID: \"0a07290b-c095-4e6e-b86f-6d9abbbd9ce0\") " pod="openshift-insights/insights-runtime-extractor-49t8t"
May 25 12:15:14.013643 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:14.013541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a07290b-c095-4e6e-b86f-6d9abbbd9ce0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-49t8t\" (UID: \"0a07290b-c095-4e6e-b86f-6d9abbbd9ce0\") " pod="openshift-insights/insights-runtime-extractor-49t8t"
May 25 12:15:14.013643 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:14.013564    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a07290b-c095-4e6e-b86f-6d9abbbd9ce0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-49t8t\" (UID: \"0a07290b-c095-4e6e-b86f-6d9abbbd9ce0\") " pod="openshift-insights/insights-runtime-extractor-49t8t"
May 25 12:15:14.114487 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:14.114446    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a07290b-c095-4e6e-b86f-6d9abbbd9ce0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-49t8t\" (UID: \"0a07290b-c095-4e6e-b86f-6d9abbbd9ce0\") " pod="openshift-insights/insights-runtime-extractor-49t8t"
May 25 12:15:14.114487 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:14.114494    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a07290b-c095-4e6e-b86f-6d9abbbd9ce0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-49t8t\" (UID: \"0a07290b-c095-4e6e-b86f-6d9abbbd9ce0\") " pod="openshift-insights/insights-runtime-extractor-49t8t"
May 25 12:15:14.114707 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:14.114551    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a07290b-c095-4e6e-b86f-6d9abbbd9ce0-data-volume\") pod \"insights-runtime-extractor-49t8t\" (UID: \"0a07290b-c095-4e6e-b86f-6d9abbbd9ce0\") " pod="openshift-insights/insights-runtime-extractor-49t8t"
May 25 12:15:14.114707 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:14.114570    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xnv8h\" (UniqueName: \"kubernetes.io/projected/0a07290b-c095-4e6e-b86f-6d9abbbd9ce0-kube-api-access-xnv8h\") pod \"insights-runtime-extractor-49t8t\" (UID: \"0a07290b-c095-4e6e-b86f-6d9abbbd9ce0\") " pod="openshift-insights/insights-runtime-extractor-49t8t"
May 25 12:15:14.114707 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:14.114605    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a07290b-c095-4e6e-b86f-6d9abbbd9ce0-crio-socket\") pod \"insights-runtime-extractor-49t8t\" (UID: \"0a07290b-c095-4e6e-b86f-6d9abbbd9ce0\") " pod="openshift-insights/insights-runtime-extractor-49t8t"
May 25 12:15:14.114707 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:14.114682    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0a07290b-c095-4e6e-b86f-6d9abbbd9ce0-crio-socket\") pod \"insights-runtime-extractor-49t8t\" (UID: \"0a07290b-c095-4e6e-b86f-6d9abbbd9ce0\") " pod="openshift-insights/insights-runtime-extractor-49t8t"
May 25 12:15:14.115306 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:14.115236    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0a07290b-c095-4e6e-b86f-6d9abbbd9ce0-data-volume\") pod \"insights-runtime-extractor-49t8t\" (UID: \"0a07290b-c095-4e6e-b86f-6d9abbbd9ce0\") " pod="openshift-insights/insights-runtime-extractor-49t8t"
May 25 12:15:14.115514 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:14.115487    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0a07290b-c095-4e6e-b86f-6d9abbbd9ce0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-49t8t\" (UID: \"0a07290b-c095-4e6e-b86f-6d9abbbd9ce0\") " pod="openshift-insights/insights-runtime-extractor-49t8t"
May 25 12:15:14.119232 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:14.119205    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0a07290b-c095-4e6e-b86f-6d9abbbd9ce0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-49t8t\" (UID: \"0a07290b-c095-4e6e-b86f-6d9abbbd9ce0\") " pod="openshift-insights/insights-runtime-extractor-49t8t"
May 25 12:15:14.125339 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:14.125315    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnv8h\" (UniqueName: \"kubernetes.io/projected/0a07290b-c095-4e6e-b86f-6d9abbbd9ce0-kube-api-access-xnv8h\") pod \"insights-runtime-extractor-49t8t\" (UID: \"0a07290b-c095-4e6e-b86f-6d9abbbd9ce0\") " pod="openshift-insights/insights-runtime-extractor-49t8t"
May 25 12:15:14.251682 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:14.251591    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-49t8t"
May 25 12:15:14.388975 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:14.388942    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-49t8t"]
May 25 12:15:14.392110 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:15:14.392086    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a07290b_c095_4e6e_b86f_6d9abbbd9ce0.slice/crio-8be5728f733b912cc3d73db79dfebd2e538943a09bb123d1d35531aadc3b8b60 WatchSource:0}: Error finding container 8be5728f733b912cc3d73db79dfebd2e538943a09bb123d1d35531aadc3b8b60: Status 404 returned error can't find the container with id 8be5728f733b912cc3d73db79dfebd2e538943a09bb123d1d35531aadc3b8b60
May 25 12:15:14.455214 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:14.455186    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-49t8t" event={"ID":"0a07290b-c095-4e6e-b86f-6d9abbbd9ce0","Type":"ContainerStarted","Data":"9a4b1f1333d830379dd5be8593c0f61ca31b7bbe67dda35886dfcd90fef3b04d"}
May 25 12:15:14.455338 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:14.455220    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-49t8t" event={"ID":"0a07290b-c095-4e6e-b86f-6d9abbbd9ce0","Type":"ContainerStarted","Data":"8be5728f733b912cc3d73db79dfebd2e538943a09bb123d1d35531aadc3b8b60"}
May 25 12:15:15.459259 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:15.459174    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-49t8t" event={"ID":"0a07290b-c095-4e6e-b86f-6d9abbbd9ce0","Type":"ContainerStarted","Data":"86b336fc27c92c4c81b16381ae977e619e2fdfe0cb802b85debac816542bb8e4"}
May 25 12:15:17.465490 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:17.465456    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-49t8t" event={"ID":"0a07290b-c095-4e6e-b86f-6d9abbbd9ce0","Type":"ContainerStarted","Data":"19910fe751df006dcc8b273b63bae10dda099f596a50560934765dbeeaccfa68"}
May 25 12:15:17.486536 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:17.486486    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-49t8t" podStartSLOduration=2.19921537 podStartE2EDuration="4.48646852s" podCreationTimestamp="2026-05-25 12:15:13 +0000 UTC" firstStartedPulling="2026-05-25 12:15:14.444827207 +0000 UTC m=+136.086021962" lastFinishedPulling="2026-05-25 12:15:16.732080356 +0000 UTC m=+138.373275112" observedRunningTime="2026-05-25 12:15:17.484957902 +0000 UTC m=+139.126152703" watchObservedRunningTime="2026-05-25 12:15:17.48646852 +0000 UTC m=+139.127663299"
May 25 12:15:22.543918 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:22.543876    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567" podUID="a651c313-a83a-44ed-9ee1-e86abaf28f6b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 25 12:15:22.544297 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:22.543949    2578 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567"
May 25 12:15:22.544469 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:22.544438    2578 kuberuntime_manager.go:1107] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"b815d54da91ef55f7a84e65ceec1e88be073e9871455002a8652e6afbe4c0fbe"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567" containerMessage="Container service-proxy failed liveness probe, will be restarted"
May 25 12:15:22.544507 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:22.544490    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567" podUID="a651c313-a83a-44ed-9ee1-e86abaf28f6b" containerName="service-proxy" containerID="cri-o://b815d54da91ef55f7a84e65ceec1e88be073e9871455002a8652e6afbe4c0fbe" gracePeriod=30
May 25 12:15:23.482891 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:23.482846    2578 generic.go:358] "Generic (PLEG): container finished" podID="a651c313-a83a-44ed-9ee1-e86abaf28f6b" containerID="b815d54da91ef55f7a84e65ceec1e88be073e9871455002a8652e6afbe4c0fbe" exitCode=2
May 25 12:15:23.483103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:23.482909    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567" event={"ID":"a651c313-a83a-44ed-9ee1-e86abaf28f6b","Type":"ContainerDied","Data":"b815d54da91ef55f7a84e65ceec1e88be073e9871455002a8652e6afbe4c0fbe"}
May 25 12:15:23.483103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:23.482941    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-65455b8849-fx567" event={"ID":"a651c313-a83a-44ed-9ee1-e86abaf28f6b","Type":"ContainerStarted","Data":"a4bdacf6705e81a2b22880d976020cbb2e175b6e0cc8bee0118283e62f1feafc"}
May 25 12:15:28.040493 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.040456    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ssq8w"]
May 25 12:15:28.043635 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.043618    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.045960 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.045936    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-85pp4\""
May 25 12:15:28.046332 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.046313    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 25 12:15:28.047180 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.047150    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 25 12:15:28.047180 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.047163    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 25 12:15:28.047303 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.047213    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 25 12:15:28.047303 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.047264    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 25 12:15:28.047303 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.047160    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 25 12:15:28.130014 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.129964    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6-node-exporter-accelerators-collector-config\") pod \"node-exporter-ssq8w\" (UID: \"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6\") " pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.130014 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.130018    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ssq8w\" (UID: \"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6\") " pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.130278 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.130108    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6-sys\") pod \"node-exporter-ssq8w\" (UID: \"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6\") " pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.130278 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.130136    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6-node-exporter-wtmp\") pod \"node-exporter-ssq8w\" (UID: \"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6\") " pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.130278 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.130164    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6-node-exporter-textfile\") pod \"node-exporter-ssq8w\" (UID: \"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6\") " pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.130278 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.130185    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6-root\") pod \"node-exporter-ssq8w\" (UID: \"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6\") " pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.130278 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.130208    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6-node-exporter-tls\") pod \"node-exporter-ssq8w\" (UID: \"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6\") " pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.130278 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.130253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6-metrics-client-ca\") pod \"node-exporter-ssq8w\" (UID: \"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6\") " pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.130454 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.130286    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9vwr\" (UniqueName: \"kubernetes.io/projected/bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6-kube-api-access-g9vwr\") pod \"node-exporter-ssq8w\" (UID: \"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6\") " pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.231706 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.231649    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g9vwr\" (UniqueName: \"kubernetes.io/projected/bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6-kube-api-access-g9vwr\") pod \"node-exporter-ssq8w\" (UID: \"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6\") " pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.231706 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.231713    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6-node-exporter-accelerators-collector-config\") pod \"node-exporter-ssq8w\" (UID: \"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6\") " pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.231941 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.231804    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ssq8w\" (UID: \"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6\") " pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.231941 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.231852    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6-sys\") pod \"node-exporter-ssq8w\" (UID: \"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6\") " pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.231941 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.231869    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6-node-exporter-wtmp\") pod \"node-exporter-ssq8w\" (UID: \"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6\") " pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.231941 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.231917    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6-node-exporter-textfile\") pod \"node-exporter-ssq8w\" (UID: \"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6\") " pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.232163 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.231959    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6-root\") pod \"node-exporter-ssq8w\" (UID: \"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6\") " pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.232163 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.231993    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6-node-exporter-tls\") pod \"node-exporter-ssq8w\" (UID: \"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6\") " pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.232163 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.232013    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6-node-exporter-wtmp\") pod \"node-exporter-ssq8w\" (UID: \"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6\") " pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.232163 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.232018    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6-metrics-client-ca\") pod \"node-exporter-ssq8w\" (UID: \"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6\") " pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.232163 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.232018    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6-sys\") pod \"node-exporter-ssq8w\" (UID: \"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6\") " pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.232163 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.232041    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6-root\") pod \"node-exporter-ssq8w\" (UID: \"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6\") " pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.232419 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.232282    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6-node-exporter-textfile\") pod \"node-exporter-ssq8w\" (UID: \"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6\") " pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.232419 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.232352    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6-node-exporter-accelerators-collector-config\") pod \"node-exporter-ssq8w\" (UID: \"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6\") " pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.232499 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.232441    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6-metrics-client-ca\") pod \"node-exporter-ssq8w\" (UID: \"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6\") " pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.234187 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.234165    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ssq8w\" (UID: \"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6\") " pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.234285 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.234235    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6-node-exporter-tls\") pod \"node-exporter-ssq8w\" (UID: \"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6\") " pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.240893 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.240870    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9vwr\" (UniqueName: \"kubernetes.io/projected/bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6-kube-api-access-g9vwr\") pod \"node-exporter-ssq8w\" (UID: \"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6\") " pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.352255 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.352174    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ssq8w"
May 25 12:15:28.360877 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:15:28.360841    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd0f4aaa_5a16_4c9e_8f43_8bc70809cdd6.slice/crio-5849f6c2a2f8b783c7a2353d7c625c884d6cef34d1910e6681d0946de8761696 WatchSource:0}: Error finding container 5849f6c2a2f8b783c7a2353d7c625c884d6cef34d1910e6681d0946de8761696: Status 404 returned error can't find the container with id 5849f6c2a2f8b783c7a2353d7c625c884d6cef34d1910e6681d0946de8761696
May 25 12:15:28.495793 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:28.495756    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ssq8w" event={"ID":"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6","Type":"ContainerStarted","Data":"5849f6c2a2f8b783c7a2353d7c625c884d6cef34d1910e6681d0946de8761696"}
May 25 12:15:29.500112 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:29.500051    2578 generic.go:358] "Generic (PLEG): container finished" podID="bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6" containerID="5cd4b84ddbf294f3d55e92762ab3dde623a987d6ee040fcd78dceab1830d54c7" exitCode=0
May 25 12:15:29.500112 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:29.500091    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ssq8w" event={"ID":"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6","Type":"ContainerDied","Data":"5cd4b84ddbf294f3d55e92762ab3dde623a987d6ee040fcd78dceab1830d54c7"}
May 25 12:15:30.504666 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:30.504629    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ssq8w" event={"ID":"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6","Type":"ContainerStarted","Data":"a4a3f68321618e3078d217889685acaa2717c7dfd24cd022f1d15e5060952f1c"}
May 25 12:15:30.504666 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:30.504669    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ssq8w" event={"ID":"bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6","Type":"ContainerStarted","Data":"0df1651f445211d9985d9f235cf6fb72713b65290a4791a2f2fd370cd6334469"}
May 25 12:15:35.267392 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:15:35.267346    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6f577cd989-k9p4z" podUID="a42b2c14-3506-4a13-9c37-e917e26a9307"
May 25 12:15:35.283547 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:15:35.283510    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-xk9w8" podUID="6bb75886-2392-4014-b64b-e61e092ffbd9"
May 25 12:15:35.289678 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:15:35.289646    2578 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-mzmd6" podUID="587db4ef-f6a5-48c3-9e78-957d815002a4"
May 25 12:15:35.517260 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:35.517228    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:15:35.517260 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:35.517243    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xk9w8"
May 25 12:15:35.517524 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:35.517228    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mzmd6"
May 25 12:15:40.234328 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:40.234292    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:15:40.234328 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:40.234338    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/587db4ef-f6a5-48c3-9e78-957d815002a4-cert\") pod \"ingress-canary-mzmd6\" (UID: \"587db4ef-f6a5-48c3-9e78-957d815002a4\") " pod="openshift-ingress-canary/ingress-canary-mzmd6"
May 25 12:15:40.234850 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:40.234374    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6bb75886-2392-4014-b64b-e61e092ffbd9-metrics-tls\") pod \"dns-default-xk9w8\" (UID: \"6bb75886-2392-4014-b64b-e61e092ffbd9\") " pod="openshift-dns/dns-default-xk9w8"
May 25 12:15:40.236778 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:40.236751    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6bb75886-2392-4014-b64b-e61e092ffbd9-metrics-tls\") pod \"dns-default-xk9w8\" (UID: \"6bb75886-2392-4014-b64b-e61e092ffbd9\") " pod="openshift-dns/dns-default-xk9w8"
May 25 12:15:40.236905 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:40.236816    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/587db4ef-f6a5-48c3-9e78-957d815002a4-cert\") pod \"ingress-canary-mzmd6\" (UID: \"587db4ef-f6a5-48c3-9e78-957d815002a4\") " pod="openshift-ingress-canary/ingress-canary-mzmd6"
May 25 12:15:40.236905 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:40.236880    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls\") pod \"image-registry-6f577cd989-k9p4z\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") " pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:15:40.321244 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:40.321204    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-sw9bq\""
May 25 12:15:40.321244 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:40.321232    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s7xbv\""
May 25 12:15:40.321466 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:40.321254    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c8dc6\""
May 25 12:15:40.328317 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:40.328295    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mzmd6"
May 25 12:15:40.328371 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:40.328315    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xk9w8"
May 25 12:15:40.328497 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:40.328479    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:15:40.502813 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:40.502700    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ssq8w" podStartSLOduration=11.572179452 podStartE2EDuration="12.502681464s" podCreationTimestamp="2026-05-25 12:15:28 +0000 UTC" firstStartedPulling="2026-05-25 12:15:28.362968231 +0000 UTC m=+150.004163000" lastFinishedPulling="2026-05-25 12:15:29.293470246 +0000 UTC m=+150.934665012" observedRunningTime="2026-05-25 12:15:30.526755025 +0000 UTC m=+152.167949801" watchObservedRunningTime="2026-05-25 12:15:40.502681464 +0000 UTC m=+162.143876249"
May 25 12:15:40.503867 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:40.503842    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xk9w8"]
May 25 12:15:40.506688 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:15:40.506655    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb75886_2392_4014_b64b_e61e092ffbd9.slice/crio-5724aef977bf987cb47c9a1de9b8a5ded92df2d783ee5cbbb28c0463b4f0e08a WatchSource:0}: Error finding container 5724aef977bf987cb47c9a1de9b8a5ded92df2d783ee5cbbb28c0463b4f0e08a: Status 404 returned error can't find the container with id 5724aef977bf987cb47c9a1de9b8a5ded92df2d783ee5cbbb28c0463b4f0e08a
May 25 12:15:40.530268 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:40.530232    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xk9w8" event={"ID":"6bb75886-2392-4014-b64b-e61e092ffbd9","Type":"ContainerStarted","Data":"5724aef977bf987cb47c9a1de9b8a5ded92df2d783ee5cbbb28c0463b4f0e08a"}
May 25 12:15:40.699867 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:40.699833    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mzmd6"]
May 25 12:15:40.703092 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:40.702988    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f577cd989-k9p4z"]
May 25 12:15:40.703269 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:15:40.703242    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587db4ef_f6a5_48c3_9e78_957d815002a4.slice/crio-e64e9902141b316816c39e21b40f45daa6471f28f8bdc3c401cc593bb5af0b74 WatchSource:0}: Error finding container e64e9902141b316816c39e21b40f45daa6471f28f8bdc3c401cc593bb5af0b74: Status 404 returned error can't find the container with id e64e9902141b316816c39e21b40f45daa6471f28f8bdc3c401cc593bb5af0b74
May 25 12:15:40.706924 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:15:40.706897    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda42b2c14_3506_4a13_9c37_e917e26a9307.slice/crio-cab26512c28eaf1e58dee150ea236a6890830565cc508b87860f38969e234a41 WatchSource:0}: Error finding container cab26512c28eaf1e58dee150ea236a6890830565cc508b87860f38969e234a41: Status 404 returned error can't find the container with id cab26512c28eaf1e58dee150ea236a6890830565cc508b87860f38969e234a41
May 25 12:15:41.535381 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:41.535328    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mzmd6" event={"ID":"587db4ef-f6a5-48c3-9e78-957d815002a4","Type":"ContainerStarted","Data":"e64e9902141b316816c39e21b40f45daa6471f28f8bdc3c401cc593bb5af0b74"}
May 25 12:15:41.536915 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:41.536870    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f577cd989-k9p4z" event={"ID":"a42b2c14-3506-4a13-9c37-e917e26a9307","Type":"ContainerStarted","Data":"5049109c826eb181e0c25ae3f7c1818a24be626c182a4015a6590f4f8e00ec64"}
May 25 12:15:41.536915 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:41.536909    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f577cd989-k9p4z" event={"ID":"a42b2c14-3506-4a13-9c37-e917e26a9307","Type":"ContainerStarted","Data":"cab26512c28eaf1e58dee150ea236a6890830565cc508b87860f38969e234a41"}
May 25 12:15:41.537143 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:41.537002    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:15:42.540846 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:42.540820    2578 generic.go:358] "Generic (PLEG): container finished" podID="00a70888-8d3b-47a2-b012-fa50c61fd91d" containerID="752e4c68d6ecbfa1b74abafc14cc1946de4d946291af2a0645c986618a5ec462" exitCode=255
May 25 12:15:42.541229 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:42.540902    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d77d4df8-q8k4f" event={"ID":"00a70888-8d3b-47a2-b012-fa50c61fd91d","Type":"ContainerDied","Data":"752e4c68d6ecbfa1b74abafc14cc1946de4d946291af2a0645c986618a5ec462"}
May 25 12:15:42.541337 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:42.541320    2578 scope.go:117] "RemoveContainer" containerID="752e4c68d6ecbfa1b74abafc14cc1946de4d946291af2a0645c986618a5ec462"
May 25 12:15:42.544858 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:42.544297    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xk9w8" event={"ID":"6bb75886-2392-4014-b64b-e61e092ffbd9","Type":"ContainerStarted","Data":"b3bffce4e9aef1012b1318ddf55f71bd6c29eba328c09cf7bb1a50baa1a51553"}
May 25 12:15:42.550895 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:42.550871    2578 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d77d4df8-q8k4f"
May 25 12:15:42.559563 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:42.559506    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6f577cd989-k9p4z" podStartSLOduration=163.55948464 podStartE2EDuration="2m43.55948464s" podCreationTimestamp="2026-05-25 12:12:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:15:41.559429257 +0000 UTC m=+163.200624035" watchObservedRunningTime="2026-05-25 12:15:42.55948464 +0000 UTC m=+164.200679419"
May 25 12:15:42.577323 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:42.577279    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d44874984-xhvxn" podUID="5f49b1b6-b6fb-4d59-9e56-163789263c0e" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.9:8000/healthz\": dial tcp 10.132.0.9:8000: connect: connection refused"
May 25 12:15:43.220307 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:43.220264    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d44874984-xhvxn" podUID="5f49b1b6-b6fb-4d59-9e56-163789263c0e" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.9:8000/readyz\": dial tcp 10.132.0.9:8000: connect: connection refused"
May 25 12:15:43.547672 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:43.547635    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mzmd6" event={"ID":"587db4ef-f6a5-48c3-9e78-957d815002a4","Type":"ContainerStarted","Data":"8913b35fe2e45ccc1f91710eafaa722c9b5c4a95135e163f1fbb2f36fbbcc618"}
May 25 12:15:43.548937 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:43.548912    2578 generic.go:358] "Generic (PLEG): container finished" podID="5f49b1b6-b6fb-4d59-9e56-163789263c0e" containerID="8614720c0194789be0a6896d36c6c28e1edf3ae18d98bff26702b95cd0dc3cdd" exitCode=1
May 25 12:15:43.549086 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:43.548985    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d44874984-xhvxn" event={"ID":"5f49b1b6-b6fb-4d59-9e56-163789263c0e","Type":"ContainerDied","Data":"8614720c0194789be0a6896d36c6c28e1edf3ae18d98bff26702b95cd0dc3cdd"}
May 25 12:15:43.549311 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:43.549283    2578 scope.go:117] "RemoveContainer" containerID="8614720c0194789be0a6896d36c6c28e1edf3ae18d98bff26702b95cd0dc3cdd"
May 25 12:15:43.550599 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:43.550576    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67d77d4df8-q8k4f" event={"ID":"00a70888-8d3b-47a2-b012-fa50c61fd91d","Type":"ContainerStarted","Data":"81ab17a43067538f509358c877edd1f1294a9dcbc74ee2321caf2e0508d7bd72"}
May 25 12:15:43.552111 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:43.552090    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xk9w8" event={"ID":"6bb75886-2392-4014-b64b-e61e092ffbd9","Type":"ContainerStarted","Data":"a348dd60dec1b4e8646b60dc67c58281d9fe0d828e9139e5eb4d1c416c1d2459"}
May 25 12:15:43.552276 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:43.552257    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xk9w8"
May 25 12:15:43.569443 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:43.569386    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mzmd6" podStartSLOduration=129.764692306 podStartE2EDuration="2m11.569366318s" podCreationTimestamp="2026-05-25 12:13:32 +0000 UTC" firstStartedPulling="2026-05-25 12:15:40.705477222 +0000 UTC m=+162.346671977" lastFinishedPulling="2026-05-25 12:15:42.510151231 +0000 UTC m=+164.151345989" observedRunningTime="2026-05-25 12:15:43.568010344 +0000 UTC m=+165.209205133" watchObservedRunningTime="2026-05-25 12:15:43.569366318 +0000 UTC m=+165.210561096"
May 25 12:15:43.623120 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:43.623045    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-xk9w8" podStartSLOduration=130.201807116 podStartE2EDuration="2m11.623021731s" podCreationTimestamp="2026-05-25 12:13:32 +0000 UTC" firstStartedPulling="2026-05-25 12:15:40.50856231 +0000 UTC m=+162.149757066" lastFinishedPulling="2026-05-25 12:15:41.929776924 +0000 UTC m=+163.570971681" observedRunningTime="2026-05-25 12:15:43.601725731 +0000 UTC m=+165.242920508" watchObservedRunningTime="2026-05-25 12:15:43.623021731 +0000 UTC m=+165.264216510"
May 25 12:15:44.559231 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:44.559191    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d44874984-xhvxn" event={"ID":"5f49b1b6-b6fb-4d59-9e56-163789263c0e","Type":"ContainerStarted","Data":"664399cef1df690321999780438cc91ac0e2e223e6fad8ae103e31c25feb80d1"}
May 25 12:15:44.559698 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:44.559624    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d44874984-xhvxn"
May 25 12:15:44.560693 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:44.560673    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d44874984-xhvxn"
May 25 12:15:53.562150 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:15:53.562119    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xk9w8"
May 25 12:16:00.332734 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:00.332696    2578 patch_prober.go:28] interesting pod/image-registry-6f577cd989-k9p4z container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 25 12:16:00.333123 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:00.332767    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6f577cd989-k9p4z" podUID="a42b2c14-3506-4a13-9c37-e917e26a9307" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:16:02.548163 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:02.548134    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:16:04.403130 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:04.403087    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f577cd989-k9p4z"]
May 25 12:16:29.422253 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:29.422193    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6f577cd989-k9p4z" podUID="a42b2c14-3506-4a13-9c37-e917e26a9307" containerName="registry" containerID="cri-o://5049109c826eb181e0c25ae3f7c1818a24be626c182a4015a6590f4f8e00ec64" gracePeriod=30
May 25 12:16:30.658309 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.658283    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:16:30.681001 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.680965    2578 generic.go:358] "Generic (PLEG): container finished" podID="a42b2c14-3506-4a13-9c37-e917e26a9307" containerID="5049109c826eb181e0c25ae3f7c1818a24be626c182a4015a6590f4f8e00ec64" exitCode=0
May 25 12:16:30.681209 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.681032    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f577cd989-k9p4z"
May 25 12:16:30.681209 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.681078    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f577cd989-k9p4z" event={"ID":"a42b2c14-3506-4a13-9c37-e917e26a9307","Type":"ContainerDied","Data":"5049109c826eb181e0c25ae3f7c1818a24be626c182a4015a6590f4f8e00ec64"}
May 25 12:16:30.681209 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.681124    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f577cd989-k9p4z" event={"ID":"a42b2c14-3506-4a13-9c37-e917e26a9307","Type":"ContainerDied","Data":"cab26512c28eaf1e58dee150ea236a6890830565cc508b87860f38969e234a41"}
May 25 12:16:30.681209 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.681158    2578 scope.go:117] "RemoveContainer" containerID="5049109c826eb181e0c25ae3f7c1818a24be626c182a4015a6590f4f8e00ec64"
May 25 12:16:30.692176 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.692147    2578 scope.go:117] "RemoveContainer" containerID="5049109c826eb181e0c25ae3f7c1818a24be626c182a4015a6590f4f8e00ec64"
May 25 12:16:30.692599 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:16:30.692567    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5049109c826eb181e0c25ae3f7c1818a24be626c182a4015a6590f4f8e00ec64\": container with ID starting with 5049109c826eb181e0c25ae3f7c1818a24be626c182a4015a6590f4f8e00ec64 not found: ID does not exist" containerID="5049109c826eb181e0c25ae3f7c1818a24be626c182a4015a6590f4f8e00ec64"
May 25 12:16:30.692709 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.692613    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5049109c826eb181e0c25ae3f7c1818a24be626c182a4015a6590f4f8e00ec64"} err="failed to get container status \"5049109c826eb181e0c25ae3f7c1818a24be626c182a4015a6590f4f8e00ec64\": rpc error: code = NotFound desc = could not find container \"5049109c826eb181e0c25ae3f7c1818a24be626c182a4015a6590f4f8e00ec64\": container with ID starting with 5049109c826eb181e0c25ae3f7c1818a24be626c182a4015a6590f4f8e00ec64 not found: ID does not exist"
May 25 12:16:30.698853 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.698830    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls\") pod \"a42b2c14-3506-4a13-9c37-e917e26a9307\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") "
May 25 12:16:30.698945 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.698872    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a42b2c14-3506-4a13-9c37-e917e26a9307-ca-trust-extracted\") pod \"a42b2c14-3506-4a13-9c37-e917e26a9307\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") "
May 25 12:16:30.698945 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.698904    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-bound-sa-token\") pod \"a42b2c14-3506-4a13-9c37-e917e26a9307\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") "
May 25 12:16:30.699070 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.698953    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svjn5\" (UniqueName: \"kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-kube-api-access-svjn5\") pod \"a42b2c14-3506-4a13-9c37-e917e26a9307\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") "
May 25 12:16:30.699070 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.698992    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a42b2c14-3506-4a13-9c37-e917e26a9307-installation-pull-secrets\") pod \"a42b2c14-3506-4a13-9c37-e917e26a9307\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") "
May 25 12:16:30.699070 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.699019    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a42b2c14-3506-4a13-9c37-e917e26a9307-trusted-ca\") pod \"a42b2c14-3506-4a13-9c37-e917e26a9307\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") "
May 25 12:16:30.699219 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.699094    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-certificates\") pod \"a42b2c14-3506-4a13-9c37-e917e26a9307\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") "
May 25 12:16:30.699219 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.699121    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a42b2c14-3506-4a13-9c37-e917e26a9307-image-registry-private-configuration\") pod \"a42b2c14-3506-4a13-9c37-e917e26a9307\" (UID: \"a42b2c14-3506-4a13-9c37-e917e26a9307\") "
May 25 12:16:30.699711 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.699680    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a42b2c14-3506-4a13-9c37-e917e26a9307-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a42b2c14-3506-4a13-9c37-e917e26a9307" (UID: "a42b2c14-3506-4a13-9c37-e917e26a9307"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:16:30.699842 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.699789    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a42b2c14-3506-4a13-9c37-e917e26a9307" (UID: "a42b2c14-3506-4a13-9c37-e917e26a9307"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:16:30.702073 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.702026    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a42b2c14-3506-4a13-9c37-e917e26a9307" (UID: "a42b2c14-3506-4a13-9c37-e917e26a9307"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:16:30.702200 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.702137    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a42b2c14-3506-4a13-9c37-e917e26a9307-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a42b2c14-3506-4a13-9c37-e917e26a9307" (UID: "a42b2c14-3506-4a13-9c37-e917e26a9307"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:16:30.702326 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.702191    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a42b2c14-3506-4a13-9c37-e917e26a9307" (UID: "a42b2c14-3506-4a13-9c37-e917e26a9307"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:16:30.702387 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.702329    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-kube-api-access-svjn5" (OuterVolumeSpecName: "kube-api-access-svjn5") pod "a42b2c14-3506-4a13-9c37-e917e26a9307" (UID: "a42b2c14-3506-4a13-9c37-e917e26a9307"). InnerVolumeSpecName "kube-api-access-svjn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:16:30.702857 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.702829    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a42b2c14-3506-4a13-9c37-e917e26a9307-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a42b2c14-3506-4a13-9c37-e917e26a9307" (UID: "a42b2c14-3506-4a13-9c37-e917e26a9307"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:16:30.708045 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.708023    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a42b2c14-3506-4a13-9c37-e917e26a9307-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a42b2c14-3506-4a13-9c37-e917e26a9307" (UID: "a42b2c14-3506-4a13-9c37-e917e26a9307"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:16:30.799818 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.799780    2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-certificates\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:16:30.799818 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.799817    2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a42b2c14-3506-4a13-9c37-e917e26a9307-image-registry-private-configuration\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:16:30.800035 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.799832    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-registry-tls\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:16:30.800035 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.799845    2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a42b2c14-3506-4a13-9c37-e917e26a9307-ca-trust-extracted\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:16:30.800035 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.799857    2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-bound-sa-token\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:16:30.800035 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.799869    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svjn5\" (UniqueName: \"kubernetes.io/projected/a42b2c14-3506-4a13-9c37-e917e26a9307-kube-api-access-svjn5\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:16:30.800035 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.799881    2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a42b2c14-3506-4a13-9c37-e917e26a9307-installation-pull-secrets\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:16:30.800035 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.799922    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a42b2c14-3506-4a13-9c37-e917e26a9307-trusted-ca\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:16:30.998616 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:30.998582    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f577cd989-k9p4z"]
May 25 12:16:31.002401 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:31.002378    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f577cd989-k9p4z"]
May 25 12:16:32.939908 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:16:32.939873    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a42b2c14-3506-4a13-9c37-e917e26a9307" path="/var/lib/kubelet/pods/a42b2c14-3506-4a13-9c37-e917e26a9307/volumes"
May 25 12:17:57.567790 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:17:57.567741    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-h5ggj"]
May 25 12:17:57.568386 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:17:57.568111    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a42b2c14-3506-4a13-9c37-e917e26a9307" containerName="registry"
May 25 12:17:57.568386 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:17:57.568129    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a42b2c14-3506-4a13-9c37-e917e26a9307" containerName="registry"
May 25 12:17:57.568386 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:17:57.568187    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="a42b2c14-3506-4a13-9c37-e917e26a9307" containerName="registry"
May 25 12:17:57.570530 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:17:57.570508    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-h5ggj"
May 25 12:17:57.574344 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:17:57.574316    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
May 25 12:17:57.574488 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:17:57.574316    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
May 25 12:17:57.574488 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:17:57.574373    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-d9dg2\""
May 25 12:17:57.574488 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:17:57.574386    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
May 25 12:17:57.584602 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:17:57.584569    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-h5ggj"]
May 25 12:17:57.701016 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:17:57.697621    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/6cb3ad3b-b2de-41a0-b685-3f5189f3d611-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-h5ggj\" (UID: \"6cb3ad3b-b2de-41a0-b685-3f5189f3d611\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-h5ggj"
May 25 12:17:57.701016 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:17:57.697736    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk6dh\" (UniqueName: \"kubernetes.io/projected/6cb3ad3b-b2de-41a0-b685-3f5189f3d611-kube-api-access-tk6dh\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-h5ggj\" (UID: \"6cb3ad3b-b2de-41a0-b685-3f5189f3d611\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-h5ggj"
May 25 12:17:57.798660 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:17:57.798626    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/6cb3ad3b-b2de-41a0-b685-3f5189f3d611-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-h5ggj\" (UID: \"6cb3ad3b-b2de-41a0-b685-3f5189f3d611\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-h5ggj"
May 25 12:17:57.798797 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:17:57.798670    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tk6dh\" (UniqueName: \"kubernetes.io/projected/6cb3ad3b-b2de-41a0-b685-3f5189f3d611-kube-api-access-tk6dh\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-h5ggj\" (UID: \"6cb3ad3b-b2de-41a0-b685-3f5189f3d611\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-h5ggj"
May 25 12:17:57.800903 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:17:57.800879    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/6cb3ad3b-b2de-41a0-b685-3f5189f3d611-certificates\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-h5ggj\" (UID: \"6cb3ad3b-b2de-41a0-b685-3f5189f3d611\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-h5ggj"
May 25 12:17:57.808647 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:17:57.808620    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk6dh\" (UniqueName: \"kubernetes.io/projected/6cb3ad3b-b2de-41a0-b685-3f5189f3d611-kube-api-access-tk6dh\") pod \"custom-metrics-autoscaler-operator-bbf89fd5d-h5ggj\" (UID: \"6cb3ad3b-b2de-41a0-b685-3f5189f3d611\") " pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-h5ggj"
May 25 12:17:57.881216 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:17:57.881150    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-h5ggj"
May 25 12:17:58.015590 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:17:58.015410    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-h5ggj"]
May 25 12:17:58.018404 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:17:58.018372    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb3ad3b_b2de_41a0_b685_3f5189f3d611.slice/crio-a8372bed5c58df4fabd2c9b404c15de54e030dab27b86895a623a2e61d927bf5 WatchSource:0}: Error finding container a8372bed5c58df4fabd2c9b404c15de54e030dab27b86895a623a2e61d927bf5: Status 404 returned error can't find the container with id a8372bed5c58df4fabd2c9b404c15de54e030dab27b86895a623a2e61d927bf5
May 25 12:17:58.881001 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:17:58.880967    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/ovn-acl-logging/0.log"
May 25 12:17:58.881464 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:17:58.881184    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/ovn-acl-logging/0.log"
May 25 12:17:58.887609 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:17:58.887588    2578 kubelet.go:1628] "Image garbage collection succeeded"
May 25 12:17:58.905503 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:17:58.905471    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-h5ggj" event={"ID":"6cb3ad3b-b2de-41a0-b685-3f5189f3d611","Type":"ContainerStarted","Data":"a8372bed5c58df4fabd2c9b404c15de54e030dab27b86895a623a2e61d927bf5"}
May 25 12:18:06.477505 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:06.477468    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-vxjbb"]
May 25 12:18:06.480581 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:06.480557    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-vxjbb"
May 25 12:18:06.483388 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:06.483363    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-xj8ll\""
May 25 12:18:06.483518 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:06.483369    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
May 25 12:18:06.483518 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:06.483397    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
May 25 12:18:06.489074 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:06.489031    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-vxjbb"]
May 25 12:18:06.559735 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:06.559697    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/0c271934-1560-48c4-a0af-e48574720e47-cabundle0\") pod \"keda-operator-ffbb595cb-vxjbb\" (UID: \"0c271934-1560-48c4-a0af-e48574720e47\") " pod="openshift-keda/keda-operator-ffbb595cb-vxjbb"
May 25 12:18:06.559735 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:06.559738    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln4mg\" (UniqueName: \"kubernetes.io/projected/0c271934-1560-48c4-a0af-e48574720e47-kube-api-access-ln4mg\") pod \"keda-operator-ffbb595cb-vxjbb\" (UID: \"0c271934-1560-48c4-a0af-e48574720e47\") " pod="openshift-keda/keda-operator-ffbb595cb-vxjbb"
May 25 12:18:06.559976 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:06.559763    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0c271934-1560-48c4-a0af-e48574720e47-certificates\") pod \"keda-operator-ffbb595cb-vxjbb\" (UID: \"0c271934-1560-48c4-a0af-e48574720e47\") " pod="openshift-keda/keda-operator-ffbb595cb-vxjbb"
May 25 12:18:06.660515 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:06.660472    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/0c271934-1560-48c4-a0af-e48574720e47-cabundle0\") pod \"keda-operator-ffbb595cb-vxjbb\" (UID: \"0c271934-1560-48c4-a0af-e48574720e47\") " pod="openshift-keda/keda-operator-ffbb595cb-vxjbb"
May 25 12:18:06.660721 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:06.660520    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ln4mg\" (UniqueName: \"kubernetes.io/projected/0c271934-1560-48c4-a0af-e48574720e47-kube-api-access-ln4mg\") pod \"keda-operator-ffbb595cb-vxjbb\" (UID: \"0c271934-1560-48c4-a0af-e48574720e47\") " pod="openshift-keda/keda-operator-ffbb595cb-vxjbb"
May 25 12:18:06.660721 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:06.660550    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0c271934-1560-48c4-a0af-e48574720e47-certificates\") pod \"keda-operator-ffbb595cb-vxjbb\" (UID: \"0c271934-1560-48c4-a0af-e48574720e47\") " pod="openshift-keda/keda-operator-ffbb595cb-vxjbb"
May 25 12:18:06.660721 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:18:06.660715    2578 projected.go:264] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
May 25 12:18:06.660885 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:18:06.660735    2578 secret.go:281] references non-existent secret key: ca.crt
May 25 12:18:06.660885 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:18:06.660746    2578 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 25 12:18:06.660885 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:18:06.660762    2578 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-vxjbb: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
May 25 12:18:06.660885 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:18:06.660854    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c271934-1560-48c4-a0af-e48574720e47-certificates podName:0c271934-1560-48c4-a0af-e48574720e47 nodeName:}" failed. No retries permitted until 2026-05-25 12:18:07.160832265 +0000 UTC m=+308.802027026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/0c271934-1560-48c4-a0af-e48574720e47-certificates") pod "keda-operator-ffbb595cb-vxjbb" (UID: "0c271934-1560-48c4-a0af-e48574720e47") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
May 25 12:18:06.661282 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:06.661262    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/0c271934-1560-48c4-a0af-e48574720e47-cabundle0\") pod \"keda-operator-ffbb595cb-vxjbb\" (UID: \"0c271934-1560-48c4-a0af-e48574720e47\") " pod="openshift-keda/keda-operator-ffbb595cb-vxjbb"
May 25 12:18:06.669690 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:06.669665    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln4mg\" (UniqueName: \"kubernetes.io/projected/0c271934-1560-48c4-a0af-e48574720e47-kube-api-access-ln4mg\") pod \"keda-operator-ffbb595cb-vxjbb\" (UID: \"0c271934-1560-48c4-a0af-e48574720e47\") " pod="openshift-keda/keda-operator-ffbb595cb-vxjbb"
May 25 12:18:06.927974 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:06.927933    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-h5ggj" event={"ID":"6cb3ad3b-b2de-41a0-b685-3f5189f3d611","Type":"ContainerStarted","Data":"f9a83b0f21527923fe6bad10cb9f9797ad9f6828f58e61ef6406eb6ac56a851d"}
May 25 12:18:06.928168 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:06.928002    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-h5ggj"
May 25 12:18:06.949669 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:06.949616    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-h5ggj" podStartSLOduration=2.011612746 podStartE2EDuration="9.949570822s" podCreationTimestamp="2026-05-25 12:17:57 +0000 UTC" firstStartedPulling="2026-05-25 12:17:58.020024689 +0000 UTC m=+299.661219443" lastFinishedPulling="2026-05-25 12:18:05.957982763 +0000 UTC m=+307.599177519" observedRunningTime="2026-05-25 12:18:06.948133466 +0000 UTC m=+308.589328233" watchObservedRunningTime="2026-05-25 12:18:06.949570822 +0000 UTC m=+308.590765599"
May 25 12:18:07.165026 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:07.164987    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0c271934-1560-48c4-a0af-e48574720e47-certificates\") pod \"keda-operator-ffbb595cb-vxjbb\" (UID: \"0c271934-1560-48c4-a0af-e48574720e47\") " pod="openshift-keda/keda-operator-ffbb595cb-vxjbb"
May 25 12:18:07.165242 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:18:07.165180    2578 secret.go:281] references non-existent secret key: ca.crt
May 25 12:18:07.165242 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:18:07.165206    2578 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 25 12:18:07.165242 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:18:07.165219    2578 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-vxjbb: references non-existent secret key: ca.crt
May 25 12:18:07.165390 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:18:07.165289    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c271934-1560-48c4-a0af-e48574720e47-certificates podName:0c271934-1560-48c4-a0af-e48574720e47 nodeName:}" failed. No retries permitted until 2026-05-25 12:18:08.165267893 +0000 UTC m=+309.806462662 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/0c271934-1560-48c4-a0af-e48574720e47-certificates") pod "keda-operator-ffbb595cb-vxjbb" (UID: "0c271934-1560-48c4-a0af-e48574720e47") : references non-existent secret key: ca.crt
May 25 12:18:08.173389 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:08.173345    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0c271934-1560-48c4-a0af-e48574720e47-certificates\") pod \"keda-operator-ffbb595cb-vxjbb\" (UID: \"0c271934-1560-48c4-a0af-e48574720e47\") " pod="openshift-keda/keda-operator-ffbb595cb-vxjbb"
May 25 12:18:08.173801 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:18:08.173525    2578 secret.go:281] references non-existent secret key: ca.crt
May 25 12:18:08.173801 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:18:08.173549    2578 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 25 12:18:08.173801 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:18:08.173560    2578 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-vxjbb: references non-existent secret key: ca.crt
May 25 12:18:08.173801 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:18:08.173630    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c271934-1560-48c4-a0af-e48574720e47-certificates podName:0c271934-1560-48c4-a0af-e48574720e47 nodeName:}" failed. No retries permitted until 2026-05-25 12:18:10.173607284 +0000 UTC m=+311.814802038 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/0c271934-1560-48c4-a0af-e48574720e47-certificates") pod "keda-operator-ffbb595cb-vxjbb" (UID: "0c271934-1560-48c4-a0af-e48574720e47") : references non-existent secret key: ca.crt
May 25 12:18:10.188010 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:10.187966    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0c271934-1560-48c4-a0af-e48574720e47-certificates\") pod \"keda-operator-ffbb595cb-vxjbb\" (UID: \"0c271934-1560-48c4-a0af-e48574720e47\") " pod="openshift-keda/keda-operator-ffbb595cb-vxjbb"
May 25 12:18:10.188438 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:18:10.188107    2578 secret.go:281] references non-existent secret key: ca.crt
May 25 12:18:10.188438 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:18:10.188119    2578 projected.go:277] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
May 25 12:18:10.188438 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:18:10.188128    2578 projected.go:194] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-ffbb595cb-vxjbb: references non-existent secret key: ca.crt
May 25 12:18:10.188438 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:18:10.188178    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c271934-1560-48c4-a0af-e48574720e47-certificates podName:0c271934-1560-48c4-a0af-e48574720e47 nodeName:}" failed. No retries permitted until 2026-05-25 12:18:14.18816389 +0000 UTC m=+315.829358645 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/0c271934-1560-48c4-a0af-e48574720e47-certificates") pod "keda-operator-ffbb595cb-vxjbb" (UID: "0c271934-1560-48c4-a0af-e48574720e47") : references non-existent secret key: ca.crt
May 25 12:18:14.217511 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:14.217469    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0c271934-1560-48c4-a0af-e48574720e47-certificates\") pod \"keda-operator-ffbb595cb-vxjbb\" (UID: \"0c271934-1560-48c4-a0af-e48574720e47\") " pod="openshift-keda/keda-operator-ffbb595cb-vxjbb"
May 25 12:18:14.219850 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:14.219829    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/0c271934-1560-48c4-a0af-e48574720e47-certificates\") pod \"keda-operator-ffbb595cb-vxjbb\" (UID: \"0c271934-1560-48c4-a0af-e48574720e47\") " pod="openshift-keda/keda-operator-ffbb595cb-vxjbb"
May 25 12:18:14.291044 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:14.291016    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-ffbb595cb-vxjbb"
May 25 12:18:14.412916 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:14.412885    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-ffbb595cb-vxjbb"]
May 25 12:18:14.415480 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:18:14.415452    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c271934_1560_48c4_a0af_e48574720e47.slice/crio-440d86d5914a3584229f715274c49b7a371ebe080d0175808e48c6f2515d6b8a WatchSource:0}: Error finding container 440d86d5914a3584229f715274c49b7a371ebe080d0175808e48c6f2515d6b8a: Status 404 returned error can't find the container with id 440d86d5914a3584229f715274c49b7a371ebe080d0175808e48c6f2515d6b8a
May 25 12:18:14.417163 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:14.417144    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:18:14.947649 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:14.947614    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-vxjbb" event={"ID":"0c271934-1560-48c4-a0af-e48574720e47","Type":"ContainerStarted","Data":"440d86d5914a3584229f715274c49b7a371ebe080d0175808e48c6f2515d6b8a"}
May 25 12:18:17.960488 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:17.960405    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-ffbb595cb-vxjbb" event={"ID":"0c271934-1560-48c4-a0af-e48574720e47","Type":"ContainerStarted","Data":"abd937dd8c7f5611d798daa5e713af50b11a6866be19916b4f5c81324b15f27b"}
May 25 12:18:17.960826 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:17.960513    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-ffbb595cb-vxjbb"
May 25 12:18:27.932854 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:27.932825    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-bbf89fd5d-h5ggj"
May 25 12:18:27.952549 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:27.952488    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-keda/keda-operator-ffbb595cb-vxjbb" podStartSLOduration=18.781157673 podStartE2EDuration="21.952473061s" podCreationTimestamp="2026-05-25 12:18:06 +0000 UTC" firstStartedPulling="2026-05-25 12:18:14.417303818 +0000 UTC m=+316.058498573" lastFinishedPulling="2026-05-25 12:18:17.588619189 +0000 UTC m=+319.229813961" observedRunningTime="2026-05-25 12:18:17.978476656 +0000 UTC m=+319.619671432" watchObservedRunningTime="2026-05-25 12:18:27.952473061 +0000 UTC m=+329.593667837"
May 25 12:18:38.965699 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:18:38.965665    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-ffbb595cb-vxjbb"
May 25 12:19:15.916348 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:19:15.916315    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-6b94ff949c-xb6cq"]
May 25 12:19:15.919162 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:19:15.919144    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6b94ff949c-xb6cq"
May 25 12:19:15.922428 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:19:15.922408    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
May 25 12:19:15.922528 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:19:15.922426    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 25 12:19:15.922528 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:19:15.922461    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 25 12:19:15.923692 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:19:15.923674    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-g7mnb\""
May 25 12:19:15.936255 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:19:15.936234    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6b94ff949c-xb6cq"]
May 25 12:19:16.056473 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:19:16.056432    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mvtn\" (UniqueName: \"kubernetes.io/projected/210b07bc-b316-4131-98c1-1b36b5e45d4e-kube-api-access-4mvtn\") pod \"llmisvc-controller-manager-6b94ff949c-xb6cq\" (UID: \"210b07bc-b316-4131-98c1-1b36b5e45d4e\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-xb6cq"
May 25 12:19:16.056666 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:19:16.056484    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/210b07bc-b316-4131-98c1-1b36b5e45d4e-cert\") pod \"llmisvc-controller-manager-6b94ff949c-xb6cq\" (UID: \"210b07bc-b316-4131-98c1-1b36b5e45d4e\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-xb6cq"
May 25 12:19:16.157628 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:19:16.157593    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/210b07bc-b316-4131-98c1-1b36b5e45d4e-cert\") pod \"llmisvc-controller-manager-6b94ff949c-xb6cq\" (UID: \"210b07bc-b316-4131-98c1-1b36b5e45d4e\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-xb6cq"
May 25 12:19:16.157840 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:19:16.157663    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4mvtn\" (UniqueName: \"kubernetes.io/projected/210b07bc-b316-4131-98c1-1b36b5e45d4e-kube-api-access-4mvtn\") pod \"llmisvc-controller-manager-6b94ff949c-xb6cq\" (UID: \"210b07bc-b316-4131-98c1-1b36b5e45d4e\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-xb6cq"
May 25 12:19:16.160076 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:19:16.160041    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/210b07bc-b316-4131-98c1-1b36b5e45d4e-cert\") pod \"llmisvc-controller-manager-6b94ff949c-xb6cq\" (UID: \"210b07bc-b316-4131-98c1-1b36b5e45d4e\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-xb6cq"
May 25 12:19:16.172421 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:19:16.172371    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mvtn\" (UniqueName: \"kubernetes.io/projected/210b07bc-b316-4131-98c1-1b36b5e45d4e-kube-api-access-4mvtn\") pod \"llmisvc-controller-manager-6b94ff949c-xb6cq\" (UID: \"210b07bc-b316-4131-98c1-1b36b5e45d4e\") " pod="kserve/llmisvc-controller-manager-6b94ff949c-xb6cq"
May 25 12:19:16.228707 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:19:16.228663    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-6b94ff949c-xb6cq"
May 25 12:19:16.350441 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:19:16.350413    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-6b94ff949c-xb6cq"]
May 25 12:19:16.354074 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:19:16.354029    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod210b07bc_b316_4131_98c1_1b36b5e45d4e.slice/crio-58170e93d0cfb509214d4a4f60b234cc268e71315a0d6bc0a7e9a546b91bdcb6 WatchSource:0}: Error finding container 58170e93d0cfb509214d4a4f60b234cc268e71315a0d6bc0a7e9a546b91bdcb6: Status 404 returned error can't find the container with id 58170e93d0cfb509214d4a4f60b234cc268e71315a0d6bc0a7e9a546b91bdcb6
May 25 12:19:17.111533 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:19:17.111494    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6b94ff949c-xb6cq" event={"ID":"210b07bc-b316-4131-98c1-1b36b5e45d4e","Type":"ContainerStarted","Data":"58170e93d0cfb509214d4a4f60b234cc268e71315a0d6bc0a7e9a546b91bdcb6"}
May 25 12:19:19.118421 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:19:19.118381    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-6b94ff949c-xb6cq" event={"ID":"210b07bc-b316-4131-98c1-1b36b5e45d4e","Type":"ContainerStarted","Data":"79dc8d257decb7be87dad2f1428087995feabcac672f9841f80fa8cd87ff2d1c"}
May 25 12:19:19.118809 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:19:19.118544    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-6b94ff949c-xb6cq"
May 25 12:19:19.137127 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:19:19.137082    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-6b94ff949c-xb6cq" podStartSLOduration=1.505568447 podStartE2EDuration="4.137053528s" podCreationTimestamp="2026-05-25 12:19:15 +0000 UTC" firstStartedPulling="2026-05-25 12:19:16.355237672 +0000 UTC m=+377.996432429" lastFinishedPulling="2026-05-25 12:19:18.986722742 +0000 UTC m=+380.627917510" observedRunningTime="2026-05-25 12:19:19.135631868 +0000 UTC m=+380.776826644" watchObservedRunningTime="2026-05-25 12:19:19.137053528 +0000 UTC m=+380.778248303"
May 25 12:19:50.123458 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:19:50.123429    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-6b94ff949c-xb6cq"
May 25 12:20:24.335385 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:20:24.335299    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-696fc77849-8l6v7"]
May 25 12:20:24.338255 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:20:24.338236    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-8l6v7"
May 25 12:20:24.342720 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:20:24.342691    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
May 25 12:20:24.342720 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:20:24.342713    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-tlt24\""
May 25 12:20:24.349712 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:20:24.349676    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-8l6v7"]
May 25 12:20:24.451699 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:20:24.451649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/611dfffc-5590-4116-a890-8da7c60c9f0f-cert\") pod \"odh-model-controller-696fc77849-8l6v7\" (UID: \"611dfffc-5590-4116-a890-8da7c60c9f0f\") " pod="kserve/odh-model-controller-696fc77849-8l6v7"
May 25 12:20:24.451699 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:20:24.451704    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89tlf\" (UniqueName: \"kubernetes.io/projected/611dfffc-5590-4116-a890-8da7c60c9f0f-kube-api-access-89tlf\") pod \"odh-model-controller-696fc77849-8l6v7\" (UID: \"611dfffc-5590-4116-a890-8da7c60c9f0f\") " pod="kserve/odh-model-controller-696fc77849-8l6v7"
May 25 12:20:24.552917 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:20:24.552878    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/611dfffc-5590-4116-a890-8da7c60c9f0f-cert\") pod \"odh-model-controller-696fc77849-8l6v7\" (UID: \"611dfffc-5590-4116-a890-8da7c60c9f0f\") " pod="kserve/odh-model-controller-696fc77849-8l6v7"
May 25 12:20:24.553141 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:20:24.552926    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-89tlf\" (UniqueName: \"kubernetes.io/projected/611dfffc-5590-4116-a890-8da7c60c9f0f-kube-api-access-89tlf\") pod \"odh-model-controller-696fc77849-8l6v7\" (UID: \"611dfffc-5590-4116-a890-8da7c60c9f0f\") " pod="kserve/odh-model-controller-696fc77849-8l6v7"
May 25 12:20:24.555401 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:20:24.555377    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/611dfffc-5590-4116-a890-8da7c60c9f0f-cert\") pod \"odh-model-controller-696fc77849-8l6v7\" (UID: \"611dfffc-5590-4116-a890-8da7c60c9f0f\") " pod="kserve/odh-model-controller-696fc77849-8l6v7"
May 25 12:20:24.564768 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:20:24.564742    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89tlf\" (UniqueName: \"kubernetes.io/projected/611dfffc-5590-4116-a890-8da7c60c9f0f-kube-api-access-89tlf\") pod \"odh-model-controller-696fc77849-8l6v7\" (UID: \"611dfffc-5590-4116-a890-8da7c60c9f0f\") " pod="kserve/odh-model-controller-696fc77849-8l6v7"
May 25 12:20:24.648784 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:20:24.648687    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-8l6v7"
May 25 12:20:24.781733 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:20:24.781560    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-8l6v7"]
May 25 12:20:24.784275 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:20:24.784240    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod611dfffc_5590_4116_a890_8da7c60c9f0f.slice/crio-b9bed171742f3673313223363739130251b31fcea76eb3750d99d49dc4d6bcc5 WatchSource:0}: Error finding container b9bed171742f3673313223363739130251b31fcea76eb3750d99d49dc4d6bcc5: Status 404 returned error can't find the container with id b9bed171742f3673313223363739130251b31fcea76eb3750d99d49dc4d6bcc5
May 25 12:20:25.296901 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:20:25.296858    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-8l6v7" event={"ID":"611dfffc-5590-4116-a890-8da7c60c9f0f","Type":"ContainerStarted","Data":"b9bed171742f3673313223363739130251b31fcea76eb3750d99d49dc4d6bcc5"}
May 25 12:20:28.305916 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:20:28.305879    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-8l6v7" event={"ID":"611dfffc-5590-4116-a890-8da7c60c9f0f","Type":"ContainerStarted","Data":"2859043d5c4f80d6bef00ba386b1781ccc9c033dd38d07cc3a9f7aea5ccba231"}
May 25 12:20:28.306342 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:20:28.306023    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-696fc77849-8l6v7"
May 25 12:20:28.334466 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:20:28.334419    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/odh-model-controller-696fc77849-8l6v7" podStartSLOduration=1.574352134 podStartE2EDuration="4.334403749s" podCreationTimestamp="2026-05-25 12:20:24 +0000 UTC" firstStartedPulling="2026-05-25 12:20:24.785456573 +0000 UTC m=+446.426651329" lastFinishedPulling="2026-05-25 12:20:27.545508189 +0000 UTC m=+449.186702944" observedRunningTime="2026-05-25 12:20:28.334039235 +0000 UTC m=+449.975234016" watchObservedRunningTime="2026-05-25 12:20:28.334403749 +0000 UTC m=+449.975598525"
May 25 12:20:39.311353 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:20:39.311312    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-696fc77849-8l6v7"
May 25 12:21:01.348367 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:01.348331    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5"]
May 25 12:21:01.353615 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:01.353593    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5"
May 25 12:21:01.359020 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:01.358993    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 12:21:01.359407 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:01.359385    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-v7cv4\""
May 25 12:21:01.360365 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:01.360335    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
May 25 12:21:01.361426 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:01.361400    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 12:21:01.361426 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:01.361418    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
May 25 12:21:01.362211 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:01.362187    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5"]
May 25 12:21:01.440444 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:01.440397    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35d8edba-6dc8-441f-b65f-9d4e667b4dcf-proxy-tls\") pod \"isvc-xgboost-graph-predictor-669d8d6456-x8tg5\" (UID: \"35d8edba-6dc8-441f-b65f-9d4e667b4dcf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5"
May 25 12:21:01.440641 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:01.440455    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35d8edba-6dc8-441f-b65f-9d4e667b4dcf-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-669d8d6456-x8tg5\" (UID: \"35d8edba-6dc8-441f-b65f-9d4e667b4dcf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5"
May 25 12:21:01.440641 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:01.440530    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxrp6\" (UniqueName: \"kubernetes.io/projected/35d8edba-6dc8-441f-b65f-9d4e667b4dcf-kube-api-access-bxrp6\") pod \"isvc-xgboost-graph-predictor-669d8d6456-x8tg5\" (UID: \"35d8edba-6dc8-441f-b65f-9d4e667b4dcf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5"
May 25 12:21:01.440641 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:01.440602    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35d8edba-6dc8-441f-b65f-9d4e667b4dcf-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-669d8d6456-x8tg5\" (UID: \"35d8edba-6dc8-441f-b65f-9d4e667b4dcf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5"
May 25 12:21:01.541616 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:01.541578    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35d8edba-6dc8-441f-b65f-9d4e667b4dcf-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-669d8d6456-x8tg5\" (UID: \"35d8edba-6dc8-441f-b65f-9d4e667b4dcf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5"
May 25 12:21:01.541616 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:01.541619    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35d8edba-6dc8-441f-b65f-9d4e667b4dcf-proxy-tls\") pod \"isvc-xgboost-graph-predictor-669d8d6456-x8tg5\" (UID: \"35d8edba-6dc8-441f-b65f-9d4e667b4dcf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5"
May 25 12:21:01.541895 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:01.541648    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35d8edba-6dc8-441f-b65f-9d4e667b4dcf-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-669d8d6456-x8tg5\" (UID: \"35d8edba-6dc8-441f-b65f-9d4e667b4dcf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5"
May 25 12:21:01.541895 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:01.541700    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bxrp6\" (UniqueName: \"kubernetes.io/projected/35d8edba-6dc8-441f-b65f-9d4e667b4dcf-kube-api-access-bxrp6\") pod \"isvc-xgboost-graph-predictor-669d8d6456-x8tg5\" (UID: \"35d8edba-6dc8-441f-b65f-9d4e667b4dcf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5"
May 25 12:21:01.541895 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:21:01.541746    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-predictor-serving-cert: secret "isvc-xgboost-graph-predictor-serving-cert" not found
May 25 12:21:01.541895 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:21:01.541826    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35d8edba-6dc8-441f-b65f-9d4e667b4dcf-proxy-tls podName:35d8edba-6dc8-441f-b65f-9d4e667b4dcf nodeName:}" failed. No retries permitted until 2026-05-25 12:21:02.041804398 +0000 UTC m=+483.682999153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/35d8edba-6dc8-441f-b65f-9d4e667b4dcf-proxy-tls") pod "isvc-xgboost-graph-predictor-669d8d6456-x8tg5" (UID: "35d8edba-6dc8-441f-b65f-9d4e667b4dcf") : secret "isvc-xgboost-graph-predictor-serving-cert" not found
May 25 12:21:01.542164 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:01.542041    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35d8edba-6dc8-441f-b65f-9d4e667b4dcf-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-669d8d6456-x8tg5\" (UID: \"35d8edba-6dc8-441f-b65f-9d4e667b4dcf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5"
May 25 12:21:01.542370 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:01.542351    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35d8edba-6dc8-441f-b65f-9d4e667b4dcf-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-669d8d6456-x8tg5\" (UID: \"35d8edba-6dc8-441f-b65f-9d4e667b4dcf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5"
May 25 12:21:01.558850 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:01.558820    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxrp6\" (UniqueName: \"kubernetes.io/projected/35d8edba-6dc8-441f-b65f-9d4e667b4dcf-kube-api-access-bxrp6\") pod \"isvc-xgboost-graph-predictor-669d8d6456-x8tg5\" (UID: \"35d8edba-6dc8-441f-b65f-9d4e667b4dcf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5"
May 25 12:21:02.046187 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:02.046150    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35d8edba-6dc8-441f-b65f-9d4e667b4dcf-proxy-tls\") pod \"isvc-xgboost-graph-predictor-669d8d6456-x8tg5\" (UID: \"35d8edba-6dc8-441f-b65f-9d4e667b4dcf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5"
May 25 12:21:02.048597 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:02.048570    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35d8edba-6dc8-441f-b65f-9d4e667b4dcf-proxy-tls\") pod \"isvc-xgboost-graph-predictor-669d8d6456-x8tg5\" (UID: \"35d8edba-6dc8-441f-b65f-9d4e667b4dcf\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5"
May 25 12:21:02.265145 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:02.265111    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5"
May 25 12:21:02.411729 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:02.411688    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5"]
May 25 12:21:02.414637 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:21:02.414600    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35d8edba_6dc8_441f_b65f_9d4e667b4dcf.slice/crio-69a78fe21e663bbde559f4e87a020a5dabceda93b879d6c1de5498a5687970af WatchSource:0}: Error finding container 69a78fe21e663bbde559f4e87a020a5dabceda93b879d6c1de5498a5687970af: Status 404 returned error can't find the container with id 69a78fe21e663bbde559f4e87a020a5dabceda93b879d6c1de5498a5687970af
May 25 12:21:03.400544 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:03.400499    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5" event={"ID":"35d8edba-6dc8-441f-b65f-9d4e667b4dcf","Type":"ContainerStarted","Data":"69a78fe21e663bbde559f4e87a020a5dabceda93b879d6c1de5498a5687970af"}
May 25 12:21:06.410853 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:06.410756    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5" event={"ID":"35d8edba-6dc8-441f-b65f-9d4e667b4dcf","Type":"ContainerStarted","Data":"d23fcbe2ce3f6b39e31682713848ce25a98be7516bc232b5f4004bb765e6086b"}
May 25 12:21:10.421782 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:10.421743    2578 generic.go:358] "Generic (PLEG): container finished" podID="35d8edba-6dc8-441f-b65f-9d4e667b4dcf" containerID="d23fcbe2ce3f6b39e31682713848ce25a98be7516bc232b5f4004bb765e6086b" exitCode=0
May 25 12:21:10.422185 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:10.421786    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5" event={"ID":"35d8edba-6dc8-441f-b65f-9d4e667b4dcf","Type":"ContainerDied","Data":"d23fcbe2ce3f6b39e31682713848ce25a98be7516bc232b5f4004bb765e6086b"}
May 25 12:21:28.476562 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:28.476517    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5" event={"ID":"35d8edba-6dc8-441f-b65f-9d4e667b4dcf","Type":"ContainerStarted","Data":"04830498fdb51ee47be891ff33c73f3f5a8143ce63583d2278a9d3d8b230ce02"}
May 25 12:21:30.483541 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:30.483496    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5" event={"ID":"35d8edba-6dc8-441f-b65f-9d4e667b4dcf","Type":"ContainerStarted","Data":"8830bdd0ca9644657cfcd5f9699476f07ee1b8cff8aee4d01a306898cc6528d3"}
May 25 12:21:30.483985 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:30.483689    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5"
May 25 12:21:30.506915 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:30.506858    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5" podStartSLOduration=2.039944624 podStartE2EDuration="29.506841972s" podCreationTimestamp="2026-05-25 12:21:01 +0000 UTC" firstStartedPulling="2026-05-25 12:21:02.416544486 +0000 UTC m=+484.057739241" lastFinishedPulling="2026-05-25 12:21:29.883441832 +0000 UTC m=+511.524636589" observedRunningTime="2026-05-25 12:21:30.505468921 +0000 UTC m=+512.146663697" watchObservedRunningTime="2026-05-25 12:21:30.506841972 +0000 UTC m=+512.148036746"
May 25 12:21:31.485886 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:31.485854    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5"
May 25 12:21:31.486944 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:31.486916    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5" podUID="35d8edba-6dc8-441f-b65f-9d4e667b4dcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
May 25 12:21:32.488810 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:32.488770    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5" podUID="35d8edba-6dc8-441f-b65f-9d4e667b4dcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
May 25 12:21:37.492679 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:37.492646    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5"
May 25 12:21:37.493265 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:37.493235    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5" podUID="35d8edba-6dc8-441f-b65f-9d4e667b4dcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
May 25 12:21:47.496237 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:47.496146    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5" podUID="35d8edba-6dc8-441f-b65f-9d4e667b4dcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
May 25 12:21:57.493766 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:21:57.493726    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5" podUID="35d8edba-6dc8-441f-b65f-9d4e667b4dcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
May 25 12:22:07.493714 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:07.493664    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5" podUID="35d8edba-6dc8-441f-b65f-9d4e667b4dcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
May 25 12:22:17.493452 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:17.493405    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5" podUID="35d8edba-6dc8-441f-b65f-9d4e667b4dcf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.17:8080: connect: connection refused"
May 25 12:22:20.784678 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:20.784639    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58"]
May 25 12:22:20.787545 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:20.787528    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58"
May 25 12:22:20.790216 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:20.790193    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-1742e-serving-cert\""
May 25 12:22:20.790374 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:20.790199    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-1742e-kube-rbac-proxy-sar-config\""
May 25 12:22:20.796050 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:20.796025    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58"]
May 25 12:22:20.879731 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:20.879685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cffb6412-1856-40a9-bc55-12b353ed3168-proxy-tls\") pod \"switch-graph-1742e-68c468964c-d9m58\" (UID: \"cffb6412-1856-40a9-bc55-12b353ed3168\") " pod="kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58"
May 25 12:22:20.879907 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:20.879740    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cffb6412-1856-40a9-bc55-12b353ed3168-openshift-service-ca-bundle\") pod \"switch-graph-1742e-68c468964c-d9m58\" (UID: \"cffb6412-1856-40a9-bc55-12b353ed3168\") " pod="kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58"
May 25 12:22:20.981139 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:20.981103    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cffb6412-1856-40a9-bc55-12b353ed3168-proxy-tls\") pod \"switch-graph-1742e-68c468964c-d9m58\" (UID: \"cffb6412-1856-40a9-bc55-12b353ed3168\") " pod="kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58"
May 25 12:22:20.981139 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:20.981150    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cffb6412-1856-40a9-bc55-12b353ed3168-openshift-service-ca-bundle\") pod \"switch-graph-1742e-68c468964c-d9m58\" (UID: \"cffb6412-1856-40a9-bc55-12b353ed3168\") " pod="kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58"
May 25 12:22:20.981381 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:22:20.981263    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-1742e-serving-cert: secret "switch-graph-1742e-serving-cert" not found
May 25 12:22:20.981381 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:22:20.981327    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cffb6412-1856-40a9-bc55-12b353ed3168-proxy-tls podName:cffb6412-1856-40a9-bc55-12b353ed3168 nodeName:}" failed. No retries permitted until 2026-05-25 12:22:21.48131127 +0000 UTC m=+563.122506024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cffb6412-1856-40a9-bc55-12b353ed3168-proxy-tls") pod "switch-graph-1742e-68c468964c-d9m58" (UID: "cffb6412-1856-40a9-bc55-12b353ed3168") : secret "switch-graph-1742e-serving-cert" not found
May 25 12:22:20.981752 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:20.981734    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cffb6412-1856-40a9-bc55-12b353ed3168-openshift-service-ca-bundle\") pod \"switch-graph-1742e-68c468964c-d9m58\" (UID: \"cffb6412-1856-40a9-bc55-12b353ed3168\") " pod="kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58"
May 25 12:22:21.485194 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:21.485149    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cffb6412-1856-40a9-bc55-12b353ed3168-proxy-tls\") pod \"switch-graph-1742e-68c468964c-d9m58\" (UID: \"cffb6412-1856-40a9-bc55-12b353ed3168\") " pod="kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58"
May 25 12:22:21.487603 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:21.487571    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cffb6412-1856-40a9-bc55-12b353ed3168-proxy-tls\") pod \"switch-graph-1742e-68c468964c-d9m58\" (UID: \"cffb6412-1856-40a9-bc55-12b353ed3168\") " pod="kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58"
May 25 12:22:21.698538 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:21.698498    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58"
May 25 12:22:21.824103 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:21.824047    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58"]
May 25 12:22:21.826832 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:22:21.826804    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcffb6412_1856_40a9_bc55_12b353ed3168.slice/crio-b8d303db32ed7554a17e77a5c26d90e7cb9b499edcb24244d662fb2ce611b263 WatchSource:0}: Error finding container b8d303db32ed7554a17e77a5c26d90e7cb9b499edcb24244d662fb2ce611b263: Status 404 returned error can't find the container with id b8d303db32ed7554a17e77a5c26d90e7cb9b499edcb24244d662fb2ce611b263
May 25 12:22:22.623387 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:22.623347    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58" event={"ID":"cffb6412-1856-40a9-bc55-12b353ed3168","Type":"ContainerStarted","Data":"b8d303db32ed7554a17e77a5c26d90e7cb9b499edcb24244d662fb2ce611b263"}
May 25 12:22:24.629720 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:24.629665    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58" event={"ID":"cffb6412-1856-40a9-bc55-12b353ed3168","Type":"ContainerStarted","Data":"7995348819ed815d9f5b3ca54c07ec56550668296b00e81b79adcb1ea86a2ed7"}
May 25 12:22:24.630197 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:24.629794    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58"
May 25 12:22:24.647304 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:24.647253    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58" podStartSLOduration=2.036633386 podStartE2EDuration="4.647237775s" podCreationTimestamp="2026-05-25 12:22:20 +0000 UTC" firstStartedPulling="2026-05-25 12:22:21.828439615 +0000 UTC m=+563.469634370" lastFinishedPulling="2026-05-25 12:22:24.439044004 +0000 UTC m=+566.080238759" observedRunningTime="2026-05-25 12:22:24.645999618 +0000 UTC m=+566.287194394" watchObservedRunningTime="2026-05-25 12:22:24.647237775 +0000 UTC m=+566.288432552"
May 25 12:22:27.494047 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:27.494021    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5"
May 25 12:22:30.639724 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:30.639695    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58"
May 25 12:22:30.862543 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:30.862504    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58"]
May 25 12:22:30.862763 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:30.862739    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58" podUID="cffb6412-1856-40a9-bc55-12b353ed3168" containerName="switch-graph-1742e" containerID="cri-o://7995348819ed815d9f5b3ca54c07ec56550668296b00e81b79adcb1ea86a2ed7" gracePeriod=30
May 25 12:22:35.637784 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:35.637744    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58" podUID="cffb6412-1856-40a9-bc55-12b353ed3168" containerName="switch-graph-1742e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:22:40.638740 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:40.638697    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58" podUID="cffb6412-1856-40a9-bc55-12b353ed3168" containerName="switch-graph-1742e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:22:45.639335 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:45.639285    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58" podUID="cffb6412-1856-40a9-bc55-12b353ed3168" containerName="switch-graph-1742e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:22:45.639816 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:45.639390    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58"
May 25 12:22:50.638085 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:50.638012    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58" podUID="cffb6412-1856-40a9-bc55-12b353ed3168" containerName="switch-graph-1742e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:22:50.772524 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:50.772485    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5"]
May 25 12:22:50.775437 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:50.775419    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5"
May 25 12:22:50.778335 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:50.778310    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
May 25 12:22:50.778445 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:50.778363    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
May 25 12:22:50.786574 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:50.786547    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5"]
May 25 12:22:50.905650 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:50.905558    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee9e2f49-0644-46d5-9659-1d685023f00f-openshift-service-ca-bundle\") pod \"model-chainer-77c65d8d88-vkzr5\" (UID: \"ee9e2f49-0644-46d5-9659-1d685023f00f\") " pod="kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5"
May 25 12:22:50.905650 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:50.905631    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee9e2f49-0644-46d5-9659-1d685023f00f-proxy-tls\") pod \"model-chainer-77c65d8d88-vkzr5\" (UID: \"ee9e2f49-0644-46d5-9659-1d685023f00f\") " pod="kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5"
May 25 12:22:51.006978 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:51.006939    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee9e2f49-0644-46d5-9659-1d685023f00f-proxy-tls\") pod \"model-chainer-77c65d8d88-vkzr5\" (UID: \"ee9e2f49-0644-46d5-9659-1d685023f00f\") " pod="kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5"
May 25 12:22:51.007192 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:51.007019    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee9e2f49-0644-46d5-9659-1d685023f00f-openshift-service-ca-bundle\") pod \"model-chainer-77c65d8d88-vkzr5\" (UID: \"ee9e2f49-0644-46d5-9659-1d685023f00f\") " pod="kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5"
May 25 12:22:51.007814 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:51.007776    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee9e2f49-0644-46d5-9659-1d685023f00f-openshift-service-ca-bundle\") pod \"model-chainer-77c65d8d88-vkzr5\" (UID: \"ee9e2f49-0644-46d5-9659-1d685023f00f\") " pod="kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5"
May 25 12:22:51.009546 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:51.009515    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee9e2f49-0644-46d5-9659-1d685023f00f-proxy-tls\") pod \"model-chainer-77c65d8d88-vkzr5\" (UID: \"ee9e2f49-0644-46d5-9659-1d685023f00f\") " pod="kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5"
May 25 12:22:51.085509 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:51.085471    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5"
May 25 12:22:51.218182 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:51.218148    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5"]
May 25 12:22:51.221461 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:22:51.221425    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee9e2f49_0644_46d5_9659_1d685023f00f.slice/crio-c1e45c064be2a9f8e6a8321e179f623b72c7f1dc0761ca1153ef1a90cf8aefe3 WatchSource:0}: Error finding container c1e45c064be2a9f8e6a8321e179f623b72c7f1dc0761ca1153ef1a90cf8aefe3: Status 404 returned error can't find the container with id c1e45c064be2a9f8e6a8321e179f623b72c7f1dc0761ca1153ef1a90cf8aefe3
May 25 12:22:51.709713 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:51.709662    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5" event={"ID":"ee9e2f49-0644-46d5-9659-1d685023f00f","Type":"ContainerStarted","Data":"6d3b42cee67c4dd307e393491cbe116b1c12d39edc1e81e585fca26cdcdef002"}
May 25 12:22:51.709713 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:51.709715    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5" event={"ID":"ee9e2f49-0644-46d5-9659-1d685023f00f","Type":"ContainerStarted","Data":"c1e45c064be2a9f8e6a8321e179f623b72c7f1dc0761ca1153ef1a90cf8aefe3"}
May 25 12:22:51.710153 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:51.709809    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5"
May 25 12:22:51.728550 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:51.728500    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5" podStartSLOduration=1.728485063 podStartE2EDuration="1.728485063s" podCreationTimestamp="2026-05-25 12:22:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:22:51.727338598 +0000 UTC m=+593.368533375" watchObservedRunningTime="2026-05-25 12:22:51.728485063 +0000 UTC m=+593.369679840"
May 25 12:22:55.637991 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:55.637949    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58" podUID="cffb6412-1856-40a9-bc55-12b353ed3168" containerName="switch-graph-1742e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:22:57.719051 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:57.719020    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5"
May 25 12:22:58.904616 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:58.904587    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/ovn-acl-logging/0.log"
May 25 12:22:58.905460 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:22:58.905442    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/ovn-acl-logging/0.log"
May 25 12:23:00.638707 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:00.638662    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58" podUID="cffb6412-1856-40a9-bc55-12b353ed3168" containerName="switch-graph-1742e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:23:00.863320 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:00.863284    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5"]
May 25 12:23:00.863554 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:00.863513    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5" podUID="ee9e2f49-0644-46d5-9659-1d685023f00f" containerName="model-chainer" containerID="cri-o://6d3b42cee67c4dd307e393491cbe116b1c12d39edc1e81e585fca26cdcdef002" gracePeriod=30
May 25 12:23:00.952906 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:00.952865    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5"]
May 25 12:23:00.953514 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:00.953457    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5" podUID="35d8edba-6dc8-441f-b65f-9d4e667b4dcf" containerName="kube-rbac-proxy" containerID="cri-o://8830bdd0ca9644657cfcd5f9699476f07ee1b8cff8aee4d01a306898cc6528d3" gracePeriod=30
May 25 12:23:00.953669 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:00.953608    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5" podUID="35d8edba-6dc8-441f-b65f-9d4e667b4dcf" containerName="kserve-container" containerID="cri-o://04830498fdb51ee47be891ff33c73f3f5a8143ce63583d2278a9d3d8b230ce02" gracePeriod=30
May 25 12:23:01.074397 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:01.074373    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58"
May 25 12:23:01.189248 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:01.189161    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cffb6412-1856-40a9-bc55-12b353ed3168-openshift-service-ca-bundle\") pod \"cffb6412-1856-40a9-bc55-12b353ed3168\" (UID: \"cffb6412-1856-40a9-bc55-12b353ed3168\") "
May 25 12:23:01.189248 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:01.189208    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cffb6412-1856-40a9-bc55-12b353ed3168-proxy-tls\") pod \"cffb6412-1856-40a9-bc55-12b353ed3168\" (UID: \"cffb6412-1856-40a9-bc55-12b353ed3168\") "
May 25 12:23:01.189591 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:01.189562    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cffb6412-1856-40a9-bc55-12b353ed3168-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "cffb6412-1856-40a9-bc55-12b353ed3168" (UID: "cffb6412-1856-40a9-bc55-12b353ed3168"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:23:01.191408 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:01.191384    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cffb6412-1856-40a9-bc55-12b353ed3168-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cffb6412-1856-40a9-bc55-12b353ed3168" (UID: "cffb6412-1856-40a9-bc55-12b353ed3168"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:23:01.290210 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:01.290173    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cffb6412-1856-40a9-bc55-12b353ed3168-openshift-service-ca-bundle\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:23:01.290210 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:01.290203    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cffb6412-1856-40a9-bc55-12b353ed3168-proxy-tls\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:23:01.739965 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:01.739920    2578 generic.go:358] "Generic (PLEG): container finished" podID="35d8edba-6dc8-441f-b65f-9d4e667b4dcf" containerID="8830bdd0ca9644657cfcd5f9699476f07ee1b8cff8aee4d01a306898cc6528d3" exitCode=2
May 25 12:23:01.740370 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:01.739993    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5" event={"ID":"35d8edba-6dc8-441f-b65f-9d4e667b4dcf","Type":"ContainerDied","Data":"8830bdd0ca9644657cfcd5f9699476f07ee1b8cff8aee4d01a306898cc6528d3"}
May 25 12:23:01.740966 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:01.740946    2578 generic.go:358] "Generic (PLEG): container finished" podID="cffb6412-1856-40a9-bc55-12b353ed3168" containerID="7995348819ed815d9f5b3ca54c07ec56550668296b00e81b79adcb1ea86a2ed7" exitCode=0
May 25 12:23:01.741034 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:01.741022    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58" event={"ID":"cffb6412-1856-40a9-bc55-12b353ed3168","Type":"ContainerDied","Data":"7995348819ed815d9f5b3ca54c07ec56550668296b00e81b79adcb1ea86a2ed7"}
May 25 12:23:01.741105 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:01.741033    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58"
May 25 12:23:01.741105 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:01.741047    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58" event={"ID":"cffb6412-1856-40a9-bc55-12b353ed3168","Type":"ContainerDied","Data":"b8d303db32ed7554a17e77a5c26d90e7cb9b499edcb24244d662fb2ce611b263"}
May 25 12:23:01.741105 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:01.741087    2578 scope.go:117] "RemoveContainer" containerID="7995348819ed815d9f5b3ca54c07ec56550668296b00e81b79adcb1ea86a2ed7"
May 25 12:23:01.748823 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:01.748800    2578 scope.go:117] "RemoveContainer" containerID="7995348819ed815d9f5b3ca54c07ec56550668296b00e81b79adcb1ea86a2ed7"
May 25 12:23:01.749106 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:23:01.749086    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7995348819ed815d9f5b3ca54c07ec56550668296b00e81b79adcb1ea86a2ed7\": container with ID starting with 7995348819ed815d9f5b3ca54c07ec56550668296b00e81b79adcb1ea86a2ed7 not found: ID does not exist" containerID="7995348819ed815d9f5b3ca54c07ec56550668296b00e81b79adcb1ea86a2ed7"
May 25 12:23:01.749172 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:01.749121    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7995348819ed815d9f5b3ca54c07ec56550668296b00e81b79adcb1ea86a2ed7"} err="failed to get container status \"7995348819ed815d9f5b3ca54c07ec56550668296b00e81b79adcb1ea86a2ed7\": rpc error: code = NotFound desc = could not find container \"7995348819ed815d9f5b3ca54c07ec56550668296b00e81b79adcb1ea86a2ed7\": container with ID starting with 7995348819ed815d9f5b3ca54c07ec56550668296b00e81b79adcb1ea86a2ed7 not found: ID does not exist"
May 25 12:23:01.766280 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:01.766245    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58"]
May 25 12:23:01.771857 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:01.771822    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-1742e-68c468964c-d9m58"]
May 25 12:23:02.489735 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:02.489681    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5" podUID="35d8edba-6dc8-441f-b65f-9d4e667b4dcf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.17:8643/healthz\": dial tcp 10.132.0.17:8643: connect: connection refused"
May 25 12:23:02.717560 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:02.717518    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5" podUID="ee9e2f49-0644-46d5-9659-1d685023f00f" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:23:02.939275 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:02.939244    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cffb6412-1856-40a9-bc55-12b353ed3168" path="/var/lib/kubelet/pods/cffb6412-1856-40a9-bc55-12b353ed3168/volumes"
May 25 12:23:04.700068 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:04.700025    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5"
May 25 12:23:04.752518 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:04.752477    2578 generic.go:358] "Generic (PLEG): container finished" podID="35d8edba-6dc8-441f-b65f-9d4e667b4dcf" containerID="04830498fdb51ee47be891ff33c73f3f5a8143ce63583d2278a9d3d8b230ce02" exitCode=0
May 25 12:23:04.752686 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:04.752553    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5"
May 25 12:23:04.752686 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:04.752546    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5" event={"ID":"35d8edba-6dc8-441f-b65f-9d4e667b4dcf","Type":"ContainerDied","Data":"04830498fdb51ee47be891ff33c73f3f5a8143ce63583d2278a9d3d8b230ce02"}
May 25 12:23:04.752686 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:04.752667    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5" event={"ID":"35d8edba-6dc8-441f-b65f-9d4e667b4dcf","Type":"ContainerDied","Data":"69a78fe21e663bbde559f4e87a020a5dabceda93b879d6c1de5498a5687970af"}
May 25 12:23:04.752798 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:04.752690    2578 scope.go:117] "RemoveContainer" containerID="8830bdd0ca9644657cfcd5f9699476f07ee1b8cff8aee4d01a306898cc6528d3"
May 25 12:23:04.760067 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:04.760033    2578 scope.go:117] "RemoveContainer" containerID="04830498fdb51ee47be891ff33c73f3f5a8143ce63583d2278a9d3d8b230ce02"
May 25 12:23:04.766994 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:04.766975    2578 scope.go:117] "RemoveContainer" containerID="d23fcbe2ce3f6b39e31682713848ce25a98be7516bc232b5f4004bb765e6086b"
May 25 12:23:04.773795 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:04.773775    2578 scope.go:117] "RemoveContainer" containerID="8830bdd0ca9644657cfcd5f9699476f07ee1b8cff8aee4d01a306898cc6528d3"
May 25 12:23:04.774042 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:23:04.774024    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8830bdd0ca9644657cfcd5f9699476f07ee1b8cff8aee4d01a306898cc6528d3\": container with ID starting with 8830bdd0ca9644657cfcd5f9699476f07ee1b8cff8aee4d01a306898cc6528d3 not found: ID does not exist" containerID="8830bdd0ca9644657cfcd5f9699476f07ee1b8cff8aee4d01a306898cc6528d3"
May 25 12:23:04.774096 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:04.774068    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8830bdd0ca9644657cfcd5f9699476f07ee1b8cff8aee4d01a306898cc6528d3"} err="failed to get container status \"8830bdd0ca9644657cfcd5f9699476f07ee1b8cff8aee4d01a306898cc6528d3\": rpc error: code = NotFound desc = could not find container \"8830bdd0ca9644657cfcd5f9699476f07ee1b8cff8aee4d01a306898cc6528d3\": container with ID starting with 8830bdd0ca9644657cfcd5f9699476f07ee1b8cff8aee4d01a306898cc6528d3 not found: ID does not exist"
May 25 12:23:04.774135 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:04.774100    2578 scope.go:117] "RemoveContainer" containerID="04830498fdb51ee47be891ff33c73f3f5a8143ce63583d2278a9d3d8b230ce02"
May 25 12:23:04.774304 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:23:04.774288    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04830498fdb51ee47be891ff33c73f3f5a8143ce63583d2278a9d3d8b230ce02\": container with ID starting with 04830498fdb51ee47be891ff33c73f3f5a8143ce63583d2278a9d3d8b230ce02 not found: ID does not exist" containerID="04830498fdb51ee47be891ff33c73f3f5a8143ce63583d2278a9d3d8b230ce02"
May 25 12:23:04.774350 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:04.774310    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04830498fdb51ee47be891ff33c73f3f5a8143ce63583d2278a9d3d8b230ce02"} err="failed to get container status \"04830498fdb51ee47be891ff33c73f3f5a8143ce63583d2278a9d3d8b230ce02\": rpc error: code = NotFound desc = could not find container \"04830498fdb51ee47be891ff33c73f3f5a8143ce63583d2278a9d3d8b230ce02\": container with ID starting with 04830498fdb51ee47be891ff33c73f3f5a8143ce63583d2278a9d3d8b230ce02 not found: ID does not exist"
May 25 12:23:04.774350 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:04.774324    2578 scope.go:117] "RemoveContainer" containerID="d23fcbe2ce3f6b39e31682713848ce25a98be7516bc232b5f4004bb765e6086b"
May 25 12:23:04.774558 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:23:04.774539    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d23fcbe2ce3f6b39e31682713848ce25a98be7516bc232b5f4004bb765e6086b\": container with ID starting with d23fcbe2ce3f6b39e31682713848ce25a98be7516bc232b5f4004bb765e6086b not found: ID does not exist" containerID="d23fcbe2ce3f6b39e31682713848ce25a98be7516bc232b5f4004bb765e6086b"
May 25 12:23:04.774603 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:04.774563    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d23fcbe2ce3f6b39e31682713848ce25a98be7516bc232b5f4004bb765e6086b"} err="failed to get container status \"d23fcbe2ce3f6b39e31682713848ce25a98be7516bc232b5f4004bb765e6086b\": rpc error: code = NotFound desc = could not find container \"d23fcbe2ce3f6b39e31682713848ce25a98be7516bc232b5f4004bb765e6086b\": container with ID starting with d23fcbe2ce3f6b39e31682713848ce25a98be7516bc232b5f4004bb765e6086b not found: ID does not exist"
May 25 12:23:04.818776 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:04.818685    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35d8edba-6dc8-441f-b65f-9d4e667b4dcf-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") pod \"35d8edba-6dc8-441f-b65f-9d4e667b4dcf\" (UID: \"35d8edba-6dc8-441f-b65f-9d4e667b4dcf\") "
May 25 12:23:04.818776 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:04.818753    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35d8edba-6dc8-441f-b65f-9d4e667b4dcf-kserve-provision-location\") pod \"35d8edba-6dc8-441f-b65f-9d4e667b4dcf\" (UID: \"35d8edba-6dc8-441f-b65f-9d4e667b4dcf\") "
May 25 12:23:04.818776 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:04.818776    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxrp6\" (UniqueName: \"kubernetes.io/projected/35d8edba-6dc8-441f-b65f-9d4e667b4dcf-kube-api-access-bxrp6\") pod \"35d8edba-6dc8-441f-b65f-9d4e667b4dcf\" (UID: \"35d8edba-6dc8-441f-b65f-9d4e667b4dcf\") "
May 25 12:23:04.819035 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:04.818830    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35d8edba-6dc8-441f-b65f-9d4e667b4dcf-proxy-tls\") pod \"35d8edba-6dc8-441f-b65f-9d4e667b4dcf\" (UID: \"35d8edba-6dc8-441f-b65f-9d4e667b4dcf\") "
May 25 12:23:04.819125 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:04.819101    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35d8edba-6dc8-441f-b65f-9d4e667b4dcf-isvc-xgboost-graph-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-graph-kube-rbac-proxy-sar-config") pod "35d8edba-6dc8-441f-b65f-9d4e667b4dcf" (UID: "35d8edba-6dc8-441f-b65f-9d4e667b4dcf"). InnerVolumeSpecName "isvc-xgboost-graph-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:23:04.819125 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:04.819113    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d8edba-6dc8-441f-b65f-9d4e667b4dcf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "35d8edba-6dc8-441f-b65f-9d4e667b4dcf" (UID: "35d8edba-6dc8-441f-b65f-9d4e667b4dcf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 12:23:04.820888 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:04.820864    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35d8edba-6dc8-441f-b65f-9d4e667b4dcf-kube-api-access-bxrp6" (OuterVolumeSpecName: "kube-api-access-bxrp6") pod "35d8edba-6dc8-441f-b65f-9d4e667b4dcf" (UID: "35d8edba-6dc8-441f-b65f-9d4e667b4dcf"). InnerVolumeSpecName "kube-api-access-bxrp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 12:23:04.820974 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:04.820890    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35d8edba-6dc8-441f-b65f-9d4e667b4dcf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "35d8edba-6dc8-441f-b65f-9d4e667b4dcf" (UID: "35d8edba-6dc8-441f-b65f-9d4e667b4dcf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:23:04.919619 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:04.919565    2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35d8edba-6dc8-441f-b65f-9d4e667b4dcf-kserve-provision-location\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:23:04.919619 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:04.919611    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxrp6\" (UniqueName: \"kubernetes.io/projected/35d8edba-6dc8-441f-b65f-9d4e667b4dcf-kube-api-access-bxrp6\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:23:04.919619 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:04.919622    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35d8edba-6dc8-441f-b65f-9d4e667b4dcf-proxy-tls\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:23:04.919619 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:04.919632    2578 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/35d8edba-6dc8-441f-b65f-9d4e667b4dcf-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:23:05.071332 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:05.071241    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5"]
May 25 12:23:05.073635 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:05.073608    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-x8tg5"]
May 25 12:23:06.939977 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:06.939932    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35d8edba-6dc8-441f-b65f-9d4e667b4dcf" path="/var/lib/kubelet/pods/35d8edba-6dc8-441f-b65f-9d4e667b4dcf/volumes"
May 25 12:23:07.718186 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:07.718145    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5" podUID="ee9e2f49-0644-46d5-9659-1d685023f00f" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:23:12.717742 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:12.717683    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5" podUID="ee9e2f49-0644-46d5-9659-1d685023f00f" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:23:12.718147 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:12.717803    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5"
May 25 12:23:17.717601 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:17.717517    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5" podUID="ee9e2f49-0644-46d5-9659-1d685023f00f" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:23:22.717679 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:22.717639    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5" podUID="ee9e2f49-0644-46d5-9659-1d685023f00f" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:23:27.717362 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:27.717320    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5" podUID="ee9e2f49-0644-46d5-9659-1d685023f00f" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:23:30.995870 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:30.995846    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5"
May 25 12:23:31.105925 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.105898    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee9e2f49-0644-46d5-9659-1d685023f00f-proxy-tls\") pod \"ee9e2f49-0644-46d5-9659-1d685023f00f\" (UID: \"ee9e2f49-0644-46d5-9659-1d685023f00f\") "
May 25 12:23:31.106105 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.105957    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee9e2f49-0644-46d5-9659-1d685023f00f-openshift-service-ca-bundle\") pod \"ee9e2f49-0644-46d5-9659-1d685023f00f\" (UID: \"ee9e2f49-0644-46d5-9659-1d685023f00f\") "
May 25 12:23:31.106333 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.106311    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee9e2f49-0644-46d5-9659-1d685023f00f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "ee9e2f49-0644-46d5-9659-1d685023f00f" (UID: "ee9e2f49-0644-46d5-9659-1d685023f00f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:23:31.107873 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.107852    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee9e2f49-0644-46d5-9659-1d685023f00f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ee9e2f49-0644-46d5-9659-1d685023f00f" (UID: "ee9e2f49-0644-46d5-9659-1d685023f00f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:23:31.119032 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.118976    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z"]
May 25 12:23:31.119249 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.119236    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35d8edba-6dc8-441f-b65f-9d4e667b4dcf" containerName="kube-rbac-proxy"
May 25 12:23:31.119297 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.119251    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="35d8edba-6dc8-441f-b65f-9d4e667b4dcf" containerName="kube-rbac-proxy"
May 25 12:23:31.119297 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.119261    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee9e2f49-0644-46d5-9659-1d685023f00f" containerName="model-chainer"
May 25 12:23:31.119297 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.119266    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee9e2f49-0644-46d5-9659-1d685023f00f" containerName="model-chainer"
May 25 12:23:31.119426 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.119302    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cffb6412-1856-40a9-bc55-12b353ed3168" containerName="switch-graph-1742e"
May 25 12:23:31.119426 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.119309    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cffb6412-1856-40a9-bc55-12b353ed3168" containerName="switch-graph-1742e"
May 25 12:23:31.119426 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.119317    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35d8edba-6dc8-441f-b65f-9d4e667b4dcf" containerName="storage-initializer"
May 25 12:23:31.119426 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.119322    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="35d8edba-6dc8-441f-b65f-9d4e667b4dcf" containerName="storage-initializer"
May 25 12:23:31.119426 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.119330    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35d8edba-6dc8-441f-b65f-9d4e667b4dcf" containerName="kserve-container"
May 25 12:23:31.119426 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.119334    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="35d8edba-6dc8-441f-b65f-9d4e667b4dcf" containerName="kserve-container"
May 25 12:23:31.119426 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.119374    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="35d8edba-6dc8-441f-b65f-9d4e667b4dcf" containerName="kserve-container"
May 25 12:23:31.119426 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.119382    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="35d8edba-6dc8-441f-b65f-9d4e667b4dcf" containerName="kube-rbac-proxy"
May 25 12:23:31.119426 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.119389    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="cffb6412-1856-40a9-bc55-12b353ed3168" containerName="switch-graph-1742e"
May 25 12:23:31.119426 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.119395    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="ee9e2f49-0644-46d5-9659-1d685023f00f" containerName="model-chainer"
May 25 12:23:31.122252 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.122236    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z"
May 25 12:23:31.124648 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.124629    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-40bda-kube-rbac-proxy-sar-config\""
May 25 12:23:31.124882 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.124865    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-40bda-serving-cert\""
May 25 12:23:31.130898 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.130849    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z"]
May 25 12:23:31.206799 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.206769    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee9e2f49-0644-46d5-9659-1d685023f00f-proxy-tls\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:23:31.206799 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.206792    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee9e2f49-0644-46d5-9659-1d685023f00f-openshift-service-ca-bundle\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:23:31.307196 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.307159    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0-openshift-service-ca-bundle\") pod \"switch-graph-40bda-7ccd57fd7d-9l88z\" (UID: \"e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0\") " pod="kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z"
May 25 12:23:31.307358 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.307217    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0-proxy-tls\") pod \"switch-graph-40bda-7ccd57fd7d-9l88z\" (UID: \"e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0\") " pod="kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z"
May 25 12:23:31.408134 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.408037    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0-openshift-service-ca-bundle\") pod \"switch-graph-40bda-7ccd57fd7d-9l88z\" (UID: \"e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0\") " pod="kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z"
May 25 12:23:31.408258 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.408209    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0-proxy-tls\") pod \"switch-graph-40bda-7ccd57fd7d-9l88z\" (UID: \"e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0\") " pod="kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z"
May 25 12:23:31.408717 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.408692    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0-openshift-service-ca-bundle\") pod \"switch-graph-40bda-7ccd57fd7d-9l88z\" (UID: \"e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0\") " pod="kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z"
May 25 12:23:31.410506 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.410489    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0-proxy-tls\") pod \"switch-graph-40bda-7ccd57fd7d-9l88z\" (UID: \"e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0\") " pod="kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z"
May 25 12:23:31.432543 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.432522    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z"
May 25 12:23:31.554471 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.554437    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z"]
May 25 12:23:31.557269 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:23:31.557242    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8fd5f67_0205_4bc4_bed9_8bd0e53a55a0.slice/crio-aade8a054986888ab32a05999ff47c2b9c4ec937d9e0707854dd746a2ff8d86c WatchSource:0}: Error finding container aade8a054986888ab32a05999ff47c2b9c4ec937d9e0707854dd746a2ff8d86c: Status 404 returned error can't find the container with id aade8a054986888ab32a05999ff47c2b9c4ec937d9e0707854dd746a2ff8d86c
May 25 12:23:31.558883 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.558866    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:23:31.827510 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.827455    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z" event={"ID":"e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0","Type":"ContainerStarted","Data":"a7311e847cff69dda6e800966f731ce183297d3d307761797b3e2a2ea2b01dcb"}
May 25 12:23:31.827510 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.827507    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z"
May 25 12:23:31.827767 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.827522    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z" event={"ID":"e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0","Type":"ContainerStarted","Data":"aade8a054986888ab32a05999ff47c2b9c4ec937d9e0707854dd746a2ff8d86c"}
May 25 12:23:31.828543 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.828523    2578 generic.go:358] "Generic (PLEG): container finished" podID="ee9e2f49-0644-46d5-9659-1d685023f00f" containerID="6d3b42cee67c4dd307e393491cbe116b1c12d39edc1e81e585fca26cdcdef002" exitCode=0
May 25 12:23:31.828620 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.828556    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5" event={"ID":"ee9e2f49-0644-46d5-9659-1d685023f00f","Type":"ContainerDied","Data":"6d3b42cee67c4dd307e393491cbe116b1c12d39edc1e81e585fca26cdcdef002"}
May 25 12:23:31.828620 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.828571    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5" event={"ID":"ee9e2f49-0644-46d5-9659-1d685023f00f","Type":"ContainerDied","Data":"c1e45c064be2a9f8e6a8321e179f623b72c7f1dc0761ca1153ef1a90cf8aefe3"}
May 25 12:23:31.828620 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.828582    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5"
May 25 12:23:31.828717 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.828585    2578 scope.go:117] "RemoveContainer" containerID="6d3b42cee67c4dd307e393491cbe116b1c12d39edc1e81e585fca26cdcdef002"
May 25 12:23:31.836112 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.836095    2578 scope.go:117] "RemoveContainer" containerID="6d3b42cee67c4dd307e393491cbe116b1c12d39edc1e81e585fca26cdcdef002"
May 25 12:23:31.836353 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:23:31.836335    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d3b42cee67c4dd307e393491cbe116b1c12d39edc1e81e585fca26cdcdef002\": container with ID starting with 6d3b42cee67c4dd307e393491cbe116b1c12d39edc1e81e585fca26cdcdef002 not found: ID does not exist" containerID="6d3b42cee67c4dd307e393491cbe116b1c12d39edc1e81e585fca26cdcdef002"
May 25 12:23:31.836416 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.836364    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d3b42cee67c4dd307e393491cbe116b1c12d39edc1e81e585fca26cdcdef002"} err="failed to get container status \"6d3b42cee67c4dd307e393491cbe116b1c12d39edc1e81e585fca26cdcdef002\": rpc error: code = NotFound desc = could not find container \"6d3b42cee67c4dd307e393491cbe116b1c12d39edc1e81e585fca26cdcdef002\": container with ID starting with 6d3b42cee67c4dd307e393491cbe116b1c12d39edc1e81e585fca26cdcdef002 not found: ID does not exist"
May 25 12:23:31.846032 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.845990    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z" podStartSLOduration=0.845979157 podStartE2EDuration="845.979157ms" podCreationTimestamp="2026-05-25 12:23:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:23:31.844698224 +0000 UTC m=+633.485892992" watchObservedRunningTime="2026-05-25 12:23:31.845979157 +0000 UTC m=+633.487173927"
May 25 12:23:31.857620 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.857595    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5"]
May 25 12:23:31.861961 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:31.861943    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-77c65d8d88-vkzr5"]
May 25 12:23:32.942827 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:32.942784    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee9e2f49-0644-46d5-9659-1d685023f00f" path="/var/lib/kubelet/pods/ee9e2f49-0644-46d5-9659-1d685023f00f/volumes"
May 25 12:23:37.836822 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:23:37.836793    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z"
May 25 12:24:01.137737 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:24:01.137694    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf"]
May 25 12:24:01.142276 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:24:01.142258    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf"
May 25 12:24:01.153473 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:24:01.153454    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-833d6-kube-rbac-proxy-sar-config\""
May 25 12:24:01.153570 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:24:01.153456    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-833d6-serving-cert\""
May 25 12:24:01.164517 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:24:01.164487    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf"]
May 25 12:24:01.224719 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:24:01.224688    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6aad06eb-8734-4662-91ef-21751d4c79a3-proxy-tls\") pod \"sequence-graph-833d6-8bd85f5d6-nq6xf\" (UID: \"6aad06eb-8734-4662-91ef-21751d4c79a3\") " pod="kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf"
May 25 12:24:01.224913 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:24:01.224744    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aad06eb-8734-4662-91ef-21751d4c79a3-openshift-service-ca-bundle\") pod \"sequence-graph-833d6-8bd85f5d6-nq6xf\" (UID: \"6aad06eb-8734-4662-91ef-21751d4c79a3\") " pod="kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf"
May 25 12:24:01.325320 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:24:01.325262    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aad06eb-8734-4662-91ef-21751d4c79a3-openshift-service-ca-bundle\") pod \"sequence-graph-833d6-8bd85f5d6-nq6xf\" (UID: \"6aad06eb-8734-4662-91ef-21751d4c79a3\") " pod="kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf"
May 25 12:24:01.325490 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:24:01.325362    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6aad06eb-8734-4662-91ef-21751d4c79a3-proxy-tls\") pod \"sequence-graph-833d6-8bd85f5d6-nq6xf\" (UID: \"6aad06eb-8734-4662-91ef-21751d4c79a3\") " pod="kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf"
May 25 12:24:01.325490 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:24:01.325471    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-833d6-serving-cert: secret "sequence-graph-833d6-serving-cert" not found
May 25 12:24:01.325564 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:24:01.325554    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aad06eb-8734-4662-91ef-21751d4c79a3-proxy-tls podName:6aad06eb-8734-4662-91ef-21751d4c79a3 nodeName:}" failed. No retries permitted until 2026-05-25 12:24:01.825531079 +0000 UTC m=+663.466725836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6aad06eb-8734-4662-91ef-21751d4c79a3-proxy-tls") pod "sequence-graph-833d6-8bd85f5d6-nq6xf" (UID: "6aad06eb-8734-4662-91ef-21751d4c79a3") : secret "sequence-graph-833d6-serving-cert" not found
May 25 12:24:01.326119 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:24:01.326099    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aad06eb-8734-4662-91ef-21751d4c79a3-openshift-service-ca-bundle\") pod \"sequence-graph-833d6-8bd85f5d6-nq6xf\" (UID: \"6aad06eb-8734-4662-91ef-21751d4c79a3\") " pod="kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf"
May 25 12:24:01.827953 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:24:01.827899    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6aad06eb-8734-4662-91ef-21751d4c79a3-proxy-tls\") pod \"sequence-graph-833d6-8bd85f5d6-nq6xf\" (UID: \"6aad06eb-8734-4662-91ef-21751d4c79a3\") " pod="kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf"
May 25 12:24:01.830368 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:24:01.830345    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6aad06eb-8734-4662-91ef-21751d4c79a3-proxy-tls\") pod \"sequence-graph-833d6-8bd85f5d6-nq6xf\" (UID: \"6aad06eb-8734-4662-91ef-21751d4c79a3\") " pod="kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf"
May 25 12:24:02.051543 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:24:02.051503    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf"
May 25 12:24:02.186446 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:24:02.186424    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf"]
May 25 12:24:02.187957 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:24:02.187931    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aad06eb_8734_4662_91ef_21751d4c79a3.slice/crio-b04df47209fbe702ae1bbac391cdc5616d0ec7dfc7a044553ad32ef0e38736eb WatchSource:0}: Error finding container b04df47209fbe702ae1bbac391cdc5616d0ec7dfc7a044553ad32ef0e38736eb: Status 404 returned error can't find the container with id b04df47209fbe702ae1bbac391cdc5616d0ec7dfc7a044553ad32ef0e38736eb
May 25 12:24:02.913685 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:24:02.913645    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf" event={"ID":"6aad06eb-8734-4662-91ef-21751d4c79a3","Type":"ContainerStarted","Data":"66499ae104ffa0bedb63644d07df639a5049221fb26c289da43d7484e23a15e7"}
May 25 12:24:02.913685 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:24:02.913681    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf" event={"ID":"6aad06eb-8734-4662-91ef-21751d4c79a3","Type":"ContainerStarted","Data":"b04df47209fbe702ae1bbac391cdc5616d0ec7dfc7a044553ad32ef0e38736eb"}
May 25 12:24:02.913951 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:24:02.913712    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf"
May 25 12:24:02.945551 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:24:02.945501    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf" podStartSLOduration=1.945488551 podStartE2EDuration="1.945488551s" podCreationTimestamp="2026-05-25 12:24:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:24:02.944750482 +0000 UTC m=+664.585945282" watchObservedRunningTime="2026-05-25 12:24:02.945488551 +0000 UTC m=+664.586683328"
May 25 12:24:08.921911 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:24:08.921874    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf"
May 25 12:27:58.923126 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:27:58.923099    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/ovn-acl-logging/0.log"
May 25 12:27:58.925649 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:27:58.925627    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/ovn-acl-logging/0.log"
May 25 12:31:45.772086 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:31:45.772031    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z"]
May 25 12:31:45.774409 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:31:45.772284    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z" podUID="e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0" containerName="switch-graph-40bda" containerID="cri-o://a7311e847cff69dda6e800966f731ce183297d3d307761797b3e2a2ea2b01dcb" gracePeriod=30
May 25 12:31:47.836098 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:31:47.836046    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z" podUID="e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0" containerName="switch-graph-40bda" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:31:52.836074 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:31:52.836031    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z" podUID="e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0" containerName="switch-graph-40bda" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:31:57.835408 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:31:57.835369    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z" podUID="e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0" containerName="switch-graph-40bda" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:31:57.835852 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:31:57.835487    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z"
May 25 12:32:02.835616 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:02.835577    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z" podUID="e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0" containerName="switch-graph-40bda" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:32:07.835924 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:07.835882    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z" podUID="e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0" containerName="switch-graph-40bda" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:32:12.835524 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:12.835475    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z" podUID="e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0" containerName="switch-graph-40bda" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:32:15.832553 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:15.832517    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf"]
May 25 12:32:15.832973 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:15.832820    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf" podUID="6aad06eb-8734-4662-91ef-21751d4c79a3" containerName="sequence-graph-833d6" containerID="cri-o://66499ae104ffa0bedb63644d07df639a5049221fb26c289da43d7484e23a15e7" gracePeriod=30
May 25 12:32:15.917456 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:15.917433    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z"
May 25 12:32:16.017569 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:16.017537    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0-openshift-service-ca-bundle\") pod \"e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0\" (UID: \"e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0\") "
May 25 12:32:16.017763 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:16.017602    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0-proxy-tls\") pod \"e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0\" (UID: \"e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0\") "
May 25 12:32:16.017980 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:16.017948    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0" (UID: "e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:32:16.019560 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:16.019533    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0" (UID: "e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:32:16.118397 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:16.118368    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0-proxy-tls\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:32:16.118397 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:16.118394    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0-openshift-service-ca-bundle\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:32:16.229457 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:16.229423    2578 generic.go:358] "Generic (PLEG): container finished" podID="e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0" containerID="a7311e847cff69dda6e800966f731ce183297d3d307761797b3e2a2ea2b01dcb" exitCode=0
May 25 12:32:16.229610 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:16.229483    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z"
May 25 12:32:16.229610 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:16.229485    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z" event={"ID":"e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0","Type":"ContainerDied","Data":"a7311e847cff69dda6e800966f731ce183297d3d307761797b3e2a2ea2b01dcb"}
May 25 12:32:16.229610 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:16.229583    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z" event={"ID":"e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0","Type":"ContainerDied","Data":"aade8a054986888ab32a05999ff47c2b9c4ec937d9e0707854dd746a2ff8d86c"}
May 25 12:32:16.229610 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:16.229596    2578 scope.go:117] "RemoveContainer" containerID="a7311e847cff69dda6e800966f731ce183297d3d307761797b3e2a2ea2b01dcb"
May 25 12:32:16.237627 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:16.237596    2578 scope.go:117] "RemoveContainer" containerID="a7311e847cff69dda6e800966f731ce183297d3d307761797b3e2a2ea2b01dcb"
May 25 12:32:16.237908 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:32:16.237876    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7311e847cff69dda6e800966f731ce183297d3d307761797b3e2a2ea2b01dcb\": container with ID starting with a7311e847cff69dda6e800966f731ce183297d3d307761797b3e2a2ea2b01dcb not found: ID does not exist" containerID="a7311e847cff69dda6e800966f731ce183297d3d307761797b3e2a2ea2b01dcb"
May 25 12:32:16.238001 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:16.237918    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7311e847cff69dda6e800966f731ce183297d3d307761797b3e2a2ea2b01dcb"} err="failed to get container status \"a7311e847cff69dda6e800966f731ce183297d3d307761797b3e2a2ea2b01dcb\": rpc error: code = NotFound desc = could not find container \"a7311e847cff69dda6e800966f731ce183297d3d307761797b3e2a2ea2b01dcb\": container with ID starting with a7311e847cff69dda6e800966f731ce183297d3d307761797b3e2a2ea2b01dcb not found: ID does not exist"
May 25 12:32:16.256479 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:16.256451    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z"]
May 25 12:32:16.269112 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:16.269083    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-40bda-7ccd57fd7d-9l88z"]
May 25 12:32:16.938848 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:16.938808    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0" path="/var/lib/kubelet/pods/e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0/volumes"
May 25 12:32:18.919914 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:18.919876    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf" podUID="6aad06eb-8734-4662-91ef-21751d4c79a3" containerName="sequence-graph-833d6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:32:23.920337 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:23.920298    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf" podUID="6aad06eb-8734-4662-91ef-21751d4c79a3" containerName="sequence-graph-833d6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:32:28.920012 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:28.919970    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf" podUID="6aad06eb-8734-4662-91ef-21751d4c79a3" containerName="sequence-graph-833d6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:32:28.920468 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:28.920088    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf"
May 25 12:32:33.920715 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:33.920678    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf" podUID="6aad06eb-8734-4662-91ef-21751d4c79a3" containerName="sequence-graph-833d6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:32:38.920328 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:38.920292    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf" podUID="6aad06eb-8734-4662-91ef-21751d4c79a3" containerName="sequence-graph-833d6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:32:43.920465 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:43.920424    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf" podUID="6aad06eb-8734-4662-91ef-21751d4c79a3" containerName="sequence-graph-833d6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:32:45.975069 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:45.975031    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf"
May 25 12:32:46.072445 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.072410    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv"]
May 25 12:32:46.072663 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.072651    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aad06eb-8734-4662-91ef-21751d4c79a3" containerName="sequence-graph-833d6"
May 25 12:32:46.072710 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.072665    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aad06eb-8734-4662-91ef-21751d4c79a3" containerName="sequence-graph-833d6"
May 25 12:32:46.072710 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.072681    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0" containerName="switch-graph-40bda"
May 25 12:32:46.072710 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.072687    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0" containerName="switch-graph-40bda"
May 25 12:32:46.072795 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.072726    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="6aad06eb-8734-4662-91ef-21751d4c79a3" containerName="sequence-graph-833d6"
May 25 12:32:46.072795 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.072733    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="e8fd5f67-0205-4bc4-bed9-8bd0e53a55a0" containerName="switch-graph-40bda"
May 25 12:32:46.075594 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.075577    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv"
May 25 12:32:46.079205 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.079179    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-bcff4-serving-cert\""
May 25 12:32:46.079433 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.079415    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-bcff4-kube-rbac-proxy-sar-config\""
May 25 12:32:46.087627 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.087570    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv"]
May 25 12:32:46.143451 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.143413    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6aad06eb-8734-4662-91ef-21751d4c79a3-proxy-tls\") pod \"6aad06eb-8734-4662-91ef-21751d4c79a3\" (UID: \"6aad06eb-8734-4662-91ef-21751d4c79a3\") "
May 25 12:32:46.143622 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.143464    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aad06eb-8734-4662-91ef-21751d4c79a3-openshift-service-ca-bundle\") pod \"6aad06eb-8734-4662-91ef-21751d4c79a3\" (UID: \"6aad06eb-8734-4662-91ef-21751d4c79a3\") "
May 25 12:32:46.143859 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.143835    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aad06eb-8734-4662-91ef-21751d4c79a3-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "6aad06eb-8734-4662-91ef-21751d4c79a3" (UID: "6aad06eb-8734-4662-91ef-21751d4c79a3"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:32:46.145513 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.145496    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aad06eb-8734-4662-91ef-21751d4c79a3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6aad06eb-8734-4662-91ef-21751d4c79a3" (UID: "6aad06eb-8734-4662-91ef-21751d4c79a3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:32:46.244744 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.244706    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bb89a39-f92b-4976-8048-d2a5d7971dd2-proxy-tls\") pod \"ensemble-graph-bcff4-76cb6fc4b6-xrpnv\" (UID: \"2bb89a39-f92b-4976-8048-d2a5d7971dd2\") " pod="kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv"
May 25 12:32:46.244915 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.244788    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bb89a39-f92b-4976-8048-d2a5d7971dd2-openshift-service-ca-bundle\") pod \"ensemble-graph-bcff4-76cb6fc4b6-xrpnv\" (UID: \"2bb89a39-f92b-4976-8048-d2a5d7971dd2\") " pod="kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv"
May 25 12:32:46.244915 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.244831    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6aad06eb-8734-4662-91ef-21751d4c79a3-proxy-tls\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:32:46.244915 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.244842    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aad06eb-8734-4662-91ef-21751d4c79a3-openshift-service-ca-bundle\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:32:46.313105 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.313042    2578 generic.go:358] "Generic (PLEG): container finished" podID="6aad06eb-8734-4662-91ef-21751d4c79a3" containerID="66499ae104ffa0bedb63644d07df639a5049221fb26c289da43d7484e23a15e7" exitCode=0
May 25 12:32:46.313271 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.313127    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf"
May 25 12:32:46.313271 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.313128    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf" event={"ID":"6aad06eb-8734-4662-91ef-21751d4c79a3","Type":"ContainerDied","Data":"66499ae104ffa0bedb63644d07df639a5049221fb26c289da43d7484e23a15e7"}
May 25 12:32:46.313271 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.313166    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf" event={"ID":"6aad06eb-8734-4662-91ef-21751d4c79a3","Type":"ContainerDied","Data":"b04df47209fbe702ae1bbac391cdc5616d0ec7dfc7a044553ad32ef0e38736eb"}
May 25 12:32:46.313271 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.313181    2578 scope.go:117] "RemoveContainer" containerID="66499ae104ffa0bedb63644d07df639a5049221fb26c289da43d7484e23a15e7"
May 25 12:32:46.320726 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.320705    2578 scope.go:117] "RemoveContainer" containerID="66499ae104ffa0bedb63644d07df639a5049221fb26c289da43d7484e23a15e7"
May 25 12:32:46.320988 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:32:46.320965    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66499ae104ffa0bedb63644d07df639a5049221fb26c289da43d7484e23a15e7\": container with ID starting with 66499ae104ffa0bedb63644d07df639a5049221fb26c289da43d7484e23a15e7 not found: ID does not exist" containerID="66499ae104ffa0bedb63644d07df639a5049221fb26c289da43d7484e23a15e7"
May 25 12:32:46.321034 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.321007    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66499ae104ffa0bedb63644d07df639a5049221fb26c289da43d7484e23a15e7"} err="failed to get container status \"66499ae104ffa0bedb63644d07df639a5049221fb26c289da43d7484e23a15e7\": rpc error: code = NotFound desc = could not find container \"66499ae104ffa0bedb63644d07df639a5049221fb26c289da43d7484e23a15e7\": container with ID starting with 66499ae104ffa0bedb63644d07df639a5049221fb26c289da43d7484e23a15e7 not found: ID does not exist"
May 25 12:32:46.344027 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.343965    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf"]
May 25 12:32:46.345153 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.345131    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bb89a39-f92b-4976-8048-d2a5d7971dd2-proxy-tls\") pod \"ensemble-graph-bcff4-76cb6fc4b6-xrpnv\" (UID: \"2bb89a39-f92b-4976-8048-d2a5d7971dd2\") " pod="kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv"
May 25 12:32:46.345223 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.345178    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bb89a39-f92b-4976-8048-d2a5d7971dd2-openshift-service-ca-bundle\") pod \"ensemble-graph-bcff4-76cb6fc4b6-xrpnv\" (UID: \"2bb89a39-f92b-4976-8048-d2a5d7971dd2\") " pod="kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv"
May 25 12:32:46.345293 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:32:46.345276    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-bcff4-serving-cert: secret "ensemble-graph-bcff4-serving-cert" not found
May 25 12:32:46.345360 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:32:46.345347    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bb89a39-f92b-4976-8048-d2a5d7971dd2-proxy-tls podName:2bb89a39-f92b-4976-8048-d2a5d7971dd2 nodeName:}" failed. No retries permitted until 2026-05-25 12:32:46.845325807 +0000 UTC m=+1188.486520571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2bb89a39-f92b-4976-8048-d2a5d7971dd2-proxy-tls") pod "ensemble-graph-bcff4-76cb6fc4b6-xrpnv" (UID: "2bb89a39-f92b-4976-8048-d2a5d7971dd2") : secret "ensemble-graph-bcff4-serving-cert" not found
May 25 12:32:46.345938 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.345916    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bb89a39-f92b-4976-8048-d2a5d7971dd2-openshift-service-ca-bundle\") pod \"ensemble-graph-bcff4-76cb6fc4b6-xrpnv\" (UID: \"2bb89a39-f92b-4976-8048-d2a5d7971dd2\") " pod="kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv"
May 25 12:32:46.346685 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.346668    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-833d6-8bd85f5d6-nq6xf"]
May 25 12:32:46.850766 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.850728    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bb89a39-f92b-4976-8048-d2a5d7971dd2-proxy-tls\") pod \"ensemble-graph-bcff4-76cb6fc4b6-xrpnv\" (UID: \"2bb89a39-f92b-4976-8048-d2a5d7971dd2\") " pod="kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv"
May 25 12:32:46.853149 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.853117    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bb89a39-f92b-4976-8048-d2a5d7971dd2-proxy-tls\") pod \"ensemble-graph-bcff4-76cb6fc4b6-xrpnv\" (UID: \"2bb89a39-f92b-4976-8048-d2a5d7971dd2\") " pod="kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv"
May 25 12:32:46.938975 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.938939    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6aad06eb-8734-4662-91ef-21751d4c79a3" path="/var/lib/kubelet/pods/6aad06eb-8734-4662-91ef-21751d4c79a3/volumes"
May 25 12:32:46.985454 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:46.985411    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv"
May 25 12:32:47.115374 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:47.115302    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv"]
May 25 12:32:47.118306 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:32:47.118266    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb89a39_f92b_4976_8048_d2a5d7971dd2.slice/crio-b01075c98f5b6d1a19356391724f7a9d4614c2136ed261c40f40a1e208e17984 WatchSource:0}: Error finding container b01075c98f5b6d1a19356391724f7a9d4614c2136ed261c40f40a1e208e17984: Status 404 returned error can't find the container with id b01075c98f5b6d1a19356391724f7a9d4614c2136ed261c40f40a1e208e17984
May 25 12:32:47.122007 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:47.121982    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:32:47.316773 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:47.316736    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv" event={"ID":"2bb89a39-f92b-4976-8048-d2a5d7971dd2","Type":"ContainerStarted","Data":"dc9be19dacb275e0a87d3420117d67cfd57a414ecccc94e76208ceacd5948e21"}
May 25 12:32:47.316773 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:47.316776    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv" event={"ID":"2bb89a39-f92b-4976-8048-d2a5d7971dd2","Type":"ContainerStarted","Data":"b01075c98f5b6d1a19356391724f7a9d4614c2136ed261c40f40a1e208e17984"}
May 25 12:32:47.317014 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:47.316826    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv"
May 25 12:32:47.341222 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:47.341174    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv" podStartSLOduration=1.341159601 podStartE2EDuration="1.341159601s" podCreationTimestamp="2026-05-25 12:32:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:32:47.339400319 +0000 UTC m=+1188.980595095" watchObservedRunningTime="2026-05-25 12:32:47.341159601 +0000 UTC m=+1188.982354378"
May 25 12:32:53.326248 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:53.326221    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv"
May 25 12:32:56.156305 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:56.156263    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv"]
May 25 12:32:56.156691 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:56.156514    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv" podUID="2bb89a39-f92b-4976-8048-d2a5d7971dd2" containerName="ensemble-graph-bcff4" containerID="cri-o://dc9be19dacb275e0a87d3420117d67cfd57a414ecccc94e76208ceacd5948e21" gracePeriod=30
May 25 12:32:58.324073 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:58.324021    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv" podUID="2bb89a39-f92b-4976-8048-d2a5d7971dd2" containerName="ensemble-graph-bcff4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:32:58.947076 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:58.947034    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/ovn-acl-logging/0.log"
May 25 12:32:58.950111 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:32:58.950048    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/ovn-acl-logging/0.log"
May 25 12:33:03.324958 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:03.324919    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv" podUID="2bb89a39-f92b-4976-8048-d2a5d7971dd2" containerName="ensemble-graph-bcff4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:33:08.324177 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:08.324128    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv" podUID="2bb89a39-f92b-4976-8048-d2a5d7971dd2" containerName="ensemble-graph-bcff4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:33:08.324587 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:08.324241    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv"
May 25 12:33:13.324520 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:13.324483    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv" podUID="2bb89a39-f92b-4976-8048-d2a5d7971dd2" containerName="ensemble-graph-bcff4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:33:18.324606 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:18.324563    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv" podUID="2bb89a39-f92b-4976-8048-d2a5d7971dd2" containerName="ensemble-graph-bcff4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:33:23.324140 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:23.324099    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv" podUID="2bb89a39-f92b-4976-8048-d2a5d7971dd2" containerName="ensemble-graph-bcff4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:33:26.031234 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:26.031192    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8"]
May 25 12:33:26.034408 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:26.034391    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8"
May 25 12:33:26.038114 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:26.038092    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-acbca-serving-cert\""
May 25 12:33:26.038229 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:26.038092    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-acbca-kube-rbac-proxy-sar-config\""
May 25 12:33:26.047559 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:26.047530    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8"]
May 25 12:33:26.132691 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:26.132653    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ef8b206-0424-407b-a392-ecc198007e76-openshift-service-ca-bundle\") pod \"sequence-graph-acbca-854dfbbf8-g5nv8\" (UID: \"0ef8b206-0424-407b-a392-ecc198007e76\") " pod="kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8"
May 25 12:33:26.132874 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:26.132704    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ef8b206-0424-407b-a392-ecc198007e76-proxy-tls\") pod \"sequence-graph-acbca-854dfbbf8-g5nv8\" (UID: \"0ef8b206-0424-407b-a392-ecc198007e76\") " pod="kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8"
May 25 12:33:26.179258 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:33:26.179213    2578 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb89a39_f92b_4976_8048_d2a5d7971dd2.slice/crio-dc9be19dacb275e0a87d3420117d67cfd57a414ecccc94e76208ceacd5948e21.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb89a39_f92b_4976_8048_d2a5d7971dd2.slice/crio-conmon-dc9be19dacb275e0a87d3420117d67cfd57a414ecccc94e76208ceacd5948e21.scope\": RecentStats: unable to find data in memory cache]"
May 25 12:33:26.179412 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:33:26.179389    2578 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb89a39_f92b_4976_8048_d2a5d7971dd2.slice/crio-dc9be19dacb275e0a87d3420117d67cfd57a414ecccc94e76208ceacd5948e21.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb89a39_f92b_4976_8048_d2a5d7971dd2.slice/crio-conmon-dc9be19dacb275e0a87d3420117d67cfd57a414ecccc94e76208ceacd5948e21.scope\": RecentStats: unable to find data in memory cache]"
May 25 12:33:26.179510 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:33:26.179421    2578 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb89a39_f92b_4976_8048_d2a5d7971dd2.slice/crio-dc9be19dacb275e0a87d3420117d67cfd57a414ecccc94e76208ceacd5948e21.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb89a39_f92b_4976_8048_d2a5d7971dd2.slice/crio-conmon-dc9be19dacb275e0a87d3420117d67cfd57a414ecccc94e76208ceacd5948e21.scope\": RecentStats: unable to find data in memory cache]"
May 25 12:33:26.233628 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:26.233588    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ef8b206-0424-407b-a392-ecc198007e76-openshift-service-ca-bundle\") pod \"sequence-graph-acbca-854dfbbf8-g5nv8\" (UID: \"0ef8b206-0424-407b-a392-ecc198007e76\") " pod="kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8"
May 25 12:33:26.233799 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:26.233644    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ef8b206-0424-407b-a392-ecc198007e76-proxy-tls\") pod \"sequence-graph-acbca-854dfbbf8-g5nv8\" (UID: \"0ef8b206-0424-407b-a392-ecc198007e76\") " pod="kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8"
May 25 12:33:26.234267 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:26.234245    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ef8b206-0424-407b-a392-ecc198007e76-openshift-service-ca-bundle\") pod \"sequence-graph-acbca-854dfbbf8-g5nv8\" (UID: \"0ef8b206-0424-407b-a392-ecc198007e76\") " pod="kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8"
May 25 12:33:26.235971 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:26.235952    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ef8b206-0424-407b-a392-ecc198007e76-proxy-tls\") pod \"sequence-graph-acbca-854dfbbf8-g5nv8\" (UID: \"0ef8b206-0424-407b-a392-ecc198007e76\") " pod="kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8"
May 25 12:33:26.344345 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:26.344305    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8"
May 25 12:33:26.430288 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:26.430258    2578 generic.go:358] "Generic (PLEG): container finished" podID="2bb89a39-f92b-4976-8048-d2a5d7971dd2" containerID="dc9be19dacb275e0a87d3420117d67cfd57a414ecccc94e76208ceacd5948e21" exitCode=0
May 25 12:33:26.430420 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:26.430297    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv" event={"ID":"2bb89a39-f92b-4976-8048-d2a5d7971dd2","Type":"ContainerDied","Data":"dc9be19dacb275e0a87d3420117d67cfd57a414ecccc94e76208ceacd5948e21"}
May 25 12:33:26.477404 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:26.477208    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8"]
May 25 12:33:26.479416 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:33:26.479388    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef8b206_0424_407b_a392_ecc198007e76.slice/crio-ef9976752d1b6dfcc1eee2142004d19e295ff09734368b5cb8a36715ea5a7d1f WatchSource:0}: Error finding container ef9976752d1b6dfcc1eee2142004d19e295ff09734368b5cb8a36715ea5a7d1f: Status 404 returned error can't find the container with id ef9976752d1b6dfcc1eee2142004d19e295ff09734368b5cb8a36715ea5a7d1f
May 25 12:33:26.790432 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:26.790402    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv"
May 25 12:33:26.837543 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:26.837509    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bb89a39-f92b-4976-8048-d2a5d7971dd2-proxy-tls\") pod \"2bb89a39-f92b-4976-8048-d2a5d7971dd2\" (UID: \"2bb89a39-f92b-4976-8048-d2a5d7971dd2\") "
May 25 12:33:26.837690 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:26.837578    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bb89a39-f92b-4976-8048-d2a5d7971dd2-openshift-service-ca-bundle\") pod \"2bb89a39-f92b-4976-8048-d2a5d7971dd2\" (UID: \"2bb89a39-f92b-4976-8048-d2a5d7971dd2\") "
May 25 12:33:26.837960 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:26.837933    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb89a39-f92b-4976-8048-d2a5d7971dd2-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "2bb89a39-f92b-4976-8048-d2a5d7971dd2" (UID: "2bb89a39-f92b-4976-8048-d2a5d7971dd2"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:33:26.839567 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:26.839548    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bb89a39-f92b-4976-8048-d2a5d7971dd2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2bb89a39-f92b-4976-8048-d2a5d7971dd2" (UID: "2bb89a39-f92b-4976-8048-d2a5d7971dd2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:33:26.938267 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:26.938185    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bb89a39-f92b-4976-8048-d2a5d7971dd2-openshift-service-ca-bundle\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:33:26.938267 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:26.938210    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bb89a39-f92b-4976-8048-d2a5d7971dd2-proxy-tls\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:33:27.434546 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:27.434482    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv" event={"ID":"2bb89a39-f92b-4976-8048-d2a5d7971dd2","Type":"ContainerDied","Data":"b01075c98f5b6d1a19356391724f7a9d4614c2136ed261c40f40a1e208e17984"}
May 25 12:33:27.434546 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:27.434497    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv"
May 25 12:33:27.434546 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:27.434527    2578 scope.go:117] "RemoveContainer" containerID="dc9be19dacb275e0a87d3420117d67cfd57a414ecccc94e76208ceacd5948e21"
May 25 12:33:27.435992 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:27.435959    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8" event={"ID":"0ef8b206-0424-407b-a392-ecc198007e76","Type":"ContainerStarted","Data":"dbcad11b908de7b0e0b8d5de5b24f9bd633ad1141cf9cbb91a01047e19fd66ea"}
May 25 12:33:27.435992 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:27.435990    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8" event={"ID":"0ef8b206-0424-407b-a392-ecc198007e76","Type":"ContainerStarted","Data":"ef9976752d1b6dfcc1eee2142004d19e295ff09734368b5cb8a36715ea5a7d1f"}
May 25 12:33:27.436177 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:27.436088    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8"
May 25 12:33:27.460074 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:27.460021    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8" podStartSLOduration=1.460005796 podStartE2EDuration="1.460005796s" podCreationTimestamp="2026-05-25 12:33:26 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:33:27.458482072 +0000 UTC m=+1229.099676849" watchObservedRunningTime="2026-05-25 12:33:27.460005796 +0000 UTC m=+1229.101200601"
May 25 12:33:27.475321 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:27.475287    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv"]
May 25 12:33:27.482532 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:27.482507    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-bcff4-76cb6fc4b6-xrpnv"]
May 25 12:33:28.939226 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:28.939195    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bb89a39-f92b-4976-8048-d2a5d7971dd2" path="/var/lib/kubelet/pods/2bb89a39-f92b-4976-8048-d2a5d7971dd2/volumes"
May 25 12:33:33.447820 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:33.447790    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8"
May 25 12:33:36.087649 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:36.087616    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8"]
May 25 12:33:36.088026 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:36.087815    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8" podUID="0ef8b206-0424-407b-a392-ecc198007e76" containerName="sequence-graph-acbca" containerID="cri-o://dbcad11b908de7b0e0b8d5de5b24f9bd633ad1141cf9cbb91a01047e19fd66ea" gracePeriod=30
May 25 12:33:38.446433 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:38.446388    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8" podUID="0ef8b206-0424-407b-a392-ecc198007e76" containerName="sequence-graph-acbca" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:33:43.446241 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:43.446202    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8" podUID="0ef8b206-0424-407b-a392-ecc198007e76" containerName="sequence-graph-acbca" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:33:48.446320 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:48.446234    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8" podUID="0ef8b206-0424-407b-a392-ecc198007e76" containerName="sequence-graph-acbca" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:33:48.446710 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:48.446343    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8"
May 25 12:33:53.445949 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:53.445908    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8" podUID="0ef8b206-0424-407b-a392-ecc198007e76" containerName="sequence-graph-acbca" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:33:58.445925 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:33:58.445886    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8" podUID="0ef8b206-0424-407b-a392-ecc198007e76" containerName="sequence-graph-acbca" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:34:03.446360 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:03.446313    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8" podUID="0ef8b206-0424-407b-a392-ecc198007e76" containerName="sequence-graph-acbca" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:34:06.119936 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:34:06.119908    2578 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef8b206_0424_407b_a392_ecc198007e76.slice/crio-dbcad11b908de7b0e0b8d5de5b24f9bd633ad1141cf9cbb91a01047e19fd66ea.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef8b206_0424_407b_a392_ecc198007e76.slice/crio-conmon-dbcad11b908de7b0e0b8d5de5b24f9bd633ad1141cf9cbb91a01047e19fd66ea.scope\": RecentStats: unable to find data in memory cache]"
May 25 12:34:06.236517 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.236493    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8"
May 25 12:34:06.288316 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.288287    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ef8b206-0424-407b-a392-ecc198007e76-openshift-service-ca-bundle\") pod \"0ef8b206-0424-407b-a392-ecc198007e76\" (UID: \"0ef8b206-0424-407b-a392-ecc198007e76\") "
May 25 12:34:06.288468 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.288325    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ef8b206-0424-407b-a392-ecc198007e76-proxy-tls\") pod \"0ef8b206-0424-407b-a392-ecc198007e76\" (UID: \"0ef8b206-0424-407b-a392-ecc198007e76\") "
May 25 12:34:06.288608 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.288586    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ef8b206-0424-407b-a392-ecc198007e76-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "0ef8b206-0424-407b-a392-ecc198007e76" (UID: "0ef8b206-0424-407b-a392-ecc198007e76"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:34:06.290426 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.290402    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ef8b206-0424-407b-a392-ecc198007e76-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0ef8b206-0424-407b-a392-ecc198007e76" (UID: "0ef8b206-0424-407b-a392-ecc198007e76"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:34:06.389602 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.389525    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ef8b206-0424-407b-a392-ecc198007e76-openshift-service-ca-bundle\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:34:06.389602 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.389555    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0ef8b206-0424-407b-a392-ecc198007e76-proxy-tls\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:34:06.518952 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.518920    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff"]
May 25 12:34:06.519200 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.519187    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ef8b206-0424-407b-a392-ecc198007e76" containerName="sequence-graph-acbca"
May 25 12:34:06.519251 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.519202    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ef8b206-0424-407b-a392-ecc198007e76" containerName="sequence-graph-acbca"
May 25 12:34:06.519251 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.519213    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bb89a39-f92b-4976-8048-d2a5d7971dd2" containerName="ensemble-graph-bcff4"
May 25 12:34:06.519251 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.519219    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bb89a39-f92b-4976-8048-d2a5d7971dd2" containerName="ensemble-graph-bcff4"
May 25 12:34:06.519339 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.519262    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="2bb89a39-f92b-4976-8048-d2a5d7971dd2" containerName="ensemble-graph-bcff4"
May 25 12:34:06.519339 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.519272    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="0ef8b206-0424-407b-a392-ecc198007e76" containerName="sequence-graph-acbca"
May 25 12:34:06.522383 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.522366    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff"
May 25 12:34:06.526279 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.526258    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-155ee-serving-cert\""
May 25 12:34:06.526397 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.526297    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-155ee-kube-rbac-proxy-sar-config\""
May 25 12:34:06.534933 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.534910    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff"]
May 25 12:34:06.546894 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.546871    2578 generic.go:358] "Generic (PLEG): container finished" podID="0ef8b206-0424-407b-a392-ecc198007e76" containerID="dbcad11b908de7b0e0b8d5de5b24f9bd633ad1141cf9cbb91a01047e19fd66ea" exitCode=137
May 25 12:34:06.547021 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.546920    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8"
May 25 12:34:06.547021 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.546924    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8" event={"ID":"0ef8b206-0424-407b-a392-ecc198007e76","Type":"ContainerDied","Data":"dbcad11b908de7b0e0b8d5de5b24f9bd633ad1141cf9cbb91a01047e19fd66ea"}
May 25 12:34:06.547021 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.546952    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8" event={"ID":"0ef8b206-0424-407b-a392-ecc198007e76","Type":"ContainerDied","Data":"ef9976752d1b6dfcc1eee2142004d19e295ff09734368b5cb8a36715ea5a7d1f"}
May 25 12:34:06.547021 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.546967    2578 scope.go:117] "RemoveContainer" containerID="dbcad11b908de7b0e0b8d5de5b24f9bd633ad1141cf9cbb91a01047e19fd66ea"
May 25 12:34:06.554819 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.554789    2578 scope.go:117] "RemoveContainer" containerID="dbcad11b908de7b0e0b8d5de5b24f9bd633ad1141cf9cbb91a01047e19fd66ea"
May 25 12:34:06.555224 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:34:06.555199    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbcad11b908de7b0e0b8d5de5b24f9bd633ad1141cf9cbb91a01047e19fd66ea\": container with ID starting with dbcad11b908de7b0e0b8d5de5b24f9bd633ad1141cf9cbb91a01047e19fd66ea not found: ID does not exist" containerID="dbcad11b908de7b0e0b8d5de5b24f9bd633ad1141cf9cbb91a01047e19fd66ea"
May 25 12:34:06.555318 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.555236    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbcad11b908de7b0e0b8d5de5b24f9bd633ad1141cf9cbb91a01047e19fd66ea"} err="failed to get container status \"dbcad11b908de7b0e0b8d5de5b24f9bd633ad1141cf9cbb91a01047e19fd66ea\": rpc error: code = NotFound desc = could not find container \"dbcad11b908de7b0e0b8d5de5b24f9bd633ad1141cf9cbb91a01047e19fd66ea\": container with ID starting with dbcad11b908de7b0e0b8d5de5b24f9bd633ad1141cf9cbb91a01047e19fd66ea not found: ID does not exist"
May 25 12:34:06.580425 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.580400    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8"]
May 25 12:34:06.584762 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.584742    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-acbca-854dfbbf8-g5nv8"]
May 25 12:34:06.590960 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.590941    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad-openshift-service-ca-bundle\") pod \"ensemble-graph-155ee-7766bf4486-wp9ff\" (UID: \"b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad\") " pod="kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff"
May 25 12:34:06.591014 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.590969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad-proxy-tls\") pod \"ensemble-graph-155ee-7766bf4486-wp9ff\" (UID: \"b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad\") " pod="kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff"
May 25 12:34:06.691794 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.691693    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad-openshift-service-ca-bundle\") pod \"ensemble-graph-155ee-7766bf4486-wp9ff\" (UID: \"b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad\") " pod="kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff"
May 25 12:34:06.691794 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.691742    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad-proxy-tls\") pod \"ensemble-graph-155ee-7766bf4486-wp9ff\" (UID: \"b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad\") " pod="kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff"
May 25 12:34:06.692347 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.692322    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad-openshift-service-ca-bundle\") pod \"ensemble-graph-155ee-7766bf4486-wp9ff\" (UID: \"b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad\") " pod="kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff"
May 25 12:34:06.694186 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.694166    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad-proxy-tls\") pod \"ensemble-graph-155ee-7766bf4486-wp9ff\" (UID: \"b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad\") " pod="kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff"
May 25 12:34:06.832040 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.832006    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff"
May 25 12:34:06.939315 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.939280    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ef8b206-0424-407b-a392-ecc198007e76" path="/var/lib/kubelet/pods/0ef8b206-0424-407b-a392-ecc198007e76/volumes"
May 25 12:34:06.962011 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:06.961933    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff"]
May 25 12:34:06.964589 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:34:06.964558    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb18c2fc2_81a9_4ce4_a454_d2d0a831c6ad.slice/crio-06848987193b914a4ba3b4764271de4b4577b8410656ddc9da72fa3fa75a2f5c WatchSource:0}: Error finding container 06848987193b914a4ba3b4764271de4b4577b8410656ddc9da72fa3fa75a2f5c: Status 404 returned error can't find the container with id 06848987193b914a4ba3b4764271de4b4577b8410656ddc9da72fa3fa75a2f5c
May 25 12:34:07.554906 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:07.554871    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff" event={"ID":"b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad","Type":"ContainerStarted","Data":"1596f0b9f7a7d9f850b4e1b3c2d9e3a98fb0b50989e93884b096d402f06ce469"}
May 25 12:34:07.554906 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:07.554909    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff" event={"ID":"b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad","Type":"ContainerStarted","Data":"06848987193b914a4ba3b4764271de4b4577b8410656ddc9da72fa3fa75a2f5c"}
May 25 12:34:07.555331 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:07.554935    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff"
May 25 12:34:07.577451 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:07.577408    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff" podStartSLOduration=1.577393566 podStartE2EDuration="1.577393566s" podCreationTimestamp="2026-05-25 12:34:06 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:34:07.575347923 +0000 UTC m=+1269.216542699" watchObservedRunningTime="2026-05-25 12:34:07.577393566 +0000 UTC m=+1269.218588343"
May 25 12:34:13.563610 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:13.563582    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff"
May 25 12:34:36.340972 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:36.340940    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l"]
May 25 12:34:36.343964 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:36.343948    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l"
May 25 12:34:36.347240 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:36.347218    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-4d75e-serving-cert\""
May 25 12:34:36.347347 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:36.347258    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-4d75e-kube-rbac-proxy-sar-config\""
May 25 12:34:36.356522 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:36.356502    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l"]
May 25 12:34:36.386217 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:36.386193    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7-proxy-tls\") pod \"sequence-graph-4d75e-7f95b4f9f7-6b88l\" (UID: \"cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7\") " pod="kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l"
May 25 12:34:36.386320 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:36.386232    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7-openshift-service-ca-bundle\") pod \"sequence-graph-4d75e-7f95b4f9f7-6b88l\" (UID: \"cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7\") " pod="kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l"
May 25 12:34:36.487750 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:36.487711    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7-proxy-tls\") pod \"sequence-graph-4d75e-7f95b4f9f7-6b88l\" (UID: \"cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7\") " pod="kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l"
May 25 12:34:36.487875 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:36.487779    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7-openshift-service-ca-bundle\") pod \"sequence-graph-4d75e-7f95b4f9f7-6b88l\" (UID: \"cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7\") " pod="kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l"
May 25 12:34:36.487875 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:34:36.487865    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-4d75e-serving-cert: secret "sequence-graph-4d75e-serving-cert" not found
May 25 12:34:36.488021 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:34:36.487933    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7-proxy-tls podName:cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7 nodeName:}" failed. No retries permitted until 2026-05-25 12:34:36.987912345 +0000 UTC m=+1298.629107114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7-proxy-tls") pod "sequence-graph-4d75e-7f95b4f9f7-6b88l" (UID: "cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7") : secret "sequence-graph-4d75e-serving-cert" not found
May 25 12:34:36.488455 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:36.488436    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7-openshift-service-ca-bundle\") pod \"sequence-graph-4d75e-7f95b4f9f7-6b88l\" (UID: \"cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7\") " pod="kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l"
May 25 12:34:36.992388 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:36.992351    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7-proxy-tls\") pod \"sequence-graph-4d75e-7f95b4f9f7-6b88l\" (UID: \"cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7\") " pod="kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l"
May 25 12:34:36.994743 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:36.994724    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7-proxy-tls\") pod \"sequence-graph-4d75e-7f95b4f9f7-6b88l\" (UID: \"cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7\") " pod="kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l"
May 25 12:34:37.253720 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:37.253636    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l"
May 25 12:34:37.377995 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:37.377963    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l"]
May 25 12:34:37.380993 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:34:37.380969    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce79cc8_362a_41f6_8a05_a6a5bd6b1fc7.slice/crio-1f744c105bf8d453740d87f8511be090d7645a2df18723070cc80af828626ea9 WatchSource:0}: Error finding container 1f744c105bf8d453740d87f8511be090d7645a2df18723070cc80af828626ea9: Status 404 returned error can't find the container with id 1f744c105bf8d453740d87f8511be090d7645a2df18723070cc80af828626ea9
May 25 12:34:37.633205 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:37.633170    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l" event={"ID":"cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7","Type":"ContainerStarted","Data":"dfa9230ff0d67a5abd8b3b1ac3d452711f0f358110f99e01c6cbccd6bfed40de"}
May 25 12:34:37.633205 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:37.633210    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l" event={"ID":"cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7","Type":"ContainerStarted","Data":"1f744c105bf8d453740d87f8511be090d7645a2df18723070cc80af828626ea9"}
May 25 12:34:37.633401 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:37.633273    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l"
May 25 12:34:37.653507 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:37.653461    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l" podStartSLOduration=1.6534486309999998 podStartE2EDuration="1.653448631s" podCreationTimestamp="2026-05-25 12:34:36 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:34:37.652624028 +0000 UTC m=+1299.293818807" watchObservedRunningTime="2026-05-25 12:34:37.653448631 +0000 UTC m=+1299.294643407"
May 25 12:34:43.642213 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:34:43.642182    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l"
May 25 12:37:58.967941 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:37:58.967913    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/ovn-acl-logging/0.log"
May 25 12:37:58.970752 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:37:58.970731    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/ovn-acl-logging/0.log"
May 25 12:42:21.076406 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:21.076373    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff"]
May 25 12:42:21.076876 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:21.076610    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff" podUID="b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad" containerName="ensemble-graph-155ee" containerID="cri-o://1596f0b9f7a7d9f850b4e1b3c2d9e3a98fb0b50989e93884b096d402f06ce469" gracePeriod=30
May 25 12:42:23.561688 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:23.561640    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff" podUID="b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad" containerName="ensemble-graph-155ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:42:28.561998 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:28.561956    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff" podUID="b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad" containerName="ensemble-graph-155ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:42:33.562151 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:33.562111    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff" podUID="b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad" containerName="ensemble-graph-155ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:42:33.562586 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:33.562242    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff"
May 25 12:42:38.561670 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:38.561631    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff" podUID="b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad" containerName="ensemble-graph-155ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:42:43.561999 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:43.561959    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff" podUID="b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad" containerName="ensemble-graph-155ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:42:48.561400 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:48.561314    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff" podUID="b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad" containerName="ensemble-graph-155ee" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:42:51.031337 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:51.031305    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l"]
May 25 12:42:51.031659 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:51.031561    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l" podUID="cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7" containerName="sequence-graph-4d75e" containerID="cri-o://dfa9230ff0d67a5abd8b3b1ac3d452711f0f358110f99e01c6cbccd6bfed40de" gracePeriod=30
May 25 12:42:51.236314 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:51.236291    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff"
May 25 12:42:51.242646 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:51.242626    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad-openshift-service-ca-bundle\") pod \"b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad\" (UID: \"b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad\") "
May 25 12:42:51.242774 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:51.242665    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad-proxy-tls\") pod \"b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad\" (UID: \"b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad\") "
May 25 12:42:51.243074 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:51.243030    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad" (UID: "b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:42:51.245032 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:51.245002    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad" (UID: "b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:42:51.343529 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:51.343498    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad-openshift-service-ca-bundle\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:42:51.343529 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:51.343527    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad-proxy-tls\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:42:51.940490 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:51.940455    2578 generic.go:358] "Generic (PLEG): container finished" podID="b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad" containerID="1596f0b9f7a7d9f850b4e1b3c2d9e3a98fb0b50989e93884b096d402f06ce469" exitCode=0
May 25 12:42:51.940694 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:51.940523    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff" event={"ID":"b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad","Type":"ContainerDied","Data":"1596f0b9f7a7d9f850b4e1b3c2d9e3a98fb0b50989e93884b096d402f06ce469"}
May 25 12:42:51.940694 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:51.940532    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff"
May 25 12:42:51.940694 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:51.940548    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff" event={"ID":"b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad","Type":"ContainerDied","Data":"06848987193b914a4ba3b4764271de4b4577b8410656ddc9da72fa3fa75a2f5c"}
May 25 12:42:51.940694 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:51.940562    2578 scope.go:117] "RemoveContainer" containerID="1596f0b9f7a7d9f850b4e1b3c2d9e3a98fb0b50989e93884b096d402f06ce469"
May 25 12:42:51.948757 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:51.948733    2578 scope.go:117] "RemoveContainer" containerID="1596f0b9f7a7d9f850b4e1b3c2d9e3a98fb0b50989e93884b096d402f06ce469"
May 25 12:42:51.948997 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:42:51.948972    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1596f0b9f7a7d9f850b4e1b3c2d9e3a98fb0b50989e93884b096d402f06ce469\": container with ID starting with 1596f0b9f7a7d9f850b4e1b3c2d9e3a98fb0b50989e93884b096d402f06ce469 not found: ID does not exist" containerID="1596f0b9f7a7d9f850b4e1b3c2d9e3a98fb0b50989e93884b096d402f06ce469"
May 25 12:42:51.949085 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:51.949003    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1596f0b9f7a7d9f850b4e1b3c2d9e3a98fb0b50989e93884b096d402f06ce469"} err="failed to get container status \"1596f0b9f7a7d9f850b4e1b3c2d9e3a98fb0b50989e93884b096d402f06ce469\": rpc error: code = NotFound desc = could not find container \"1596f0b9f7a7d9f850b4e1b3c2d9e3a98fb0b50989e93884b096d402f06ce469\": container with ID starting with 1596f0b9f7a7d9f850b4e1b3c2d9e3a98fb0b50989e93884b096d402f06ce469 not found: ID does not exist"
May 25 12:42:51.963574 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:51.963551    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff"]
May 25 12:42:51.967740 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:51.967718    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-155ee-7766bf4486-wp9ff"]
May 25 12:42:52.939209 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:52.939178    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad" path="/var/lib/kubelet/pods/b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad/volumes"
May 25 12:42:53.640624 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:53.640588    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l" podUID="cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7" containerName="sequence-graph-4d75e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:42:58.641586 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:58.641545    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l" podUID="cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7" containerName="sequence-graph-4d75e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:42:58.985149 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:58.985048    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/ovn-acl-logging/0.log"
May 25 12:42:58.988421 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:42:58.988397    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/ovn-acl-logging/0.log"
May 25 12:43:03.640910 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:03.640871    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l" podUID="cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7" containerName="sequence-graph-4d75e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:43:03.641284 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:03.641011    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l"
May 25 12:43:08.640294 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:08.640254    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l" podUID="cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7" containerName="sequence-graph-4d75e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:43:13.640626 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:13.640587    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l" podUID="cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7" containerName="sequence-graph-4d75e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:43:18.640576 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:18.640532    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l" podUID="cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7" containerName="sequence-graph-4d75e" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:43:21.178546 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:21.178520    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l"
May 25 12:43:21.259675 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:21.259645    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7-openshift-service-ca-bundle\") pod \"cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7\" (UID: \"cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7\") "
May 25 12:43:21.259834 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:21.259735    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7-proxy-tls\") pod \"cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7\" (UID: \"cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7\") "
May 25 12:43:21.260030 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:21.260006    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7" (UID: "cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:43:21.261678 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:21.261643    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7" (UID: "cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:43:21.352962 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:21.352925    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97"]
May 25 12:43:21.353200 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:21.353188    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7" containerName="sequence-graph-4d75e"
May 25 12:43:21.353251 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:21.353202    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7" containerName="sequence-graph-4d75e"
May 25 12:43:21.353251 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:21.353219    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad" containerName="ensemble-graph-155ee"
May 25 12:43:21.353251 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:21.353225    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad" containerName="ensemble-graph-155ee"
May 25 12:43:21.353340 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:21.353264    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7" containerName="sequence-graph-4d75e"
May 25 12:43:21.353340 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:21.353274    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="b18c2fc2-81a9-4ce4-a454-d2d0a831c6ad" containerName="ensemble-graph-155ee"
May 25 12:43:21.356159 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:21.356144    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97"
May 25 12:43:21.358437 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:21.358418    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-fd4b4-serving-cert\""
May 25 12:43:21.358437 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:21.358424    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-fd4b4-kube-rbac-proxy-sar-config\""
May 25 12:43:21.360195 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:21.360177    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7-proxy-tls\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:43:21.360268 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:21.360202    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7-openshift-service-ca-bundle\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:43:21.363787 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:21.363764    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97"]
May 25 12:43:21.461467 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:21.461430    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f71d6c2-9eed-4393-bb82-022261a41e08-proxy-tls\") pod \"splitter-graph-fd4b4-74fd795bfc-rvr97\" (UID: \"7f71d6c2-9eed-4393-bb82-022261a41e08\") " pod="kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97"
May 25 12:43:21.461641 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:21.461497    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f71d6c2-9eed-4393-bb82-022261a41e08-openshift-service-ca-bundle\") pod \"splitter-graph-fd4b4-74fd795bfc-rvr97\" (UID: \"7f71d6c2-9eed-4393-bb82-022261a41e08\") " pod="kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97"
May 25 12:43:21.562414 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:21.562374    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f71d6c2-9eed-4393-bb82-022261a41e08-proxy-tls\") pod \"splitter-graph-fd4b4-74fd795bfc-rvr97\" (UID: \"7f71d6c2-9eed-4393-bb82-022261a41e08\") " pod="kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97"
May 25 12:43:21.562579 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:21.562435    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f71d6c2-9eed-4393-bb82-022261a41e08-openshift-service-ca-bundle\") pod \"splitter-graph-fd4b4-74fd795bfc-rvr97\" (UID: \"7f71d6c2-9eed-4393-bb82-022261a41e08\") " pod="kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97"
May 25 12:43:21.563006 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:21.562987    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f71d6c2-9eed-4393-bb82-022261a41e08-openshift-service-ca-bundle\") pod \"splitter-graph-fd4b4-74fd795bfc-rvr97\" (UID: \"7f71d6c2-9eed-4393-bb82-022261a41e08\") " pod="kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97"
May 25 12:43:21.564739 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:21.564723    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f71d6c2-9eed-4393-bb82-022261a41e08-proxy-tls\") pod \"splitter-graph-fd4b4-74fd795bfc-rvr97\" (UID: \"7f71d6c2-9eed-4393-bb82-022261a41e08\") " pod="kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97"
May 25 12:43:21.668023 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:21.667936    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97"
May 25 12:43:21.796789 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:21.796754    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97"]
May 25 12:43:21.799771 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:43:21.799747    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f71d6c2_9eed_4393_bb82_022261a41e08.slice/crio-f2eccb812fa3f600155a952933534ecb63927d4f0d14d28dd268b25ca3a69f88 WatchSource:0}: Error finding container f2eccb812fa3f600155a952933534ecb63927d4f0d14d28dd268b25ca3a69f88: Status 404 returned error can't find the container with id f2eccb812fa3f600155a952933534ecb63927d4f0d14d28dd268b25ca3a69f88
May 25 12:43:21.801426 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:21.801407    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 12:43:22.020914 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:22.020811    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97" event={"ID":"7f71d6c2-9eed-4393-bb82-022261a41e08","Type":"ContainerStarted","Data":"b047e657c6904bbf00c96a2ced65533b2e5c0e9468cbac45611ef37035351776"}
May 25 12:43:22.020914 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:22.020854    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97" event={"ID":"7f71d6c2-9eed-4393-bb82-022261a41e08","Type":"ContainerStarted","Data":"f2eccb812fa3f600155a952933534ecb63927d4f0d14d28dd268b25ca3a69f88"}
May 25 12:43:22.021187 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:22.020931    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97"
May 25 12:43:22.021884 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:22.021862    2578 generic.go:358] "Generic (PLEG): container finished" podID="cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7" containerID="dfa9230ff0d67a5abd8b3b1ac3d452711f0f358110f99e01c6cbccd6bfed40de" exitCode=0
May 25 12:43:22.021978 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:22.021907    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l"
May 25 12:43:22.021978 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:22.021938    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l" event={"ID":"cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7","Type":"ContainerDied","Data":"dfa9230ff0d67a5abd8b3b1ac3d452711f0f358110f99e01c6cbccd6bfed40de"}
May 25 12:43:22.021978 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:22.021967    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l" event={"ID":"cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7","Type":"ContainerDied","Data":"1f744c105bf8d453740d87f8511be090d7645a2df18723070cc80af828626ea9"}
May 25 12:43:22.022158 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:22.021981    2578 scope.go:117] "RemoveContainer" containerID="dfa9230ff0d67a5abd8b3b1ac3d452711f0f358110f99e01c6cbccd6bfed40de"
May 25 12:43:22.029717 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:22.029696    2578 scope.go:117] "RemoveContainer" containerID="dfa9230ff0d67a5abd8b3b1ac3d452711f0f358110f99e01c6cbccd6bfed40de"
May 25 12:43:22.029972 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:43:22.029953    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfa9230ff0d67a5abd8b3b1ac3d452711f0f358110f99e01c6cbccd6bfed40de\": container with ID starting with dfa9230ff0d67a5abd8b3b1ac3d452711f0f358110f99e01c6cbccd6bfed40de not found: ID does not exist" containerID="dfa9230ff0d67a5abd8b3b1ac3d452711f0f358110f99e01c6cbccd6bfed40de"
May 25 12:43:22.030042 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:22.029980    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfa9230ff0d67a5abd8b3b1ac3d452711f0f358110f99e01c6cbccd6bfed40de"} err="failed to get container status \"dfa9230ff0d67a5abd8b3b1ac3d452711f0f358110f99e01c6cbccd6bfed40de\": rpc error: code = NotFound desc = could not find container \"dfa9230ff0d67a5abd8b3b1ac3d452711f0f358110f99e01c6cbccd6bfed40de\": container with ID starting with dfa9230ff0d67a5abd8b3b1ac3d452711f0f358110f99e01c6cbccd6bfed40de not found: ID does not exist"
May 25 12:43:22.038924 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:22.038886    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97" podStartSLOduration=1.038875187 podStartE2EDuration="1.038875187s" podCreationTimestamp="2026-05-25 12:43:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:43:22.037470883 +0000 UTC m=+1823.678665659" watchObservedRunningTime="2026-05-25 12:43:22.038875187 +0000 UTC m=+1823.680069963"
May 25 12:43:22.052806 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:22.052782    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l"]
May 25 12:43:22.054736 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:22.054715    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-4d75e-7f95b4f9f7-6b88l"]
May 25 12:43:22.938912 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:22.938875    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7" path="/var/lib/kubelet/pods/cce79cc8-362a-41f6-8a05-a6a5bd6b1fc7/volumes"
May 25 12:43:28.030720 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:28.030693    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97"
May 25 12:43:31.412348 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:31.412263    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97"]
May 25 12:43:31.412668 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:31.412554    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97" podUID="7f71d6c2-9eed-4393-bb82-022261a41e08" containerName="splitter-graph-fd4b4" containerID="cri-o://b047e657c6904bbf00c96a2ced65533b2e5c0e9468cbac45611ef37035351776" gracePeriod=30
May 25 12:43:33.033906 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:33.033867    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97" podUID="7f71d6c2-9eed-4393-bb82-022261a41e08" containerName="splitter-graph-fd4b4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:43:38.029276 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:38.029238    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97" podUID="7f71d6c2-9eed-4393-bb82-022261a41e08" containerName="splitter-graph-fd4b4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:43:43.029555 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:43.029514    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97" podUID="7f71d6c2-9eed-4393-bb82-022261a41e08" containerName="splitter-graph-fd4b4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:43:43.029992 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:43.029619    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97"
May 25 12:43:48.029565 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:48.029521    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97" podUID="7f71d6c2-9eed-4393-bb82-022261a41e08" containerName="splitter-graph-fd4b4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:43:53.029358 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:53.029318    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97" podUID="7f71d6c2-9eed-4393-bb82-022261a41e08" containerName="splitter-graph-fd4b4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:43:58.029153 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:43:58.029111    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97" podUID="7f71d6c2-9eed-4393-bb82-022261a41e08" containerName="splitter-graph-fd4b4" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:44:01.262461 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:01.262425    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk"]
May 25 12:44:01.265175 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:01.265159    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk"
May 25 12:44:01.267626 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:01.267605    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-09598-serving-cert\""
May 25 12:44:01.267730 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:01.267607    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-09598-kube-rbac-proxy-sar-config\""
May 25 12:44:01.279049 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:01.279023    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk"]
May 25 12:44:01.351583 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:01.351543    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35d829e9-8c0c-4d4a-a685-18450e9ba74f-proxy-tls\") pod \"switch-graph-09598-5987fb5786-6srjk\" (UID: \"35d829e9-8c0c-4d4a-a685-18450e9ba74f\") " pod="kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk"
May 25 12:44:01.351583 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:01.351591    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35d829e9-8c0c-4d4a-a685-18450e9ba74f-openshift-service-ca-bundle\") pod \"switch-graph-09598-5987fb5786-6srjk\" (UID: \"35d829e9-8c0c-4d4a-a685-18450e9ba74f\") " pod="kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk"
May 25 12:44:01.452882 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:01.452849    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35d829e9-8c0c-4d4a-a685-18450e9ba74f-proxy-tls\") pod \"switch-graph-09598-5987fb5786-6srjk\" (UID: \"35d829e9-8c0c-4d4a-a685-18450e9ba74f\") " pod="kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk"
May 25 12:44:01.453040 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:01.452899    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35d829e9-8c0c-4d4a-a685-18450e9ba74f-openshift-service-ca-bundle\") pod \"switch-graph-09598-5987fb5786-6srjk\" (UID: \"35d829e9-8c0c-4d4a-a685-18450e9ba74f\") " pod="kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk"
May 25 12:44:01.453519 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:01.453501    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35d829e9-8c0c-4d4a-a685-18450e9ba74f-openshift-service-ca-bundle\") pod \"switch-graph-09598-5987fb5786-6srjk\" (UID: \"35d829e9-8c0c-4d4a-a685-18450e9ba74f\") " pod="kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk"
May 25 12:44:01.456206 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:01.456183    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35d829e9-8c0c-4d4a-a685-18450e9ba74f-proxy-tls\") pod \"switch-graph-09598-5987fb5786-6srjk\" (UID: \"35d829e9-8c0c-4d4a-a685-18450e9ba74f\") " pod="kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk"
May 25 12:44:01.558629 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:01.558606    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97"
May 25 12:44:01.577615 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:01.577590    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk"
May 25 12:44:01.654657 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:01.654626    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f71d6c2-9eed-4393-bb82-022261a41e08-proxy-tls\") pod \"7f71d6c2-9eed-4393-bb82-022261a41e08\" (UID: \"7f71d6c2-9eed-4393-bb82-022261a41e08\") "
May 25 12:44:01.654813 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:01.654719    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f71d6c2-9eed-4393-bb82-022261a41e08-openshift-service-ca-bundle\") pod \"7f71d6c2-9eed-4393-bb82-022261a41e08\" (UID: \"7f71d6c2-9eed-4393-bb82-022261a41e08\") "
May 25 12:44:01.655098 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:01.655052    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f71d6c2-9eed-4393-bb82-022261a41e08-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "7f71d6c2-9eed-4393-bb82-022261a41e08" (UID: "7f71d6c2-9eed-4393-bb82-022261a41e08"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:44:01.657032 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:01.656997    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f71d6c2-9eed-4393-bb82-022261a41e08-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7f71d6c2-9eed-4393-bb82-022261a41e08" (UID: "7f71d6c2-9eed-4393-bb82-022261a41e08"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:44:01.703235 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:01.703195    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk"]
May 25 12:44:01.706901 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:44:01.706878    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35d829e9_8c0c_4d4a_a685_18450e9ba74f.slice/crio-987a4e56e3579a6c5b287dd038119bc2d49f2c66ea6dfa3edc4319cc5a54b9e2 WatchSource:0}: Error finding container 987a4e56e3579a6c5b287dd038119bc2d49f2c66ea6dfa3edc4319cc5a54b9e2: Status 404 returned error can't find the container with id 987a4e56e3579a6c5b287dd038119bc2d49f2c66ea6dfa3edc4319cc5a54b9e2
May 25 12:44:01.755957 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:01.755935    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7f71d6c2-9eed-4393-bb82-022261a41e08-proxy-tls\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:44:01.756036 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:01.755960    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f71d6c2-9eed-4393-bb82-022261a41e08-openshift-service-ca-bundle\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:44:02.124974 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:02.124938    2578 generic.go:358] "Generic (PLEG): container finished" podID="7f71d6c2-9eed-4393-bb82-022261a41e08" containerID="b047e657c6904bbf00c96a2ced65533b2e5c0e9468cbac45611ef37035351776" exitCode=0
May 25 12:44:02.125178 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:02.125008    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97"
May 25 12:44:02.125178 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:02.125025    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97" event={"ID":"7f71d6c2-9eed-4393-bb82-022261a41e08","Type":"ContainerDied","Data":"b047e657c6904bbf00c96a2ced65533b2e5c0e9468cbac45611ef37035351776"}
May 25 12:44:02.125178 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:02.125074    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97" event={"ID":"7f71d6c2-9eed-4393-bb82-022261a41e08","Type":"ContainerDied","Data":"f2eccb812fa3f600155a952933534ecb63927d4f0d14d28dd268b25ca3a69f88"}
May 25 12:44:02.125178 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:02.125096    2578 scope.go:117] "RemoveContainer" containerID="b047e657c6904bbf00c96a2ced65533b2e5c0e9468cbac45611ef37035351776"
May 25 12:44:02.126406 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:02.126379    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk" event={"ID":"35d829e9-8c0c-4d4a-a685-18450e9ba74f","Type":"ContainerStarted","Data":"788f6584f3f1a5194ae3c5b8b9277b92cf25a06d445497f485ab1644dde54e18"}
May 25 12:44:02.126518 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:02.126413    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk" event={"ID":"35d829e9-8c0c-4d4a-a685-18450e9ba74f","Type":"ContainerStarted","Data":"987a4e56e3579a6c5b287dd038119bc2d49f2c66ea6dfa3edc4319cc5a54b9e2"}
May 25 12:44:02.126518 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:02.126459    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk"
May 25 12:44:02.133045 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:02.133025    2578 scope.go:117] "RemoveContainer" containerID="b047e657c6904bbf00c96a2ced65533b2e5c0e9468cbac45611ef37035351776"
May 25 12:44:02.133309 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:44:02.133286    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b047e657c6904bbf00c96a2ced65533b2e5c0e9468cbac45611ef37035351776\": container with ID starting with b047e657c6904bbf00c96a2ced65533b2e5c0e9468cbac45611ef37035351776 not found: ID does not exist" containerID="b047e657c6904bbf00c96a2ced65533b2e5c0e9468cbac45611ef37035351776"
May 25 12:44:02.133405 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:02.133319    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b047e657c6904bbf00c96a2ced65533b2e5c0e9468cbac45611ef37035351776"} err="failed to get container status \"b047e657c6904bbf00c96a2ced65533b2e5c0e9468cbac45611ef37035351776\": rpc error: code = NotFound desc = could not find container \"b047e657c6904bbf00c96a2ced65533b2e5c0e9468cbac45611ef37035351776\": container with ID starting with b047e657c6904bbf00c96a2ced65533b2e5c0e9468cbac45611ef37035351776 not found: ID does not exist"
May 25 12:44:02.147667 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:02.147624    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk" podStartSLOduration=1.147612559 podStartE2EDuration="1.147612559s" podCreationTimestamp="2026-05-25 12:44:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:44:02.146962057 +0000 UTC m=+1863.788156838" watchObservedRunningTime="2026-05-25 12:44:02.147612559 +0000 UTC m=+1863.788807336"
May 25 12:44:02.164042 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:02.164014    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97"]
May 25 12:44:02.168087 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:02.168047    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-fd4b4-74fd795bfc-rvr97"]
May 25 12:44:02.940012 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:02.939981    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f71d6c2-9eed-4393-bb82-022261a41e08" path="/var/lib/kubelet/pods/7f71d6c2-9eed-4393-bb82-022261a41e08/volumes"
May 25 12:44:08.135962 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:08.135932    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk"
May 25 12:44:31.630687 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:31.630653    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5"]
May 25 12:44:31.631028 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:31.630915    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f71d6c2-9eed-4393-bb82-022261a41e08" containerName="splitter-graph-fd4b4"
May 25 12:44:31.631028 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:31.630926    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f71d6c2-9eed-4393-bb82-022261a41e08" containerName="splitter-graph-fd4b4"
May 25 12:44:31.631028 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:31.630986    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="7f71d6c2-9eed-4393-bb82-022261a41e08" containerName="splitter-graph-fd4b4"
May 25 12:44:31.633540 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:31.633524    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5"
May 25 12:44:31.636179 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:31.636158    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-8902c-serving-cert\""
May 25 12:44:31.636303 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:31.636241    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-8902c-kube-rbac-proxy-sar-config\""
May 25 12:44:31.643577 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:31.643555    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5"]
May 25 12:44:31.660028 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:31.660002    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dda39130-52e9-4d65-8fc4-731c4b3c7c6c-proxy-tls\") pod \"splitter-graph-8902c-6b89df54cb-65tk5\" (UID: \"dda39130-52e9-4d65-8fc4-731c4b3c7c6c\") " pod="kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5"
May 25 12:44:31.660137 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:31.660032    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dda39130-52e9-4d65-8fc4-731c4b3c7c6c-openshift-service-ca-bundle\") pod \"splitter-graph-8902c-6b89df54cb-65tk5\" (UID: \"dda39130-52e9-4d65-8fc4-731c4b3c7c6c\") " pod="kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5"
May 25 12:44:31.761210 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:31.761183    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dda39130-52e9-4d65-8fc4-731c4b3c7c6c-proxy-tls\") pod \"splitter-graph-8902c-6b89df54cb-65tk5\" (UID: \"dda39130-52e9-4d65-8fc4-731c4b3c7c6c\") " pod="kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5"
May 25 12:44:31.761320 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:31.761216    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dda39130-52e9-4d65-8fc4-731c4b3c7c6c-openshift-service-ca-bundle\") pod \"splitter-graph-8902c-6b89df54cb-65tk5\" (UID: \"dda39130-52e9-4d65-8fc4-731c4b3c7c6c\") " pod="kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5"
May 25 12:44:31.761375 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:44:31.761336    2578 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-8902c-serving-cert: secret "splitter-graph-8902c-serving-cert" not found
May 25 12:44:31.761417 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:44:31.761406    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dda39130-52e9-4d65-8fc4-731c4b3c7c6c-proxy-tls podName:dda39130-52e9-4d65-8fc4-731c4b3c7c6c nodeName:}" failed. No retries permitted until 2026-05-25 12:44:32.261386634 +0000 UTC m=+1893.902581398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dda39130-52e9-4d65-8fc4-731c4b3c7c6c-proxy-tls") pod "splitter-graph-8902c-6b89df54cb-65tk5" (UID: "dda39130-52e9-4d65-8fc4-731c4b3c7c6c") : secret "splitter-graph-8902c-serving-cert" not found
May 25 12:44:31.761761 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:31.761743    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dda39130-52e9-4d65-8fc4-731c4b3c7c6c-openshift-service-ca-bundle\") pod \"splitter-graph-8902c-6b89df54cb-65tk5\" (UID: \"dda39130-52e9-4d65-8fc4-731c4b3c7c6c\") " pod="kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5"
May 25 12:44:32.264655 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:32.264616    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dda39130-52e9-4d65-8fc4-731c4b3c7c6c-proxy-tls\") pod \"splitter-graph-8902c-6b89df54cb-65tk5\" (UID: \"dda39130-52e9-4d65-8fc4-731c4b3c7c6c\") " pod="kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5"
May 25 12:44:32.266930 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:32.266909    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dda39130-52e9-4d65-8fc4-731c4b3c7c6c-proxy-tls\") pod \"splitter-graph-8902c-6b89df54cb-65tk5\" (UID: \"dda39130-52e9-4d65-8fc4-731c4b3c7c6c\") " pod="kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5"
May 25 12:44:32.542680 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:32.542650    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5"
May 25 12:44:32.663147 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:32.663117    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5"]
May 25 12:44:32.666207 ip-10-0-143-107 kubenswrapper[2578]: W0525 12:44:32.666178    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda39130_52e9_4d65_8fc4_731c4b3c7c6c.slice/crio-7a39990ee767323853c6916b35f8f75332cc3ccac99a13e6ad8777d4613e5604 WatchSource:0}: Error finding container 7a39990ee767323853c6916b35f8f75332cc3ccac99a13e6ad8777d4613e5604: Status 404 returned error can't find the container with id 7a39990ee767323853c6916b35f8f75332cc3ccac99a13e6ad8777d4613e5604
May 25 12:44:33.213363 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:33.213325    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5" event={"ID":"dda39130-52e9-4d65-8fc4-731c4b3c7c6c","Type":"ContainerStarted","Data":"7ad79d172c4d37c8c3c28f90aadc92a8b82b768aadc78e97d291a4cf0979d376"}
May 25 12:44:33.213363 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:33.213363    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5" event={"ID":"dda39130-52e9-4d65-8fc4-731c4b3c7c6c","Type":"ContainerStarted","Data":"7a39990ee767323853c6916b35f8f75332cc3ccac99a13e6ad8777d4613e5604"}
May 25 12:44:33.213611 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:33.213451    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5"
May 25 12:44:33.232854 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:33.232814    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5" podStartSLOduration=2.232803997 podStartE2EDuration="2.232803997s" podCreationTimestamp="2026-05-25 12:44:31 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 12:44:33.231665301 +0000 UTC m=+1894.872860077" watchObservedRunningTime="2026-05-25 12:44:33.232803997 +0000 UTC m=+1894.873998772"
May 25 12:44:39.221680 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:44:39.221650    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5"
May 25 12:47:59.002200 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:47:59.002169    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/ovn-acl-logging/0.log"
May 25 12:47:59.005918 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:47:59.005895    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/ovn-acl-logging/0.log"
May 25 12:52:46.420374 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:52:46.420337    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5"]
May 25 12:52:46.420977 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:52:46.420580    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5" podUID="dda39130-52e9-4d65-8fc4-731c4b3c7c6c" containerName="splitter-graph-8902c" containerID="cri-o://7ad79d172c4d37c8c3c28f90aadc92a8b82b768aadc78e97d291a4cf0979d376" gracePeriod=30
May 25 12:52:49.220266 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:52:49.220224    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5" podUID="dda39130-52e9-4d65-8fc4-731c4b3c7c6c" containerName="splitter-graph-8902c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:52:54.220636 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:52:54.220597    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5" podUID="dda39130-52e9-4d65-8fc4-731c4b3c7c6c" containerName="splitter-graph-8902c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:52:59.019266 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:52:59.019159    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/ovn-acl-logging/0.log"
May 25 12:52:59.024613 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:52:59.024596    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/ovn-acl-logging/0.log"
May 25 12:52:59.220945 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:52:59.220907    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5" podUID="dda39130-52e9-4d65-8fc4-731c4b3c7c6c" containerName="splitter-graph-8902c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:52:59.221149 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:52:59.221018    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5"
May 25 12:53:04.220581 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:53:04.220535    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5" podUID="dda39130-52e9-4d65-8fc4-731c4b3c7c6c" containerName="splitter-graph-8902c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:53:09.221117 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:53:09.221077    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5" podUID="dda39130-52e9-4d65-8fc4-731c4b3c7c6c" containerName="splitter-graph-8902c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:53:14.220672 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:53:14.220634    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5" podUID="dda39130-52e9-4d65-8fc4-731c4b3c7c6c" containerName="splitter-graph-8902c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 12:53:16.456126 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:53:16.456094    2578 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda39130_52e9_4d65_8fc4_731c4b3c7c6c.slice/crio-conmon-7ad79d172c4d37c8c3c28f90aadc92a8b82b768aadc78e97d291a4cf0979d376.scope\": RecentStats: unable to find data in memory cache]"
May 25 12:53:16.565186 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:53:16.565163    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5"
May 25 12:53:16.622502 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:53:16.622466    2578 generic.go:358] "Generic (PLEG): container finished" podID="dda39130-52e9-4d65-8fc4-731c4b3c7c6c" containerID="7ad79d172c4d37c8c3c28f90aadc92a8b82b768aadc78e97d291a4cf0979d376" exitCode=0
May 25 12:53:16.622671 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:53:16.622542    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5"
May 25 12:53:16.622671 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:53:16.622542    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5" event={"ID":"dda39130-52e9-4d65-8fc4-731c4b3c7c6c","Type":"ContainerDied","Data":"7ad79d172c4d37c8c3c28f90aadc92a8b82b768aadc78e97d291a4cf0979d376"}
May 25 12:53:16.622671 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:53:16.622583    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5" event={"ID":"dda39130-52e9-4d65-8fc4-731c4b3c7c6c","Type":"ContainerDied","Data":"7a39990ee767323853c6916b35f8f75332cc3ccac99a13e6ad8777d4613e5604"}
May 25 12:53:16.622671 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:53:16.622604    2578 scope.go:117] "RemoveContainer" containerID="7ad79d172c4d37c8c3c28f90aadc92a8b82b768aadc78e97d291a4cf0979d376"
May 25 12:53:16.631144 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:53:16.631121    2578 scope.go:117] "RemoveContainer" containerID="7ad79d172c4d37c8c3c28f90aadc92a8b82b768aadc78e97d291a4cf0979d376"
May 25 12:53:16.631399 ip-10-0-143-107 kubenswrapper[2578]: E0525 12:53:16.631378    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ad79d172c4d37c8c3c28f90aadc92a8b82b768aadc78e97d291a4cf0979d376\": container with ID starting with 7ad79d172c4d37c8c3c28f90aadc92a8b82b768aadc78e97d291a4cf0979d376 not found: ID does not exist" containerID="7ad79d172c4d37c8c3c28f90aadc92a8b82b768aadc78e97d291a4cf0979d376"
May 25 12:53:16.631481 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:53:16.631407    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ad79d172c4d37c8c3c28f90aadc92a8b82b768aadc78e97d291a4cf0979d376"} err="failed to get container status \"7ad79d172c4d37c8c3c28f90aadc92a8b82b768aadc78e97d291a4cf0979d376\": rpc error: code = NotFound desc = could not find container \"7ad79d172c4d37c8c3c28f90aadc92a8b82b768aadc78e97d291a4cf0979d376\": container with ID starting with 7ad79d172c4d37c8c3c28f90aadc92a8b82b768aadc78e97d291a4cf0979d376 not found: ID does not exist"
May 25 12:53:16.694379 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:53:16.694282    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dda39130-52e9-4d65-8fc4-731c4b3c7c6c-openshift-service-ca-bundle\") pod \"dda39130-52e9-4d65-8fc4-731c4b3c7c6c\" (UID: \"dda39130-52e9-4d65-8fc4-731c4b3c7c6c\") "
May 25 12:53:16.694379 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:53:16.694369    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dda39130-52e9-4d65-8fc4-731c4b3c7c6c-proxy-tls\") pod \"dda39130-52e9-4d65-8fc4-731c4b3c7c6c\" (UID: \"dda39130-52e9-4d65-8fc4-731c4b3c7c6c\") "
May 25 12:53:16.694654 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:53:16.694630    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda39130-52e9-4d65-8fc4-731c4b3c7c6c-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "dda39130-52e9-4d65-8fc4-731c4b3c7c6c" (UID: "dda39130-52e9-4d65-8fc4-731c4b3c7c6c"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 12:53:16.696540 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:53:16.696514    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dda39130-52e9-4d65-8fc4-731c4b3c7c6c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dda39130-52e9-4d65-8fc4-731c4b3c7c6c" (UID: "dda39130-52e9-4d65-8fc4-731c4b3c7c6c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 12:53:16.795705 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:53:16.795671    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dda39130-52e9-4d65-8fc4-731c4b3c7c6c-openshift-service-ca-bundle\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:53:16.795705 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:53:16.795699    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dda39130-52e9-4d65-8fc4-731c4b3c7c6c-proxy-tls\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 12:53:16.943778 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:53:16.943753    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5"]
May 25 12:53:16.948608 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:53:16.948547    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-8902c-6b89df54cb-65tk5"]
May 25 12:53:18.939681 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:53:18.939650    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dda39130-52e9-4d65-8fc4-731c4b3c7c6c" path="/var/lib/kubelet/pods/dda39130-52e9-4d65-8fc4-731c4b3c7c6c/volumes"
May 25 12:57:59.036590 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:57:59.036481    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/ovn-acl-logging/0.log"
May 25 12:57:59.042043 ip-10-0-143-107 kubenswrapper[2578]: I0525 12:57:59.042024    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/ovn-acl-logging/0.log"
May 25 13:00:20.690313 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:20.690280    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk"]
May 25 13:00:20.690846 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:20.690520    2578 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk" podUID="35d829e9-8c0c-4d4a-a685-18450e9ba74f" containerName="switch-graph-09598" containerID="cri-o://788f6584f3f1a5194ae3c5b8b9277b92cf25a06d445497f485ab1644dde54e18" gracePeriod=30
May 25 13:00:23.134109 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:23.134049    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk" podUID="35d829e9-8c0c-4d4a-a685-18450e9ba74f" containerName="switch-graph-09598" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:00:28.134149 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:28.134101    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk" podUID="35d829e9-8c0c-4d4a-a685-18450e9ba74f" containerName="switch-graph-09598" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:00:33.133932 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:33.133887    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk" podUID="35d829e9-8c0c-4d4a-a685-18450e9ba74f" containerName="switch-graph-09598" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:00:33.134337 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:33.133996    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk"
May 25 13:00:37.119070 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:37.119038    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-09598-5987fb5786-6srjk_35d829e9-8c0c-4d4a-a685-18450e9ba74f/switch-graph-09598/0.log"
May 25 13:00:37.966497 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:37.966461    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-09598-5987fb5786-6srjk_35d829e9-8c0c-4d4a-a685-18450e9ba74f/switch-graph-09598/0.log"
May 25 13:00:38.134293 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:38.134255    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk" podUID="35d829e9-8c0c-4d4a-a685-18450e9ba74f" containerName="switch-graph-09598" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:00:38.786617 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:38.786584    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-09598-5987fb5786-6srjk_35d829e9-8c0c-4d4a-a685-18450e9ba74f/switch-graph-09598/0.log"
May 25 13:00:39.591998 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:39.591971    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-09598-5987fb5786-6srjk_35d829e9-8c0c-4d4a-a685-18450e9ba74f/switch-graph-09598/0.log"
May 25 13:00:40.448955 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:40.448928    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-09598-5987fb5786-6srjk_35d829e9-8c0c-4d4a-a685-18450e9ba74f/switch-graph-09598/0.log"
May 25 13:00:41.247267 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:41.247243    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-09598-5987fb5786-6srjk_35d829e9-8c0c-4d4a-a685-18450e9ba74f/switch-graph-09598/0.log"
May 25 13:00:42.057931 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:42.057903    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-09598-5987fb5786-6srjk_35d829e9-8c0c-4d4a-a685-18450e9ba74f/switch-graph-09598/0.log"
May 25 13:00:42.894012 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:42.893980    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-09598-5987fb5786-6srjk_35d829e9-8c0c-4d4a-a685-18450e9ba74f/switch-graph-09598/0.log"
May 25 13:00:43.133732 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:43.133695    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk" podUID="35d829e9-8c0c-4d4a-a685-18450e9ba74f" containerName="switch-graph-09598" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:00:43.694926 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:43.694898    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-09598-5987fb5786-6srjk_35d829e9-8c0c-4d4a-a685-18450e9ba74f/switch-graph-09598/0.log"
May 25 13:00:44.524771 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:44.524740    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-09598-5987fb5786-6srjk_35d829e9-8c0c-4d4a-a685-18450e9ba74f/switch-graph-09598/0.log"
May 25 13:00:45.347051 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:45.346978    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-09598-5987fb5786-6srjk_35d829e9-8c0c-4d4a-a685-18450e9ba74f/switch-graph-09598/0.log"
May 25 13:00:46.173212 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:46.173178    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-09598-5987fb5786-6srjk_35d829e9-8c0c-4d4a-a685-18450e9ba74f/switch-graph-09598/0.log"
May 25 13:00:48.134395 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:48.134356    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk" podUID="35d829e9-8c0c-4d4a-a685-18450e9ba74f" containerName="switch-graph-09598" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 13:00:50.807931 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:50.807896    2578 generic.go:358] "Generic (PLEG): container finished" podID="35d829e9-8c0c-4d4a-a685-18450e9ba74f" containerID="788f6584f3f1a5194ae3c5b8b9277b92cf25a06d445497f485ab1644dde54e18" exitCode=0
May 25 13:00:50.808300 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:50.807964    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk" event={"ID":"35d829e9-8c0c-4d4a-a685-18450e9ba74f","Type":"ContainerDied","Data":"788f6584f3f1a5194ae3c5b8b9277b92cf25a06d445497f485ab1644dde54e18"}
May 25 13:00:50.829384 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:50.829361    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk"
May 25 13:00:50.894676 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:50.894637    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35d829e9-8c0c-4d4a-a685-18450e9ba74f-openshift-service-ca-bundle\") pod \"35d829e9-8c0c-4d4a-a685-18450e9ba74f\" (UID: \"35d829e9-8c0c-4d4a-a685-18450e9ba74f\") "
May 25 13:00:50.894862 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:50.894722    2578 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35d829e9-8c0c-4d4a-a685-18450e9ba74f-proxy-tls\") pod \"35d829e9-8c0c-4d4a-a685-18450e9ba74f\" (UID: \"35d829e9-8c0c-4d4a-a685-18450e9ba74f\") "
May 25 13:00:50.894990 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:50.894967    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35d829e9-8c0c-4d4a-a685-18450e9ba74f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "35d829e9-8c0c-4d4a-a685-18450e9ba74f" (UID: "35d829e9-8c0c-4d4a-a685-18450e9ba74f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 13:00:50.896718 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:50.896694    2578 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35d829e9-8c0c-4d4a-a685-18450e9ba74f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "35d829e9-8c0c-4d4a-a685-18450e9ba74f" (UID: "35d829e9-8c0c-4d4a-a685-18450e9ba74f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 13:00:50.995942 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:50.995900    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35d829e9-8c0c-4d4a-a685-18450e9ba74f-openshift-service-ca-bundle\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 13:00:50.995942 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:50.995932    2578 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/35d829e9-8c0c-4d4a-a685-18450e9ba74f-proxy-tls\") on node \"ip-10-0-143-107.ec2.internal\" DevicePath \"\""
May 25 13:00:51.811264 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:51.811230    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk" event={"ID":"35d829e9-8c0c-4d4a-a685-18450e9ba74f","Type":"ContainerDied","Data":"987a4e56e3579a6c5b287dd038119bc2d49f2c66ea6dfa3edc4319cc5a54b9e2"}
May 25 13:00:51.811701 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:51.811277    2578 scope.go:117] "RemoveContainer" containerID="788f6584f3f1a5194ae3c5b8b9277b92cf25a06d445497f485ab1644dde54e18"
May 25 13:00:51.811701 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:51.811277    2578 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk"
May 25 13:00:51.830623 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:51.830595    2578 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk"]
May 25 13:00:51.835454 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:51.835425    2578 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-09598-5987fb5786-6srjk"]
May 25 13:00:52.939402 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:52.939367    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35d829e9-8c0c-4d4a-a685-18450e9ba74f" path="/var/lib/kubelet/pods/35d829e9-8c0c-4d4a-a685-18450e9ba74f/volumes"
May 25 13:00:54.904329 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:54.904292    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8zd5f_131acddf-521f-481d-84a3-b4e9ce65b58e/global-pull-secret-syncer/0.log"
May 25 13:00:55.035188 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:55.035157    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2fqnj_01b74961-b200-4de0-9d76-e43923f19633/konnectivity-agent/0.log"
May 25 13:00:55.180646 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:55.180571    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-107.ec2.internal_a3617d771b965ababd68b956e9ea4412/haproxy/0.log"
May 25 13:00:58.890628 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:58.890596    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ssq8w_bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6/node-exporter/0.log"
May 25 13:00:58.912403 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:58.912374    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ssq8w_bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6/kube-rbac-proxy/0.log"
May 25 13:00:58.937218 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:00:58.937190    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ssq8w_bd0f4aaa-5a16-4c9e-8f43-8bc70809cdd6/init-textfile/0.log"
May 25 13:01:02.035992 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.035954    2578 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9vddf/perf-node-gather-daemonset-vwlgs"]
May 25 13:01:02.036379 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.036213    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35d829e9-8c0c-4d4a-a685-18450e9ba74f" containerName="switch-graph-09598"
May 25 13:01:02.036379 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.036225    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="35d829e9-8c0c-4d4a-a685-18450e9ba74f" containerName="switch-graph-09598"
May 25 13:01:02.036379 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.036243    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dda39130-52e9-4d65-8fc4-731c4b3c7c6c" containerName="splitter-graph-8902c"
May 25 13:01:02.036379 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.036250    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dda39130-52e9-4d65-8fc4-731c4b3c7c6c" containerName="splitter-graph-8902c"
May 25 13:01:02.036379 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.036293    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="35d829e9-8c0c-4d4a-a685-18450e9ba74f" containerName="switch-graph-09598"
May 25 13:01:02.036379 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.036302    2578 memory_manager.go:356] "RemoveStaleState removing state" podUID="dda39130-52e9-4d65-8fc4-731c4b3c7c6c" containerName="splitter-graph-8902c"
May 25 13:01:02.040263 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.040242    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-vwlgs"
May 25 13:01:02.042694 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.042676    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9vddf\"/\"openshift-service-ca.crt\""
May 25 13:01:02.042809 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.042730    2578 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9vddf\"/\"kube-root-ca.crt\""
May 25 13:01:02.043649 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.043633    2578 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9vddf\"/\"default-dockercfg-cdbvg\""
May 25 13:01:02.050658 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.050632    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9vddf/perf-node-gather-daemonset-vwlgs"]
May 25 13:01:02.185815 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.185755    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b7c03d8d-c87b-42ee-9592-1cb4629d631d-proc\") pod \"perf-node-gather-daemonset-vwlgs\" (UID: \"b7c03d8d-c87b-42ee-9592-1cb4629d631d\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-vwlgs"
May 25 13:01:02.185994 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.185843    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb7gf\" (UniqueName: \"kubernetes.io/projected/b7c03d8d-c87b-42ee-9592-1cb4629d631d-kube-api-access-lb7gf\") pod \"perf-node-gather-daemonset-vwlgs\" (UID: \"b7c03d8d-c87b-42ee-9592-1cb4629d631d\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-vwlgs"
May 25 13:01:02.185994 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.185874    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b7c03d8d-c87b-42ee-9592-1cb4629d631d-podres\") pod \"perf-node-gather-daemonset-vwlgs\" (UID: \"b7c03d8d-c87b-42ee-9592-1cb4629d631d\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-vwlgs"
May 25 13:01:02.185994 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.185889    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7c03d8d-c87b-42ee-9592-1cb4629d631d-lib-modules\") pod \"perf-node-gather-daemonset-vwlgs\" (UID: \"b7c03d8d-c87b-42ee-9592-1cb4629d631d\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-vwlgs"
May 25 13:01:02.185994 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.185908    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7c03d8d-c87b-42ee-9592-1cb4629d631d-sys\") pod \"perf-node-gather-daemonset-vwlgs\" (UID: \"b7c03d8d-c87b-42ee-9592-1cb4629d631d\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-vwlgs"
May 25 13:01:02.286754 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.286665    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b7c03d8d-c87b-42ee-9592-1cb4629d631d-proc\") pod \"perf-node-gather-daemonset-vwlgs\" (UID: \"b7c03d8d-c87b-42ee-9592-1cb4629d631d\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-vwlgs"
May 25 13:01:02.286754 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.286733    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lb7gf\" (UniqueName: \"kubernetes.io/projected/b7c03d8d-c87b-42ee-9592-1cb4629d631d-kube-api-access-lb7gf\") pod \"perf-node-gather-daemonset-vwlgs\" (UID: \"b7c03d8d-c87b-42ee-9592-1cb4629d631d\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-vwlgs"
May 25 13:01:02.286922 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.286769    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b7c03d8d-c87b-42ee-9592-1cb4629d631d-podres\") pod \"perf-node-gather-daemonset-vwlgs\" (UID: \"b7c03d8d-c87b-42ee-9592-1cb4629d631d\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-vwlgs"
May 25 13:01:02.286922 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.286786    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7c03d8d-c87b-42ee-9592-1cb4629d631d-lib-modules\") pod \"perf-node-gather-daemonset-vwlgs\" (UID: \"b7c03d8d-c87b-42ee-9592-1cb4629d631d\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-vwlgs"
May 25 13:01:02.286922 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.286796    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b7c03d8d-c87b-42ee-9592-1cb4629d631d-proc\") pod \"perf-node-gather-daemonset-vwlgs\" (UID: \"b7c03d8d-c87b-42ee-9592-1cb4629d631d\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-vwlgs"
May 25 13:01:02.286922 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.286805    2578 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7c03d8d-c87b-42ee-9592-1cb4629d631d-sys\") pod \"perf-node-gather-daemonset-vwlgs\" (UID: \"b7c03d8d-c87b-42ee-9592-1cb4629d631d\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-vwlgs"
May 25 13:01:02.286922 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.286848    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b7c03d8d-c87b-42ee-9592-1cb4629d631d-sys\") pod \"perf-node-gather-daemonset-vwlgs\" (UID: \"b7c03d8d-c87b-42ee-9592-1cb4629d631d\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-vwlgs"
May 25 13:01:02.287111 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.286938    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b7c03d8d-c87b-42ee-9592-1cb4629d631d-podres\") pod \"perf-node-gather-daemonset-vwlgs\" (UID: \"b7c03d8d-c87b-42ee-9592-1cb4629d631d\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-vwlgs"
May 25 13:01:02.287111 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.286990    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b7c03d8d-c87b-42ee-9592-1cb4629d631d-lib-modules\") pod \"perf-node-gather-daemonset-vwlgs\" (UID: \"b7c03d8d-c87b-42ee-9592-1cb4629d631d\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-vwlgs"
May 25 13:01:02.296279 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.296260    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb7gf\" (UniqueName: \"kubernetes.io/projected/b7c03d8d-c87b-42ee-9592-1cb4629d631d-kube-api-access-lb7gf\") pod \"perf-node-gather-daemonset-vwlgs\" (UID: \"b7c03d8d-c87b-42ee-9592-1cb4629d631d\") " pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-vwlgs"
May 25 13:01:02.350114 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.350050    2578 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-vwlgs"
May 25 13:01:02.473915 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.473877    2578 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9vddf/perf-node-gather-daemonset-vwlgs"]
May 25 13:01:02.476726 ip-10-0-143-107 kubenswrapper[2578]: W0525 13:01:02.476696    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb7c03d8d_c87b_42ee_9592_1cb4629d631d.slice/crio-97d30d51e0c17ade3a915c2e3f58d4420a4da1bf2f67518c28f44ca8436a992c WatchSource:0}: Error finding container 97d30d51e0c17ade3a915c2e3f58d4420a4da1bf2f67518c28f44ca8436a992c: Status 404 returned error can't find the container with id 97d30d51e0c17ade3a915c2e3f58d4420a4da1bf2f67518c28f44ca8436a992c
May 25 13:01:02.478281 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.478264    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 13:01:02.841504 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.841416    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-vwlgs" event={"ID":"b7c03d8d-c87b-42ee-9592-1cb4629d631d","Type":"ContainerStarted","Data":"6b2305bea6477dd626afd164ecd42479b828c45214cea660dcc39626ea60a022"}
May 25 13:01:02.841504 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.841450    2578 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-vwlgs" event={"ID":"b7c03d8d-c87b-42ee-9592-1cb4629d631d","Type":"ContainerStarted","Data":"97d30d51e0c17ade3a915c2e3f58d4420a4da1bf2f67518c28f44ca8436a992c"}
May 25 13:01:02.841504 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.841474    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-vwlgs"
May 25 13:01:02.859280 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.859226    2578 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-vwlgs" podStartSLOduration=0.859212077 podStartE2EDuration="859.212077ms" podCreationTimestamp="2026-05-25 13:01:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 13:01:02.857615402 +0000 UTC m=+2884.498810188" watchObservedRunningTime="2026-05-25 13:01:02.859212077 +0000 UTC m=+2884.500406853"
May 25 13:01:02.882307 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.882281    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xk9w8_6bb75886-2392-4014-b64b-e61e092ffbd9/dns/0.log"
May 25 13:01:02.903174 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.903148    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xk9w8_6bb75886-2392-4014-b64b-e61e092ffbd9/kube-rbac-proxy/0.log"
May 25 13:01:02.976895 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:02.976868    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sg52n_02afa42e-73a7-4f9b-a5d7-609a53a507b1/dns-node-resolver/0.log"
May 25 13:01:03.499095 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:03.499051    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-j2hhk_a34e5474-3229-4811-8b0e-dee727e7d8b7/node-ca/0.log"
May 25 13:01:04.670014 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:04.669986    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mzmd6_587db4ef-f6a5-48c3-9e78-957d815002a4/serve-healthcheck-canary/0.log"
May 25 13:01:05.149781 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:05.149749    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-49t8t_0a07290b-c095-4e6e-b86f-6d9abbbd9ce0/kube-rbac-proxy/0.log"
May 25 13:01:05.172784 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:05.172757    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-49t8t_0a07290b-c095-4e6e-b86f-6d9abbbd9ce0/exporter/0.log"
May 25 13:01:05.195190 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:05.195166    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-49t8t_0a07290b-c095-4e6e-b86f-6d9abbbd9ce0/extractor/0.log"
May 25 13:01:07.244992 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:07.244963    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-6b94ff949c-xb6cq_210b07bc-b316-4131-98c1-1b36b5e45d4e/manager/0.log"
May 25 13:01:07.506894 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:07.506812    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-696fc77849-8l6v7_611dfffc-5590-4116-a890-8da7c60c9f0f/manager/0.log"
May 25 13:01:08.854254 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:08.854225    2578 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9vddf/perf-node-gather-daemonset-vwlgs"
May 25 13:01:13.061582 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:13.061556    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5xg5h_f97986ac-3ba7-4762-9e76-38ea2ac839b6/kube-multus-additional-cni-plugins/0.log"
May 25 13:01:13.085818 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:13.085786    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5xg5h_f97986ac-3ba7-4762-9e76-38ea2ac839b6/egress-router-binary-copy/0.log"
May 25 13:01:13.108732 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:13.108697    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5xg5h_f97986ac-3ba7-4762-9e76-38ea2ac839b6/cni-plugins/0.log"
May 25 13:01:13.135710 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:13.135686    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5xg5h_f97986ac-3ba7-4762-9e76-38ea2ac839b6/bond-cni-plugin/0.log"
May 25 13:01:13.159786 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:13.159762    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5xg5h_f97986ac-3ba7-4762-9e76-38ea2ac839b6/routeoverride-cni/0.log"
May 25 13:01:13.183218 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:13.183194    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5xg5h_f97986ac-3ba7-4762-9e76-38ea2ac839b6/whereabouts-cni-bincopy/0.log"
May 25 13:01:13.206140 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:13.206118    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5xg5h_f97986ac-3ba7-4762-9e76-38ea2ac839b6/whereabouts-cni/0.log"
May 25 13:01:13.612487 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:13.612458    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zwd5q_4fe8af28-e32b-4375-9cca-803603210b1a/kube-multus/0.log"
May 25 13:01:13.632993 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:13.632968    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-n9vrb_02a94135-b85a-4ab1-9dd1-2ae75e77f2de/network-metrics-daemon/0.log"
May 25 13:01:13.654710 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:13.654646    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-n9vrb_02a94135-b85a-4ab1-9dd1-2ae75e77f2de/kube-rbac-proxy/0.log"
May 25 13:01:14.896336 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:14.896307    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/ovn-controller/0.log"
May 25 13:01:14.917397 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:14.917376    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/ovn-acl-logging/0.log"
May 25 13:01:14.929225 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:14.929203    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/ovn-acl-logging/1.log"
May 25 13:01:14.951096 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:14.951048    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/kube-rbac-proxy-node/0.log"
May 25 13:01:14.974262 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:14.974238    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/kube-rbac-proxy-ovn-metrics/0.log"
May 25 13:01:14.996010 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:14.995992    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/northd/0.log"
May 25 13:01:15.017944 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:15.017925    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/nbdb/0.log"
May 25 13:01:15.041299 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:15.041282    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/sbdb/0.log"
May 25 13:01:15.140474 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:15.140447    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-qt8dc_7687fe26-96a3-43ea-9223-0c5d61cd667e/ovnkube-controller/0.log"
May 25 13:01:16.533149 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:16.533115    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-zhvk8_cae409cd-445f-40b9-9851-c89b09c1ac43/network-check-target-container/0.log"
May 25 13:01:17.521922 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:17.521896    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-qkfd7_6ddd277f-8d2e-4f5a-b2f9-788afa7f7e33/iptables-alerter/0.log"
May 25 13:01:18.178613 ip-10-0-143-107 kubenswrapper[2578]: I0525 13:01:18.178583    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-mpgf7_69c3d413-5409-4322-8e78-1cfa1d96ed00/tuned/0.log"