May 29 13:39:32.640363 ip-10-0-141-1 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 29 13:39:32.640375 ip-10-0-141-1 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 29 13:39:32.640385 ip-10-0-141-1 systemd[1]: kubelet.service: Failed with result 'resources'.
May 29 13:39:32.640709 ip-10-0-141-1 systemd[1]: Failed to start Kubernetes Kubelet.
May 29 13:39:42.764239 ip-10-0-141-1 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 29 13:39:42.764254 ip-10-0-141-1 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot d171592ae8154b469095160918e5c9b9 --
May 29 13:42:05.400347 ip-10-0-141-1 systemd[1]: Starting Kubernetes Kubelet...
May 29 13:42:05.919682 ip-10-0-141-1 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 29 13:42:05.919682 ip-10-0-141-1 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 29 13:42:05.919682 ip-10-0-141-1 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 29 13:42:05.919682 ip-10-0-141-1 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 29 13:42:05.919682 ip-10-0-141-1 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 29 13:42:05.921628 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.921535    2571 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 29 13:42:05.925604 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925586    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 29 13:42:05.925604 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925605    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 29 13:42:05.925681 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925609    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 29 13:42:05.925681 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925613    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 29 13:42:05.925681 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925616    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 29 13:42:05.925681 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925619    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 29 13:42:05.925681 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925622    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 29 13:42:05.925681 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925625    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 29 13:42:05.925681 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925628    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 29 13:42:05.925681 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925631    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 29 13:42:05.925681 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925635    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 29 13:42:05.925681 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925638    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 29 13:42:05.925681 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925641    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
May 29 13:42:05.925681 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925645    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 29 13:42:05.925681 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925647    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 29 13:42:05.925681 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925650    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 29 13:42:05.925681 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925653    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 29 13:42:05.925681 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925655    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 29 13:42:05.925681 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925658    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 29 13:42:05.925681 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925661    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 29 13:42:05.925681 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925663    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 29 13:42:05.925681 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925666    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 29 13:42:05.926180 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925669    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 29 13:42:05.926180 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925671    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 29 13:42:05.926180 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925674    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 29 13:42:05.926180 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925677    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 29 13:42:05.926180 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925679    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 29 13:42:05.926180 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925682    2571 feature_gate.go:328] unrecognized feature gate: Example
May 29 13:42:05.926180 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925685    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 29 13:42:05.926180 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925688    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 29 13:42:05.926180 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925691    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 29 13:42:05.926180 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925694    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 29 13:42:05.926180 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925697    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 29 13:42:05.926180 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925699    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 29 13:42:05.926180 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925702    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 29 13:42:05.926180 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925704    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 29 13:42:05.926180 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925707    2571 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 29 13:42:05.926180 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925710    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 29 13:42:05.926180 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925712    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 29 13:42:05.926180 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925716    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
May 29 13:42:05.926180 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925718    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 29 13:42:05.926180 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925721    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 29 13:42:05.926656 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925723    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 29 13:42:05.926656 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925726    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 29 13:42:05.926656 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925728    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 29 13:42:05.926656 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925731    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 29 13:42:05.926656 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925733    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 29 13:42:05.926656 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925736    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 29 13:42:05.926656 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925738    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 29 13:42:05.926656 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925740    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 29 13:42:05.926656 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925743    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 29 13:42:05.926656 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925745    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 29 13:42:05.926656 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925750    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 29 13:42:05.926656 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925754    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 29 13:42:05.926656 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925757    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 29 13:42:05.926656 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925760    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 29 13:42:05.926656 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925763    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 29 13:42:05.926656 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925766    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 29 13:42:05.926656 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925769    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 29 13:42:05.926656 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925771    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 29 13:42:05.926656 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925774    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 29 13:42:05.926656 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925776    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 29 13:42:05.927168 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925779    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 29 13:42:05.927168 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925781    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 29 13:42:05.927168 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925784    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 29 13:42:05.927168 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925786    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 29 13:42:05.927168 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925789    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 29 13:42:05.927168 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925791    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 29 13:42:05.927168 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925796    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 29 13:42:05.927168 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925799    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 29 13:42:05.927168 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925802    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 29 13:42:05.927168 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925805    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 29 13:42:05.927168 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925808    2571 feature_gate.go:328] unrecognized feature gate: Example2
May 29 13:42:05.927168 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925811    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 29 13:42:05.927168 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925814    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 29 13:42:05.927168 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925816    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 29 13:42:05.927168 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925818    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 29 13:42:05.927168 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925821    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 29 13:42:05.927168 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925823    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 29 13:42:05.927168 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925826    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 29 13:42:05.927168 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925830    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 29 13:42:05.927623 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925833    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 29 13:42:05.927623 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925835    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 29 13:42:05.927623 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925838    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 29 13:42:05.927623 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925840    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 29 13:42:05.927623 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.925843    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 29 13:42:05.927623 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926268    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 29 13:42:05.927623 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926275    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 29 13:42:05.927623 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926278    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 29 13:42:05.927623 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926280    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 29 13:42:05.927623 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926283    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 29 13:42:05.927623 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926286    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 29 13:42:05.927623 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926288    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 29 13:42:05.927623 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926291    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 29 13:42:05.927623 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926293    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 29 13:42:05.927623 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926296    2571 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 29 13:42:05.927623 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926299    2571 feature_gate.go:328] unrecognized feature gate: Example
May 29 13:42:05.927623 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926301    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 29 13:42:05.927623 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926305    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 29 13:42:05.927623 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926309    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 29 13:42:05.928085 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926311    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 29 13:42:05.928085 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926314    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 29 13:42:05.928085 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926316    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 29 13:42:05.928085 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926319    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 29 13:42:05.928085 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926322    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 29 13:42:05.928085 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926325    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 29 13:42:05.928085 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926327    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 29 13:42:05.928085 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926330    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 29 13:42:05.928085 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926332    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 29 13:42:05.928085 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926334    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 29 13:42:05.928085 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926337    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 29 13:42:05.928085 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926339    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 29 13:42:05.928085 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926342    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 29 13:42:05.928085 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926344    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 29 13:42:05.928085 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926347    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 29 13:42:05.928085 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926350    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 29 13:42:05.928085 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926352    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 29 13:42:05.928085 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926355    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 29 13:42:05.928085 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926357    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
May 29 13:42:05.928085 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926361    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 29 13:42:05.928590 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926365    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 29 13:42:05.928590 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926369    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 29 13:42:05.928590 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926372    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 29 13:42:05.928590 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926375    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 29 13:42:05.928590 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926378    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 29 13:42:05.928590 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926380    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 29 13:42:05.928590 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926383    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 29 13:42:05.928590 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926385    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 29 13:42:05.928590 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926388    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 29 13:42:05.928590 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926391    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 29 13:42:05.928590 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926393    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 29 13:42:05.928590 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926396    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 29 13:42:05.928590 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926398    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 29 13:42:05.928590 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926400    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 29 13:42:05.928590 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926403    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 29 13:42:05.928590 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926405    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 29 13:42:05.928590 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926408    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 29 13:42:05.928590 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926411    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 29 13:42:05.928590 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926413    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 29 13:42:05.929064 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926416    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 29 13:42:05.929064 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926418    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 29 13:42:05.929064 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926421    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 29 13:42:05.929064 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926423    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 29 13:42:05.929064 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926427    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 29 13:42:05.929064 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926430    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 29 13:42:05.929064 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926432    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 29 13:42:05.929064 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926435    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 29 13:42:05.929064 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926438    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 29 13:42:05.929064 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926440    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 29 13:42:05.929064 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926443    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 29 13:42:05.929064 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926445    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 29 13:42:05.929064 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926448    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 29 13:42:05.929064 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926451    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 29 13:42:05.929064 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926453    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 29 13:42:05.929064 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926456    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 29 13:42:05.929064 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926458    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 29 13:42:05.929064 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926461    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 29 13:42:05.929064 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926463    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 29 13:42:05.929064 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926466    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 29 13:42:05.929584 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926468    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 29 13:42:05.929584 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926470    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 29 13:42:05.929584 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926473    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 29 13:42:05.929584 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926476    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 29 13:42:05.929584 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926479    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 29 13:42:05.929584 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926481    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 29 13:42:05.929584 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926483    2571 feature_gate.go:328] unrecognized feature gate: Example2
May 29 13:42:05.929584 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926486    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
May 29 13:42:05.929584 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926488    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 29 13:42:05.929584 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926491    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 29 13:42:05.929584 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926493    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 29 13:42:05.929584 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926496    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 29 13:42:05.929584 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.926499    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 29 13:42:05.929584 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926577    2571 flags.go:64] FLAG: --address="0.0.0.0"
May 29 13:42:05.929584 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926588    2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 29 13:42:05.929584 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926612    2571 flags.go:64] FLAG: --anonymous-auth="true"
May 29 13:42:05.929584 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926618    2571 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 29 13:42:05.929584 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926622    2571 flags.go:64] FLAG: --authentication-token-webhook="false"
May 29 13:42:05.929584 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926626    2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 29 13:42:05.929584 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926630    2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 29 13:42:05.929584 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926635    2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 29 13:42:05.929584 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926639    2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 29 13:42:05.930169 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926642    2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 29 13:42:05.930169 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926646    2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 29 13:42:05.930169 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926651    2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 29 13:42:05.930169 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926654    2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 29 13:42:05.930169 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926657    2571 flags.go:64] FLAG: --cgroup-root=""
May 29 13:42:05.930169 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926661    2571 flags.go:64] FLAG: --cgroups-per-qos="true"
May 29 13:42:05.930169 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926664    2571 flags.go:64] FLAG: --client-ca-file=""
May 29 13:42:05.930169 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926667    2571 flags.go:64] FLAG: --cloud-config=""
May 29 13:42:05.930169 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926670    2571 flags.go:64] FLAG: --cloud-provider="external"
May 29 13:42:05.930169 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926673    2571 flags.go:64] FLAG: --cluster-dns="[]"
May 29 13:42:05.930169 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926680    2571 flags.go:64] FLAG: --cluster-domain=""
May 29 13:42:05.930169 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926683    2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 29 13:42:05.930169 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926686    2571 flags.go:64] FLAG: --config-dir=""
May 29 13:42:05.930169 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926689    2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 29 13:42:05.930169 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926693    2571 flags.go:64] FLAG: --container-log-max-files="5"
May 29 13:42:05.930169 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926697    2571 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 29 13:42:05.930169 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926700    2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 29 13:42:05.930169 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926703    2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 29 13:42:05.930169 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926707    2571 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 29 13:42:05.930169 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926710    2571 flags.go:64] FLAG: --contention-profiling="false"
May 29 13:42:05.930169 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926714    2571 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 29 13:42:05.930169 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926717    2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 29 13:42:05.930169 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926727    2571 flags.go:64] FLAG: --cpu-manager-policy="none"
May 29 13:42:05.930169 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926731    2571 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 29 13:42:05.930169 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926735    2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 29 13:42:05.930771 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926738    2571 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 29 13:42:05.930771 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926742    2571 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 29 13:42:05.930771 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926745    2571 flags.go:64] FLAG: --enable-load-reader="false"
May 29 13:42:05.930771 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926748    2571 flags.go:64] FLAG: --enable-server="true"
May 29 13:42:05.930771 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926751    2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 29 13:42:05.930771 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926759    2571 flags.go:64] FLAG: --event-burst="100"
May 29 13:42:05.930771 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926762    2571 flags.go:64] FLAG: --event-qps="50"
May 29 13:42:05.930771 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926765    2571 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 29 13:42:05.930771 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926768    2571 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 29 13:42:05.930771 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926772    2571 flags.go:64] FLAG: --eviction-hard=""
May 29 13:42:05.930771 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926776    2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 29 13:42:05.930771 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926779    2571 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 29 13:42:05.930771 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926782    2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 29 13:42:05.930771 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926785    2571 flags.go:64] FLAG: --eviction-soft=""
May 29 13:42:05.930771 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926788    2571 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 29 13:42:05.930771 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926791    2571 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 29 13:42:05.930771 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926794    2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 29 13:42:05.930771 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926797    2571 flags.go:64] FLAG: --experimental-mounter-path=""
May 29 13:42:05.930771 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926800    2571 flags.go:64] FLAG: --fail-cgroupv1="false"
May 29 13:42:05.930771 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926803    2571 flags.go:64] FLAG: --fail-swap-on="true"
May 29 13:42:05.930771 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926806    2571 flags.go:64] FLAG: --feature-gates=""
May 29 13:42:05.930771 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926810    2571 flags.go:64] FLAG: --file-check-frequency="20s"
May 29 13:42:05.930771 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926813    2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 29 13:42:05.930771 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926816    2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 29 13:42:05.930771 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926819    2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 29 13:42:05.931391 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926822    2571 flags.go:64] FLAG: --healthz-port="10248"
May 29 13:42:05.931391 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926825    2571 flags.go:64] FLAG: --help="false"
May 29 13:42:05.931391 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926828    2571 flags.go:64] FLAG: --hostname-override="ip-10-0-141-1.ec2.internal"
May 29 13:42:05.931391 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926831    2571 flags.go:64] FLAG: --housekeeping-interval="10s"
May 29 13:42:05.931391 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926834    2571 flags.go:64] FLAG: --http-check-frequency="20s"
May 29 13:42:05.931391 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926843    2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 29 13:42:05.931391 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926851    2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 29 13:42:05.931391 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926855    2571 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 29 13:42:05.931391 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926858    2571 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 29 13:42:05.931391 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926861    2571 flags.go:64] FLAG: --image-service-endpoint=""
May 29 13:42:05.931391 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926864    2571 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 29 13:42:05.931391 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926867    2571 flags.go:64] FLAG: --kube-api-burst="100"
May 29 13:42:05.931391 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926870    2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 29 13:42:05.931391 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926873    2571 flags.go:64] FLAG: --kube-api-qps="50"
May 29 13:42:05.931391 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926876    2571 flags.go:64] FLAG: --kube-reserved=""
May 29 13:42:05.931391 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926879    2571 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 29 13:42:05.931391 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926883    2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 29 13:42:05.931391 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926886    2571 flags.go:64] FLAG: --kubelet-cgroups=""
May 29 13:42:05.931391 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926889    2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 29 13:42:05.931391 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926892    2571 flags.go:64] FLAG: --lock-file=""
May 29 13:42:05.931391 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926895    2571 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 29 13:42:05.931391 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926898    2571 flags.go:64] FLAG: --log-flush-frequency="5s"
May 29 13:42:05.931391 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926901    2571 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 29 13:42:05.931391 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926907    2571 flags.go:64] FLAG: --log-json-split-stream="false"
May 29 13:42:05.932015 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926909    2571 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 29 13:42:05.932015 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926912    2571 flags.go:64] FLAG: --log-text-split-stream="false"
May 29 13:42:05.932015 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926915    2571 flags.go:64] FLAG: --logging-format="text"
May 29 13:42:05.932015 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926918    2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 29 13:42:05.932015 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926921    2571 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 29 13:42:05.932015 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926924    2571 flags.go:64] FLAG: --manifest-url=""
May 29 13:42:05.932015 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926927    2571 flags.go:64] FLAG: --manifest-url-header=""
May 29 13:42:05.932015 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926932    2571 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 29 13:42:05.932015 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926935    2571 flags.go:64] FLAG: --max-open-files="1000000"
May 29 13:42:05.932015 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926939    2571 flags.go:64] FLAG: --max-pods="110"
May 29 13:42:05.932015 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926942    2571 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 29 13:42:05.932015 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926945    2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 29 13:42:05.932015 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926948    2571 flags.go:64] FLAG: --memory-manager-policy="None"
May 29 13:42:05.932015 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926951    2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 29 13:42:05.932015 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926962    2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 29 13:42:05.932015 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926965    2571 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 29 13:42:05.932015 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926968    2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 29 13:42:05.932015 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926976    2571 flags.go:64] FLAG: --node-status-max-images="50"
May 29 13:42:05.932015 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926980    2571 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 29 13:42:05.932015 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926983    2571 flags.go:64] FLAG: --oom-score-adj="-999"
May 29 13:42:05.932015 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926986    2571 flags.go:64] FLAG: --pod-cidr=""
May 29 13:42:05.932015 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926989    2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 29 13:42:05.932015 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926995    2571 flags.go:64] FLAG: --pod-manifest-path=""
May 29 13:42:05.932577 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.926998    2571 flags.go:64] FLAG: --pod-max-pids="-1"
May 29 13:42:05.932577 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927001    2571 flags.go:64] FLAG: --pods-per-core="0"
May 29 13:42:05.932577 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927005    2571 flags.go:64] FLAG: --port="10250"
May 29 13:42:05.932577 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927008    2571 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 29 13:42:05.932577 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927012    2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b86259c558fd37c8"
May 29 13:42:05.932577 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927015    2571 flags.go:64] FLAG: --qos-reserved=""
May 29 13:42:05.932577 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927018    2571 flags.go:64] FLAG: --read-only-port="10255"
May 29 13:42:05.932577 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927021    2571 flags.go:64] FLAG: --register-node="true"
May 29 13:42:05.932577 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927024    2571 flags.go:64] FLAG: --register-schedulable="true"
May 29 13:42:05.932577 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927027    2571 flags.go:64] FLAG: --register-with-taints=""
May 29 13:42:05.932577 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927031    2571 flags.go:64] FLAG: --registry-burst="10"
May 29 13:42:05.932577 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927034    2571 flags.go:64] FLAG: --registry-qps="5"
May 29 13:42:05.932577 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927036    2571 flags.go:64] FLAG: --reserved-cpus=""
May 29 13:42:05.932577 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927039    2571 flags.go:64] FLAG: --reserved-memory=""
May 29 13:42:05.932577 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927043    2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 29 13:42:05.932577 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927064    2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 29 13:42:05.932577 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927067    2571 flags.go:64] FLAG: --rotate-certificates="false"
May 29 13:42:05.932577 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927070    2571 flags.go:64] FLAG: --rotate-server-certificates="false"
May 29 13:42:05.932577 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927073    2571 flags.go:64] FLAG: --runonce="false"
May 29 13:42:05.932577 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927076    2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 29 13:42:05.932577 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927080    2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 29 13:42:05.932577 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927083    2571 flags.go:64] FLAG: --seccomp-default="false"
May 29 13:42:05.932577 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927086    2571 flags.go:64] FLAG: --serialize-image-pulls="true"
May 29 13:42:05.932577 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927089    2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 29 13:42:05.932577 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927099    2571 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 29 13:42:05.932577 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927103    2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 29 13:42:05.933246 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927106    2571 flags.go:64] FLAG: --storage-driver-password="root"
May 29 13:42:05.933246 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927109    2571 flags.go:64] FLAG: --storage-driver-secure="false"
May 29 13:42:05.933246 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927112    2571 flags.go:64] FLAG: --storage-driver-table="stats"
May 29 13:42:05.933246 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927115    2571 flags.go:64] FLAG: --storage-driver-user="root"
May 29 13:42:05.933246 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927118    2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 29 13:42:05.933246 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927121    2571 flags.go:64] FLAG: --sync-frequency="1m0s"
May 29 13:42:05.933246 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927124    2571 flags.go:64] FLAG: --system-cgroups=""
May 29 13:42:05.933246 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927127    2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 29 13:42:05.933246 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927133    2571 flags.go:64] FLAG: --system-reserved-cgroup=""
May 29 13:42:05.933246 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927136    2571 flags.go:64] FLAG: --tls-cert-file=""
May 29 13:42:05.933246 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927138    2571 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 29 13:42:05.933246 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927145    2571 flags.go:64] FLAG: --tls-min-version=""
May 29 13:42:05.933246 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927148    2571 flags.go:64] FLAG: --tls-private-key-file=""
May 29 13:42:05.933246 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927151    2571 flags.go:64] FLAG: --topology-manager-policy="none"
May 29 13:42:05.933246 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927154    2571 flags.go:64] FLAG: --topology-manager-policy-options=""
May 29 13:42:05.933246 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927157    2571 flags.go:64] FLAG: --topology-manager-scope="container"
May 29 13:42:05.933246 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927160    2571 flags.go:64] FLAG: --v="2"
May 29 13:42:05.933246 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927164    2571 flags.go:64] FLAG: --version="false"
May 29 13:42:05.933246 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927169    2571 flags.go:64] FLAG: --vmodule=""
May 29 13:42:05.933246 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927173    2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 29 13:42:05.933246 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927176    2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 29 13:42:05.933246 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927284    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 29 13:42:05.933246 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927288    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 29 13:42:05.933246 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927291    2571 feature_gate.go:328] unrecognized feature gate: Example2
May 29 13:42:05.933246 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927294    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 29 13:42:05.933870 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927297    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 29 13:42:05.933870 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927299    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 29 13:42:05.933870 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927302    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 29 13:42:05.933870 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927305    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 29 13:42:05.933870 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927308    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 29 13:42:05.933870 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927313    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 29 13:42:05.933870 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927317    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 29 13:42:05.933870 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927319    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 29 13:42:05.933870 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927322    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 29 13:42:05.933870 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927325    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
May 29 13:42:05.933870 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927327    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 29 13:42:05.933870 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927330    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 29 13:42:05.933870 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927333    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 29 13:42:05.933870 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927335    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 29 13:42:05.933870 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927338    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 29 13:42:05.933870 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927341    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 29 13:42:05.933870 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927343    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 29 13:42:05.933870 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927346    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 29 13:42:05.933870 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927348    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 29 13:42:05.933870 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927351    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 29 13:42:05.934552 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927354    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 29 13:42:05.934552 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927356    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 29 13:42:05.934552 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927359    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 29 13:42:05.934552 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927361    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 29 13:42:05.934552 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927364    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 29 13:42:05.934552 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927367    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 29 13:42:05.934552 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927369    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 29 13:42:05.934552 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927372    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 29 13:42:05.934552 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927374    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 29 13:42:05.934552 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927377    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 29 13:42:05.934552 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927379    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 29 13:42:05.934552 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927382    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 29 13:42:05.934552 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927384    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 29 13:42:05.934552 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927387    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 29 13:42:05.934552 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927389    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 29 13:42:05.934552 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927392    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 29 13:42:05.934552 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927395    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 29 13:42:05.934552 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927398    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 29 13:42:05.934552 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927401    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 29 13:42:05.934552 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927404    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 29 13:42:05.935368 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927407    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 29 13:42:05.935368 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927409    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 29 13:42:05.935368 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927412    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 29 13:42:05.935368 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927414    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 29 13:42:05.935368 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927417    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 29 13:42:05.935368 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927419    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 29 13:42:05.935368 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927423    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 29 13:42:05.935368 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927425    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 29 13:42:05.935368 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927428    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 29 13:42:05.935368 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927430    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 29 13:42:05.935368 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927434    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 29 13:42:05.935368 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927437    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 29 13:42:05.935368 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927441    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 29 13:42:05.935368 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927443    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
May 29 13:42:05.935368 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927447    2571 feature_gate.go:328] unrecognized feature gate: Example
May 29 13:42:05.935368 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927449    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 29 13:42:05.935368 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927452    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 29 13:42:05.935368 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927455    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 29 13:42:05.935368 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927459    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 29 13:42:05.935854 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927463    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 29 13:42:05.935854 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927466    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 29 13:42:05.935854 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927469    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 29 13:42:05.935854 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927472    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 29 13:42:05.935854 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927475    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 29 13:42:05.935854 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927477    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 29 13:42:05.935854 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927480    2571 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 29 13:42:05.935854 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927483    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 29 13:42:05.935854 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927485    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 29 13:42:05.935854 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927487    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 29 13:42:05.935854 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927492    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 29 13:42:05.935854 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927494    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 29 13:42:05.935854 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927497    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 29 13:42:05.935854 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927500    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 29 13:42:05.935854 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927503    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 29 13:42:05.935854 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927505    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 29 13:42:05.935854 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927508    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 29 13:42:05.935854 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927511    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 29 13:42:05.935854 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927513    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 29 13:42:05.935854 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927516    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 29 13:42:05.936410 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927519    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 29 13:42:05.936410 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927522    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 29 13:42:05.936410 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.927524    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 29 13:42:05.936410 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.927530    2571 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 29 13:42:05.936648 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.936626    2571 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 29 13:42:05.936680 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.936650    2571 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 29 13:42:05.936710 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936702    2571 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 29 13:42:05.936710 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936707    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 29 13:42:05.936764 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936711    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 29 13:42:05.936764 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936715    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 29 13:42:05.936764 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936719    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 29 13:42:05.936764 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936724    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 29 13:42:05.936764 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936726    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 29 13:42:05.936764 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936729    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 29 13:42:05.936764 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936732    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
May 29 13:42:05.936764 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936735    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 29 13:42:05.936764 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936737    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 29 13:42:05.936764 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936740    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 29 13:42:05.936764 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936743    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 29 13:42:05.936764 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936746    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 29 13:42:05.936764 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936748    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 29 13:42:05.936764 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936751    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 29 13:42:05.936764 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936753    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 29 13:42:05.936764 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936756    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 29 13:42:05.936764 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936759    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 29 13:42:05.936764 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936762    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 29 13:42:05.936764 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936765    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 29 13:42:05.936764 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936767    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 29 13:42:05.937311 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936770    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 29 13:42:05.937311 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936773    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 29 13:42:05.937311 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936776    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 29 13:42:05.937311 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936780    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 29 13:42:05.937311 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936782    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 29 13:42:05.937311 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936785    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 29 13:42:05.937311 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936788    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
May 29 13:42:05.937311 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936790    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 29 13:42:05.937311 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936793    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 29 13:42:05.937311 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936795    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 29 13:42:05.937311 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936798    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 29 13:42:05.937311 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936801    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 29 13:42:05.937311 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936804    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 29 13:42:05.937311 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936806    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 29 13:42:05.937311 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936809    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 29 13:42:05.937311 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936811    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 29 13:42:05.937311 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936814    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 29 13:42:05.937311 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936817    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 29 13:42:05.937311 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936819    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 29 13:42:05.937311 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936821    2571 feature_gate.go:328] unrecognized feature gate: Example2
May 29 13:42:05.937817 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936824    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 29 13:42:05.937817 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936826    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 29 13:42:05.937817 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936829    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 29 13:42:05.937817 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936831    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 29 13:42:05.937817 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936834    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 29 13:42:05.937817 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936836    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 29 13:42:05.937817 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936839    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 29 13:42:05.937817 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936841    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 29 13:42:05.937817 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936844    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 29 13:42:05.937817 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936846    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 29 13:42:05.937817 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936849    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 29 13:42:05.937817 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936852    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 29 13:42:05.937817 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936855    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 29 13:42:05.937817 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936858    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 29 13:42:05.937817 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936861    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 29 13:42:05.937817 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936864    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 29 13:42:05.937817 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936866    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 29 13:42:05.937817 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936869    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 29 13:42:05.937817 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936871    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 29 13:42:05.938282 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936874    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 29 13:42:05.938282 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936878    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 29 13:42:05.938282 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936882    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 29 13:42:05.938282 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936885    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 29 13:42:05.938282 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936889    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 29 13:42:05.938282 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936892    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 29 13:42:05.938282 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936895    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 29 13:42:05.938282 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936898    2571 feature_gate.go:328] unrecognized feature gate: Example
May 29 13:42:05.938282 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936901    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 29 13:42:05.938282 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936904    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 29 13:42:05.938282 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936906    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 29 13:42:05.938282 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936909    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 29 13:42:05.938282 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936912    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 29 13:42:05.938282 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936914    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 29 13:42:05.938282 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936917    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 29 13:42:05.938282 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936921    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 29 13:42:05.938282 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936923    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 29 13:42:05.938282 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936926    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 29 13:42:05.938282 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936929    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 29 13:42:05.938738 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936931    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 29 13:42:05.938738 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936934    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 29 13:42:05.938738 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936937    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 29 13:42:05.938738 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936939    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 29 13:42:05.938738 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936942    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 29 13:42:05.938738 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.936944    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 29 13:42:05.938738 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.936949    2571 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 29 13:42:05.938738 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937063    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 29 13:42:05.938738 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937070    2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 29 13:42:05.938738 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937073    2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 29 13:42:05.938738 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937077    2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 29 13:42:05.938738 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937080    2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 29 13:42:05.938738 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937083    2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 29 13:42:05.938738 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937086    2571 feature_gate.go:328] unrecognized feature gate: NewOLM
May 29 13:42:05.938738 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937089    2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 29 13:42:05.938738 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937092    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 29 13:42:05.939153 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937095    2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 29 13:42:05.939153 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937098    2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 29 13:42:05.939153 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937100    2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 29 13:42:05.939153 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937103    2571 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 29 13:42:05.939153 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937105    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 29 13:42:05.939153 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937108    2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 29 13:42:05.939153 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937111    2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 29 13:42:05.939153 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937113    2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 29 13:42:05.939153 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937116    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 29 13:42:05.939153 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937118    2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 29 13:42:05.939153 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937121    2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 29 13:42:05.939153 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937124    2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 29 13:42:05.939153 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937126    2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 29 13:42:05.939153 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937129    2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 29 13:42:05.939153 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937132    2571 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 29 13:42:05.939153 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937135    2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 29 13:42:05.939153 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937137    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 29 13:42:05.939153 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937140    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 29 13:42:05.939153 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937143    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 29 13:42:05.939622 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937146    2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 29 13:42:05.939622 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937151    2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 29 13:42:05.939622 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937155    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 29 13:42:05.939622 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937159    2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 29 13:42:05.939622 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937163    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 29 13:42:05.939622 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937166    2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 29 13:42:05.939622 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937169    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 29 13:42:05.939622 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937171    2571 feature_gate.go:328] unrecognized feature gate: Example
May 29 13:42:05.939622 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937174    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 29 13:42:05.939622 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937176    2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 29 13:42:05.939622 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937179    2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 29 13:42:05.939622 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937181    2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 29 13:42:05.939622 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937184    2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 29 13:42:05.939622 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937186    2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 29 13:42:05.939622 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937189    2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 29 13:42:05.939622 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937191    2571 feature_gate.go:328] unrecognized feature gate: Example2
May 29 13:42:05.939622 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937194    2571 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 29 13:42:05.939622 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937197    2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 29 13:42:05.939622 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937199    2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 29 13:42:05.940089 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937202    2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 29 13:42:05.940089 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937204    2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 29 13:42:05.940089 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937207    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 29 13:42:05.940089 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937209    2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 29 13:42:05.940089 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937212    2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 29 13:42:05.940089 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937214    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 29 13:42:05.940089 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937217    2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 29 13:42:05.940089 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937219    2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 29 13:42:05.940089 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937223    2571 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 29 13:42:05.940089 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937225    2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 29 13:42:05.940089 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937228    2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 29 13:42:05.940089 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937231    2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 29 13:42:05.940089 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937233    2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 29 13:42:05.940089 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937236    2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 29 13:42:05.940089 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937238    2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 29 13:42:05.940089 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937241    2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 29 13:42:05.940089 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937243    2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 29 13:42:05.940089 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937247    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 29 13:42:05.940089 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937249    2571 feature_gate.go:328] unrecognized feature gate: DualReplica
May 29 13:42:05.940089 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937252    2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 29 13:42:05.940576 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937255    2571 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 29 13:42:05.940576 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937257    2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 29 13:42:05.940576 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937260    2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 29 13:42:05.940576 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937263    2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 29 13:42:05.940576 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937265    2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 29 13:42:05.940576 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937268    2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 29 13:42:05.940576 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937270    2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 29 13:42:05.940576 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937273    2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 29 13:42:05.940576 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937275    2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 29 13:42:05.940576 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937278    2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 29 13:42:05.940576 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937281    2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 29 13:42:05.940576 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937283    2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 29 13:42:05.940576 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937286    2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 29 13:42:05.940576 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937288    2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 29 13:42:05.940576 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937291    2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 29 13:42:05.940576 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937293    2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 29 13:42:05.940576 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937296    2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 29 13:42:05.940576 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937298    2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 29 13:42:05.940576 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:05.937301    2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 29 13:42:05.941035 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.937306    2571 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 29 13:42:05.941035 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.938174    2571 server.go:962] "Client rotation is on, will bootstrap in background"
May 29 13:42:05.941387 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.941372    2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 29 13:42:05.942601 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.942589    2571 server.go:1019] "Starting client certificate rotation"
May 29 13:42:05.942698 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.942683    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 29 13:42:05.942737 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.942713    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 29 13:42:05.970175 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.970151    2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 29 13:42:05.973131 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.973105    2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 29 13:42:05.984682 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.984664    2571 log.go:25] "Validated CRI v1 runtime API"
May 29 13:42:05.991325 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.991306    2571 log.go:25] "Validated CRI v1 image API"
May 29 13:42:05.992803 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.992788    2571 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 29 13:42:05.998521 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.998497    2571 fs.go:135] Filesystem UUIDs: map[7529e9bb-dc6b-46b8-bbdc-16501b0a32a7:/dev/nvme0n1p3 76dc31ce-b837-49e8-81fe-2b332ff4bb5f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
May 29 13:42:05.998614 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:05.998519    2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 29 13:42:06.005575 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.005440    2571 manager.go:217] Machine: {Timestamp:2026-05-29 13:42:06.004894029 +0000 UTC m=+0.471964101 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097877 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec277b2834a1c14672d7079debb07440 SystemUUID:ec277b28-34a1-c146-72d7-079debb07440 BootID:d171592a-e815-4b46-9095-160918e5c9b9 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:60:98:b6:8c:27 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:60:98:b6:8c:27 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:5d:b7:d4:b6:3e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 29 13:42:06.005696 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.005587    2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 29 13:42:06.005696 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.005675    2571 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 29 13:42:06.006104 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.006088    2571 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 29 13:42:06.006884 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.006857    2571 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 29 13:42:06.007024 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.006886    2571 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-1.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 29 13:42:06.007087 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.007034    2571 topology_manager.go:138] "Creating topology manager with none policy"
May 29 13:42:06.007087 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.007042    2571 container_manager_linux.go:306] "Creating device plugin manager"
May 29 13:42:06.007087 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.007064    2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 29 13:42:06.008021 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.008011    2571 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 29 13:42:06.009979 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.009965    2571 state_mem.go:36] "Initialized new in-memory state store"
May 29 13:42:06.010292 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.010282    2571 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 29 13:42:06.013088 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.013077    2571 kubelet.go:491] "Attempting to sync node with API server"
May 29 13:42:06.013121 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.013092    2571 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 29 13:42:06.013121 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.013104    2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 29 13:42:06.013121 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.013113    2571 kubelet.go:397] "Adding apiserver pod source"
May 29 13:42:06.013240 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.013127    2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 29 13:42:06.014991 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.014978    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 29 13:42:06.015061 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.014999    2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 29 13:42:06.018491 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.018475    2571 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 29 13:42:06.021103 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.021086    2571 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 29 13:42:06.023355 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.023340    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 29 13:42:06.023446 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.023361    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 29 13:42:06.023446 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.023370    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 29 13:42:06.023446 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.023379    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 29 13:42:06.023446 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.023392    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 29 13:42:06.023446 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.023401    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 29 13:42:06.023446 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.023410    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 29 13:42:06.023446 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.023418    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 29 13:42:06.023446 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.023428    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 29 13:42:06.023446 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.023436    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 29 13:42:06.023446 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.023449    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 29 13:42:06.023978 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.023459    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 29 13:42:06.023978 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.023499    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 29 13:42:06.023978 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.023509    2571 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 29 13:42:06.024618 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:06.024594    2571 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 29 13:42:06.024618 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:06.024596    2571 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-1.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 29 13:42:06.026893 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.026876    2571 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 29 13:42:06.026977 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.026923    2571 server.go:1295] "Started kubelet"
May 29 13:42:06.027039 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.026998    2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 29 13:42:06.027457 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.027398    2571 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 29 13:42:06.027529 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.027476    2571 server_v1.go:47] "podresources" method="list" useActivePods=true
May 29 13:42:06.027801 ip-10-0-141-1 systemd[1]: Started Kubernetes Kubelet.
May 29 13:42:06.028075 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.028043    2571 server.go:317] "Adding debug handlers to kubelet server"
May 29 13:42:06.028839 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.028825    2571 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 29 13:42:06.036893 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.036867    2571 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-1.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 29 13:42:06.036994 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.036944    2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 29 13:42:06.036994 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.036957    2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 29 13:42:06.037104 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.037032    2571 volume_manager.go:295] "The desired_state_of_world populator starts"
May 29 13:42:06.037104 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.037041    2571 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 29 13:42:06.037289 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.037276    2571 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 29 13:42:06.037349 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.037325    2571 reconstruct.go:97] "Volume reconstruction finished"
May 29 13:42:06.037349 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.037331    2571 reconciler.go:26] "Reconciler: start to sync state"
May 29 13:42:06.038088 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:06.036886    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-1.ec2.internal.18b40d128bf59cce  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-1.ec2.internal,UID:ip-10-0-141-1.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-1.ec2.internal,},FirstTimestamp:2026-05-29 13:42:06.026890446 +0000 UTC m=+0.493960519,LastTimestamp:2026-05-29 13:42:06.026890446 +0000 UTC m=+0.493960519,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-1.ec2.internal,}"
May 29 13:42:06.038748 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:06.038728    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-1.ec2.internal\" not found"
May 29 13:42:06.038839 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:06.038789    2571 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 29 13:42:06.038839 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.038836    2571 factory.go:55] Registering systemd factory
May 29 13:42:06.038940 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.038889    2571 factory.go:223] Registration of the systemd container factory successfully
May 29 13:42:06.039192 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.039174    2571 factory.go:153] Registering CRI-O factory
May 29 13:42:06.039257 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.039195    2571 factory.go:223] Registration of the crio container factory successfully
May 29 13:42:06.039346 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.039249    2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 29 13:42:06.039346 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.039277    2571 factory.go:103] Registering Raw factory
May 29 13:42:06.039346 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.039292    2571 manager.go:1196] Started watching for new ooms in manager
May 29 13:42:06.039662 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.039626    2571 manager.go:319] Starting recovery of all containers
May 29 13:42:06.042893 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:06.042857    2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-1.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 29 13:42:06.043019 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:06.042995    2571 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 29 13:42:06.051691 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.051530    2571 manager.go:324] Recovery completed
May 29 13:42:06.054108 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.054090    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-82n6d"
May 29 13:42:06.055767 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.055755    2571 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 29 13:42:06.057960 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.057943    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-1.ec2.internal" event="NodeHasSufficientMemory"
May 29 13:42:06.058027 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.057974    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-1.ec2.internal" event="NodeHasNoDiskPressure"
May 29 13:42:06.058027 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.057985    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-1.ec2.internal" event="NodeHasSufficientPID"
May 29 13:42:06.058539 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.058526    2571 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 29 13:42:06.058539 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.058537    2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 29 13:42:06.058622 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.058553    2571 state_mem.go:36] "Initialized new in-memory state store"
May 29 13:42:06.059645 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:06.059584    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-1.ec2.internal.18b40d128dcfb428  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-1.ec2.internal,UID:ip-10-0-141-1.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-1.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-1.ec2.internal,},FirstTimestamp:2026-05-29 13:42:06.057960488 +0000 UTC m=+0.525030567,LastTimestamp:2026-05-29 13:42:06.057960488 +0000 UTC m=+0.525030567,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-1.ec2.internal,}"
May 29 13:42:06.062419 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.062406    2571 policy_none.go:49] "None policy: Start"
May 29 13:42:06.062466 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.062423    2571 memory_manager.go:186] "Starting memorymanager" policy="None"
May 29 13:42:06.062466 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.062433    2571 state_mem.go:35] "Initializing new in-memory state store"
May 29 13:42:06.070390 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.070370    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-82n6d"
May 29 13:42:06.072011 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:06.071945    2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-1.ec2.internal.18b40d128dcffded  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-1.ec2.internal,UID:ip-10-0-141-1.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-1.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-1.ec2.internal,},FirstTimestamp:2026-05-29 13:42:06.057979373 +0000 UTC m=+0.525049445,LastTimestamp:2026-05-29 13:42:06.057979373 +0000 UTC m=+0.525049445,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-1.ec2.internal,}"
May 29 13:42:06.104689 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.104671    2571 manager.go:341] "Starting Device Plugin manager"
May 29 13:42:06.130289 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:06.104715    2571 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 29 13:42:06.130289 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.104725    2571 server.go:85] "Starting device plugin registration server"
May 29 13:42:06.130289 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.105018    2571 eviction_manager.go:189] "Eviction manager: starting control loop"
May 29 13:42:06.130289 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.105046    2571 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 29 13:42:06.130289 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.105169    2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 29 13:42:06.130289 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.105261    2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 29 13:42:06.130289 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.105270    2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 29 13:42:06.130289 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:06.105915    2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 29 13:42:06.130289 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:06.105970    2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-1.ec2.internal\" not found"
May 29 13:42:06.142154 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.142115    2571 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 29 13:42:06.143330 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.143313    2571 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 29 13:42:06.143379 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.143347    2571 status_manager.go:230] "Starting to sync pod status with apiserver"
May 29 13:42:06.143379 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.143371    2571 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 29 13:42:06.143457 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.143381    2571 kubelet.go:2451] "Starting kubelet main sync loop"
May 29 13:42:06.143457 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:06.143425    2571 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 29 13:42:06.150422 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.150397    2571 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 29 13:42:06.205820 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.205744    2571 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 29 13:42:06.206893 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.206876    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-1.ec2.internal" event="NodeHasSufficientMemory"
May 29 13:42:06.206992 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.206910    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-1.ec2.internal" event="NodeHasNoDiskPressure"
May 29 13:42:06.206992 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.206924    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-1.ec2.internal" event="NodeHasSufficientPID"
May 29 13:42:06.206992 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.206952    2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-1.ec2.internal"
May 29 13:42:06.216908 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.216888    2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-1.ec2.internal"
May 29 13:42:06.216974 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:06.216913    2571 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-1.ec2.internal\": node \"ip-10-0-141-1.ec2.internal\" not found"
May 29 13:42:06.237631 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:06.237606    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-1.ec2.internal\" not found"
May 29 13:42:06.243830 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.243809    2571 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-1.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-1.ec2.internal"]
May 29 13:42:06.243888 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.243871    2571 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 29 13:42:06.244676 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.244658    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-1.ec2.internal" event="NodeHasSufficientMemory"
May 29 13:42:06.244730 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.244689    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-1.ec2.internal" event="NodeHasNoDiskPressure"
May 29 13:42:06.244730 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.244702    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-1.ec2.internal" event="NodeHasSufficientPID"
May 29 13:42:06.245806 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.245792    2571 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 29 13:42:06.245978 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.245963    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-1.ec2.internal"
May 29 13:42:06.246013 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.245994    2571 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 29 13:42:06.246535 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.246521    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-1.ec2.internal" event="NodeHasSufficientMemory"
May 29 13:42:06.246535 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.246534    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-1.ec2.internal" event="NodeHasSufficientMemory"
May 29 13:42:06.246643 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.246557    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-1.ec2.internal" event="NodeHasNoDiskPressure"
May 29 13:42:06.246643 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.246566    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-1.ec2.internal" event="NodeHasNoDiskPressure"
May 29 13:42:06.246643 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.246571    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-1.ec2.internal" event="NodeHasSufficientPID"
May 29 13:42:06.246643 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.246587    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-1.ec2.internal" event="NodeHasSufficientPID"
May 29 13:42:06.247585 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.247569    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-1.ec2.internal"
May 29 13:42:06.247634 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.247596    2571 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 29 13:42:06.248411 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.248396    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-1.ec2.internal" event="NodeHasSufficientMemory"
May 29 13:42:06.248466 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.248425    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-1.ec2.internal" event="NodeHasNoDiskPressure"
May 29 13:42:06.248466 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.248437    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-1.ec2.internal" event="NodeHasSufficientPID"
May 29 13:42:06.268798 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:06.268775    2571 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-1.ec2.internal\" not found" node="ip-10-0-141-1.ec2.internal"
May 29 13:42:06.273322 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:06.273307    2571 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-1.ec2.internal\" not found" node="ip-10-0-141-1.ec2.internal"
May 29 13:42:06.337741 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:06.337705    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-1.ec2.internal\" not found"
May 29 13:42:06.337741 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.337745    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f36931e78e4f753d131caf513c9d0151-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-1.ec2.internal\" (UID: \"f36931e78e4f753d131caf513c9d0151\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-1.ec2.internal"
May 29 13:42:06.337915 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.337768    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f36931e78e4f753d131caf513c9d0151-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-1.ec2.internal\" (UID: \"f36931e78e4f753d131caf513c9d0151\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-1.ec2.internal"
May 29 13:42:06.337915 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.337840    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/adf64b7f1c8724ca5ea45aa00abb85e2-config\") pod \"kube-apiserver-proxy-ip-10-0-141-1.ec2.internal\" (UID: \"adf64b7f1c8724ca5ea45aa00abb85e2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-1.ec2.internal"
May 29 13:42:06.437929 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:06.437883    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-1.ec2.internal\" not found"
May 29 13:42:06.438091 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.437980    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f36931e78e4f753d131caf513c9d0151-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-1.ec2.internal\" (UID: \"f36931e78e4f753d131caf513c9d0151\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-1.ec2.internal"
May 29 13:42:06.438091 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.438015    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f36931e78e4f753d131caf513c9d0151-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-1.ec2.internal\" (UID: \"f36931e78e4f753d131caf513c9d0151\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-1.ec2.internal"
May 29 13:42:06.438091 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.438038    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/adf64b7f1c8724ca5ea45aa00abb85e2-config\") pod \"kube-apiserver-proxy-ip-10-0-141-1.ec2.internal\" (UID: \"adf64b7f1c8724ca5ea45aa00abb85e2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-1.ec2.internal"
May 29 13:42:06.438214 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.438103    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/adf64b7f1c8724ca5ea45aa00abb85e2-config\") pod \"kube-apiserver-proxy-ip-10-0-141-1.ec2.internal\" (UID: \"adf64b7f1c8724ca5ea45aa00abb85e2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-1.ec2.internal"
May 29 13:42:06.438214 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.438102    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f36931e78e4f753d131caf513c9d0151-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-1.ec2.internal\" (UID: \"f36931e78e4f753d131caf513c9d0151\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-1.ec2.internal"
May 29 13:42:06.438214 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.438103    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f36931e78e4f753d131caf513c9d0151-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-1.ec2.internal\" (UID: \"f36931e78e4f753d131caf513c9d0151\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-1.ec2.internal"
May 29 13:42:06.538620 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:06.538534    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-1.ec2.internal\" not found"
May 29 13:42:06.571719 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.571691    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-1.ec2.internal"
May 29 13:42:06.576190 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.576174    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-1.ec2.internal"
May 29 13:42:06.638915 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:06.638875    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-1.ec2.internal\" not found"
May 29 13:42:06.739220 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:06.739194    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-1.ec2.internal\" not found"
May 29 13:42:06.839641 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:06.839563    2571 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-141-1.ec2.internal\" not found"
May 29 13:42:06.885810 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.885781    2571 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 29 13:42:06.921453 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.921419    2571 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 29 13:42:06.938187 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.938165    2571 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-1.ec2.internal"
May 29 13:42:06.942560 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.942546    2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 29 13:42:06.942696 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.942679    2571 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 29 13:42:06.942738 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.942721    2571 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 29 13:42:06.942773 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.942713    2571 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 29 13:42:06.942773 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:06.942720    2571 kubelet.go:3342] "Failed creating a mirror pod" err="Post \"https://a4406c40ab9df4fde82eb5f08ab8f505-fd45bfaeb364e5b3.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": read tcp 10.0.141.1:52862->23.20.214.44:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-1.ec2.internal"
May 29 13:42:06.942773 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.942766    2571 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-1.ec2.internal"
May 29 13:42:06.960510 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:06.960492    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 29 13:42:07.014222 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.014186    2571 apiserver.go:52] "Watching apiserver"
May 29 13:42:07.022384 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.022323    2571 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 29 13:42:07.023790 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.023764    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-bz56r","openshift-cluster-node-tuning-operator/tuned-xqvgg","openshift-network-operator/iptables-alerter-k7qwn","openshift-ovn-kubernetes/ovnkube-node-p2drl","kube-system/kube-apiserver-proxy-ip-10-0-141-1.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt","openshift-dns/node-resolver-xqvkm","openshift-image-registry/node-ca-9wzn2","openshift-multus/multus-46jjq","openshift-multus/multus-additional-cni-plugins-4vtqj","openshift-multus/network-metrics-daemon-n48xk","openshift-network-diagnostics/network-check-target-vhmzh"]
May 29 13:42:07.026061 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.026031    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bz56r"
May 29 13:42:07.028817 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.028800    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 29 13:42:07.029248 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.029222    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 29 13:42:07.029388 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.029372    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6cjbk\""
May 29 13:42:07.030609 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.030585    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.030773 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.030755    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k7qwn"
May 29 13:42:07.032836 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.032817    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 29 13:42:07.032926 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.032851    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 29 13:42:07.032926 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.032900    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 29 13:42:07.033036 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.033001    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bzqgh\""
May 29 13:42:07.033110 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.033093    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.033311 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.033277    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 29 13:42:07.033438 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.033421    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 29 13:42:07.033506 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.033459    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7hggn\""
May 29 13:42:07.035632 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.035616    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt"
May 29 13:42:07.037397 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.037379    2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 29 13:42:07.037531 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.037494    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 29 13:42:07.037531 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.037531    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 29 13:42:07.037661 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.037504    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 29 13:42:07.037887 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.037869    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 29 13:42:07.037979 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.037960    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 29 13:42:07.037979 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.037965    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 29 13:42:07.038242 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.037990    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 29 13:42:07.038242 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.038147    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5jj8l\""
May 29 13:42:07.038242 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.038218    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xqvkm"
May 29 13:42:07.038404 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.038308    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 29 13:42:07.038606 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.038592    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 29 13:42:07.038729 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.038716    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vl45n\""
May 29 13:42:07.040492 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.040450    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d022803-adf3-4686-8136-a57ab660afad-agent-certs\") pod \"konnectivity-agent-bz56r\" (UID: \"2d022803-adf3-4686-8136-a57ab660afad\") " pod="kube-system/konnectivity-agent-bz56r"
May 29 13:42:07.040586 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.040505    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-etc-sysconfig\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.040586 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.040531    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-run\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.040586 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.040553    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-host\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.040586 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.040576    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-log-socket\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.040831 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.040630    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.040831 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.040697    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-sys\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.040831 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.040744    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm24k\" (UniqueName: \"kubernetes.io/projected/dda729c3-39e3-45ac-ba23-1628f3874144-kube-api-access-bm24k\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.040831 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.040787    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-host-cni-netd\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.041298 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.040847    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f1e1cf7f-6a08-455e-8c06-61778f55c020-socket-dir\") pod \"aws-ebs-csi-driver-node-chlrt\" (UID: \"f1e1cf7f-6a08-455e-8c06-61778f55c020\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt"
May 29 13:42:07.041298 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.040876    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f1e1cf7f-6a08-455e-8c06-61778f55c020-registration-dir\") pod \"aws-ebs-csi-driver-node-chlrt\" (UID: \"f1e1cf7f-6a08-455e-8c06-61778f55c020\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt"
May 29 13:42:07.041298 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.040920    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d022803-adf3-4686-8136-a57ab660afad-konnectivity-ca\") pod \"konnectivity-agent-bz56r\" (UID: \"2d022803-adf3-4686-8136-a57ab660afad\") " pod="kube-system/konnectivity-agent-bz56r"
May 29 13:42:07.041298 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.040946    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-etc-modprobe-d\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.041298 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.040968    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-etc-kubernetes\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.041298 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041014    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9wzn2"
May 29 13:42:07.041298 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041079    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-var-lib-kubelet\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.041298 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041102    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ltcdh\""
May 29 13:42:07.041298 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041104    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dda729c3-39e3-45ac-ba23-1628f3874144-tmp\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.041298 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041269    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-var-lib-openvswitch\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.041298 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041297    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f1e1cf7f-6a08-455e-8c06-61778f55c020-etc-selinux\") pod \"aws-ebs-csi-driver-node-chlrt\" (UID: \"f1e1cf7f-6a08-455e-8c06-61778f55c020\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt"
May 29 13:42:07.041633 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041323    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f1e1cf7f-6a08-455e-8c06-61778f55c020-sys-fs\") pod \"aws-ebs-csi-driver-node-chlrt\" (UID: \"f1e1cf7f-6a08-455e-8c06-61778f55c020\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt"
May 29 13:42:07.041633 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041346    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-lib-modules\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.041633 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041368    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dda729c3-39e3-45ac-ba23-1628f3874144-etc-tuned\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.041633 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041392    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-systemd-units\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.041633 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041414    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-host-slash\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.041633 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041436    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-host-run-netns\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.041633 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041458    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 29 13:42:07.041633 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041468    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-run-ovn\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.041633 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041493    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-host-run-ovn-kubernetes\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.041633 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041523    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-ovn-node-metrics-cert\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.041633 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041553    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-etc-systemd\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.042085 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041652    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 29 13:42:07.042085 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041638    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-node-log\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.042085 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041709    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj84r\" (UniqueName: \"kubernetes.io/projected/5ded8d51-ee6a-492b-a1d4-5afefbf29dec-kube-api-access-pj84r\") pod \"node-resolver-xqvkm\" (UID: \"5ded8d51-ee6a-492b-a1d4-5afefbf29dec\") " pod="openshift-dns/node-resolver-xqvkm"
May 29 13:42:07.042085 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041751    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-etc-sysctl-d\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.042085 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041776    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/82e3dbfa-9712-4124-8031-c9f9a0f45abb-host-slash\") pod \"iptables-alerter-k7qwn\" (UID: \"82e3dbfa-9712-4124-8031-c9f9a0f45abb\") " pod="openshift-network-operator/iptables-alerter-k7qwn"
May 29 13:42:07.042085 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041823    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-etc-openvswitch\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.042085 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041863    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-host-cni-bin\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.042085 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041888    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-ovnkube-config\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.042085 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041920    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-ovnkube-script-lib\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.042085 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041949    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrh2l\" (UniqueName: \"kubernetes.io/projected/f1e1cf7f-6a08-455e-8c06-61778f55c020-kube-api-access-lrh2l\") pod \"aws-ebs-csi-driver-node-chlrt\" (UID: \"f1e1cf7f-6a08-455e-8c06-61778f55c020\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt"
May 29 13:42:07.042085 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041970    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5ded8d51-ee6a-492b-a1d4-5afefbf29dec-hosts-file\") pod \"node-resolver-xqvkm\" (UID: \"5ded8d51-ee6a-492b-a1d4-5afefbf29dec\") " pod="openshift-dns/node-resolver-xqvkm"
May 29 13:42:07.042085 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.041992    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ded8d51-ee6a-492b-a1d4-5afefbf29dec-tmp-dir\") pod \"node-resolver-xqvkm\" (UID: \"5ded8d51-ee6a-492b-a1d4-5afefbf29dec\") " pod="openshift-dns/node-resolver-xqvkm"
May 29 13:42:07.042085 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.042029    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-etc-sysctl-conf\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.042085 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.042084    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/82e3dbfa-9712-4124-8031-c9f9a0f45abb-iptables-alerter-script\") pod \"iptables-alerter-k7qwn\" (UID: \"82e3dbfa-9712-4124-8031-c9f9a0f45abb\") " pod="openshift-network-operator/iptables-alerter-k7qwn"
May 29 13:42:07.042713 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.042109    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-run-systemd\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.042713 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.042132    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-run-openvswitch\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.042713 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.042156    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f1e1cf7f-6a08-455e-8c06-61778f55c020-kubelet-dir\") pod \"aws-ebs-csi-driver-node-chlrt\" (UID: \"f1e1cf7f-6a08-455e-8c06-61778f55c020\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt"
May 29 13:42:07.042713 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.042198    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57nkb\" (UniqueName: \"kubernetes.io/projected/82e3dbfa-9712-4124-8031-c9f9a0f45abb-kube-api-access-57nkb\") pod \"iptables-alerter-k7qwn\" (UID: \"82e3dbfa-9712-4124-8031-c9f9a0f45abb\") " pod="openshift-network-operator/iptables-alerter-k7qwn"
May 29 13:42:07.042713 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.042224    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-host-kubelet\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.042713 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.042263    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-env-overrides\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.042713 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.042290    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsq74\" (UniqueName: \"kubernetes.io/projected/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-kube-api-access-lsq74\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.042713 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.042338    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f1e1cf7f-6a08-455e-8c06-61778f55c020-device-dir\") pod \"aws-ebs-csi-driver-node-chlrt\" (UID: \"f1e1cf7f-6a08-455e-8c06-61778f55c020\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt"
May 29 13:42:07.043156 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.043137    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 29 13:42:07.043341 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.043326    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 29 13:42:07.043583 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.043570    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-r6rvh\""
May 29 13:42:07.043640 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.043616    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 29 13:42:07.044506 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.044490    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-46jjq"
May 29 13:42:07.046367 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.046350    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 29 13:42:07.046460 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.046446    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 29 13:42:07.046821 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.046806    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 29 13:42:07.046916 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.046901    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5dvpd\""
May 29 13:42:07.046950 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.046914    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 29 13:42:07.047149 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.047134    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4vtqj"
May 29 13:42:07.049971 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.049954    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:07.050291 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:07.050181    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n48xk" podUID="2a114752-df85-41bc-b828-17308b5311de"
May 29 13:42:07.051234 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.051214    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 29 13:42:07.053151 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.053132    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-q2ncd\""
May 29 13:42:07.053151 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.053144    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:42:07.053300 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.053172    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 29 13:42:07.053300 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:07.053208    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vhmzh" podUID="81333a26-7a37-4277-9753-ebd412eb34aa"
May 29 13:42:07.060346 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.060328    2571 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 29 13:42:07.072877 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.072844    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-28 13:37:06 +0000 UTC" deadline="2027-12-25 18:25:51.226227882 +0000 UTC"
May 29 13:42:07.072993 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.072885    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13804h43m44.153350582s"
May 29 13:42:07.087576 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.087553    2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-skrt9"
May 29 13:42:07.096135 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.096082    2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-skrt9"
May 29 13:42:07.138034 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.138005    2571 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 29 13:42:07.142725 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.142707    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-run-systemd\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.142785 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.142734    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f1e1cf7f-6a08-455e-8c06-61778f55c020-kubelet-dir\") pod \"aws-ebs-csi-driver-node-chlrt\" (UID: \"f1e1cf7f-6a08-455e-8c06-61778f55c020\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt"
May 29 13:42:07.142785 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.142757    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfcwf\" (UniqueName: \"kubernetes.io/projected/1b2d1d27-e735-4a58-a93f-5d0bf642aa78-kube-api-access-qfcwf\") pod \"multus-additional-cni-plugins-4vtqj\" (UID: \"1b2d1d27-e735-4a58-a93f-5d0bf642aa78\") " pod="openshift-multus/multus-additional-cni-plugins-4vtqj"
May 29 13:42:07.142849 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.142786    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59xl8\" (UniqueName: \"kubernetes.io/projected/af79b7f2-4958-48fc-ad8d-ef0c731dc642-kube-api-access-59xl8\") pod \"node-ca-9wzn2\" (UID: \"af79b7f2-4958-48fc-ad8d-ef0c731dc642\") " pod="openshift-image-registry/node-ca-9wzn2"
May 29 13:42:07.142849 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.142806    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-cni-binary-copy\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.142849 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.142822    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-env-overrides\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.142849 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.142831    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f1e1cf7f-6a08-455e-8c06-61778f55c020-kubelet-dir\") pod \"aws-ebs-csi-driver-node-chlrt\" (UID: \"f1e1cf7f-6a08-455e-8c06-61778f55c020\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt"
May 29 13:42:07.142849 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.142841    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lsq74\" (UniqueName: \"kubernetes.io/projected/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-kube-api-access-lsq74\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.143006 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.142881    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-run-systemd\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.143045 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143008    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d022803-adf3-4686-8136-a57ab660afad-agent-certs\") pod \"konnectivity-agent-bz56r\" (UID: \"2d022803-adf3-4686-8136-a57ab660afad\") " pod="kube-system/konnectivity-agent-bz56r"
May 29 13:42:07.143099 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143042    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-etc-sysconfig\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.143099 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143083    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-run\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.143170 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143105    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-host\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.143170 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143128    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.143170 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143159    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b2d1d27-e735-4a58-a93f-5d0bf642aa78-os-release\") pod \"multus-additional-cni-plugins-4vtqj\" (UID: \"1b2d1d27-e735-4a58-a93f-5d0bf642aa78\") " pod="openshift-multus/multus-additional-cni-plugins-4vtqj"
May 29 13:42:07.143170 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143163    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-etc-sysconfig\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.143337 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143177    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-run\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.143337 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143216    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-host\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.143337 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143222    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.143337 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143271    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-host-run-k8s-cni-cncf-io\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.143337 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143302    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f1e1cf7f-6a08-455e-8c06-61778f55c020-socket-dir\") pod \"aws-ebs-csi-driver-node-chlrt\" (UID: \"f1e1cf7f-6a08-455e-8c06-61778f55c020\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt"
May 29 13:42:07.143337 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143336    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f1e1cf7f-6a08-455e-8c06-61778f55c020-registration-dir\") pod \"aws-ebs-csi-driver-node-chlrt\" (UID: \"f1e1cf7f-6a08-455e-8c06-61778f55c020\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt"
May 29 13:42:07.143563 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143366    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-cnibin\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.143563 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143334    2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 29 13:42:07.143563 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143383    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f1e1cf7f-6a08-455e-8c06-61778f55c020-registration-dir\") pod \"aws-ebs-csi-driver-node-chlrt\" (UID: \"f1e1cf7f-6a08-455e-8c06-61778f55c020\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt"
May 29 13:42:07.143563 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143392    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-host-var-lib-kubelet\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.143563 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143416    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-multus-daemon-config\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.143563 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143439    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f1e1cf7f-6a08-455e-8c06-61778f55c020-socket-dir\") pod \"aws-ebs-csi-driver-node-chlrt\" (UID: \"f1e1cf7f-6a08-455e-8c06-61778f55c020\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt"
May 29 13:42:07.143563 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143440    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b2d1d27-e735-4a58-a93f-5d0bf642aa78-system-cni-dir\") pod \"multus-additional-cni-plugins-4vtqj\" (UID: \"1b2d1d27-e735-4a58-a93f-5d0bf642aa78\") " pod="openshift-multus/multus-additional-cni-plugins-4vtqj"
May 29 13:42:07.143563 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143482    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-etc-kubernetes\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.143563 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143516    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-var-lib-kubelet\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.143947 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143560    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b2d1d27-e735-4a58-a93f-5d0bf642aa78-cni-binary-copy\") pod \"multus-additional-cni-plugins-4vtqj\" (UID: \"1b2d1d27-e735-4a58-a93f-5d0bf642aa78\") " pod="openshift-multus/multus-additional-cni-plugins-4vtqj"
May 29 13:42:07.143947 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143583    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-etc-kubernetes\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.143947 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143587    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-var-lib-kubelet\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.143947 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143598    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lrh2l\" (UniqueName: \"kubernetes.io/projected/f1e1cf7f-6a08-455e-8c06-61778f55c020-kube-api-access-lrh2l\") pod \"aws-ebs-csi-driver-node-chlrt\" (UID: \"f1e1cf7f-6a08-455e-8c06-61778f55c020\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt"
May 29 13:42:07.143947 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143629    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-systemd-units\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.143947 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143649    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-host-slash\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.143947 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143670    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b2d1d27-e735-4a58-a93f-5d0bf642aa78-cnibin\") pod \"multus-additional-cni-plugins-4vtqj\" (UID: \"1b2d1d27-e735-4a58-a93f-5d0bf642aa78\") " pod="openshift-multus/multus-additional-cni-plugins-4vtqj"
May 29 13:42:07.143947 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143694    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/af79b7f2-4958-48fc-ad8d-ef0c731dc642-serviceca\") pod \"node-ca-9wzn2\" (UID: \"af79b7f2-4958-48fc-ad8d-ef0c731dc642\") " pod="openshift-image-registry/node-ca-9wzn2"
May 29 13:42:07.143947 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143700    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-systemd-units\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.143947 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143717    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-multus-cni-dir\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.143947 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143717    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-host-slash\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.143947 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143752    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-node-log\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.143947 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143779    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b2d1d27-e735-4a58-a93f-5d0bf642aa78-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4vtqj\" (UID: \"1b2d1d27-e735-4a58-a93f-5d0bf642aa78\") " pod="openshift-multus/multus-additional-cni-plugins-4vtqj"
May 29 13:42:07.143947 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143806    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b2d1d27-e735-4a58-a93f-5d0bf642aa78-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4vtqj\" (UID: \"1b2d1d27-e735-4a58-a93f-5d0bf642aa78\") " pod="openshift-multus/multus-additional-cni-plugins-4vtqj"
May 29 13:42:07.143947 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143846    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-node-log\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.143947 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143858    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b2d1d27-e735-4a58-a93f-5d0bf642aa78-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4vtqj\" (UID: \"1b2d1d27-e735-4a58-a93f-5d0bf642aa78\") " pod="openshift-multus/multus-additional-cni-plugins-4vtqj"
May 29 13:42:07.143947 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143887    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-host-run-multus-certs\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.144709 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143901    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-env-overrides\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.144709 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143927    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-etc-sysctl-d\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.144709 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143970    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-host-cni-bin\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.144709 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.143996    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-ovnkube-script-lib\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.144709 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144014    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlmkx\" (UniqueName: \"kubernetes.io/projected/81333a26-7a37-4277-9753-ebd412eb34aa-kube-api-access-nlmkx\") pod \"network-check-target-vhmzh\" (UID: \"81333a26-7a37-4277-9753-ebd412eb34aa\") " pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:42:07.144709 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144033    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-os-release\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.144709 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144036    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-etc-sysctl-d\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.144709 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144065    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-hostroot\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.144709 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144097    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5ded8d51-ee6a-492b-a1d4-5afefbf29dec-hosts-file\") pod \"node-resolver-xqvkm\" (UID: \"5ded8d51-ee6a-492b-a1d4-5afefbf29dec\") " pod="openshift-dns/node-resolver-xqvkm"
May 29 13:42:07.144709 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144131    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ded8d51-ee6a-492b-a1d4-5afefbf29dec-tmp-dir\") pod \"node-resolver-xqvkm\" (UID: \"5ded8d51-ee6a-492b-a1d4-5afefbf29dec\") " pod="openshift-dns/node-resolver-xqvkm"
May 29 13:42:07.144709 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144235    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/82e3dbfa-9712-4124-8031-c9f9a0f45abb-iptables-alerter-script\") pod \"iptables-alerter-k7qwn\" (UID: \"82e3dbfa-9712-4124-8031-c9f9a0f45abb\") " pod="openshift-network-operator/iptables-alerter-k7qwn"
May 29 13:42:07.144709 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144244    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5ded8d51-ee6a-492b-a1d4-5afefbf29dec-hosts-file\") pod \"node-resolver-xqvkm\" (UID: \"5ded8d51-ee6a-492b-a1d4-5afefbf29dec\") " pod="openshift-dns/node-resolver-xqvkm"
May 29 13:42:07.144709 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144264    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-host-cni-bin\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.144709 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144294    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-run-openvswitch\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.144709 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144310    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-57nkb\" (UniqueName: \"kubernetes.io/projected/82e3dbfa-9712-4124-8031-c9f9a0f45abb-kube-api-access-57nkb\") pod \"iptables-alerter-k7qwn\" (UID: \"82e3dbfa-9712-4124-8031-c9f9a0f45abb\") " pod="openshift-network-operator/iptables-alerter-k7qwn"
May 29 13:42:07.144709 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144341    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-host-kubelet\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.144709 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144346    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-run-openvswitch\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.145387 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144435    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ded8d51-ee6a-492b-a1d4-5afefbf29dec-tmp-dir\") pod \"node-resolver-xqvkm\" (UID: \"5ded8d51-ee6a-492b-a1d4-5afefbf29dec\") " pod="openshift-dns/node-resolver-xqvkm"
May 29 13:42:07.145387 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144466    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-ovnkube-script-lib\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.145387 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144468    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-host-kubelet\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.145387 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144490    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f1e1cf7f-6a08-455e-8c06-61778f55c020-device-dir\") pod \"aws-ebs-csi-driver-node-chlrt\" (UID: \"f1e1cf7f-6a08-455e-8c06-61778f55c020\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt"
May 29 13:42:07.145387 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144596    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af79b7f2-4958-48fc-ad8d-ef0c731dc642-host\") pod \"node-ca-9wzn2\" (UID: \"af79b7f2-4958-48fc-ad8d-ef0c731dc642\") " pod="openshift-image-registry/node-ca-9wzn2"
May 29 13:42:07.145387 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144613    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f1e1cf7f-6a08-455e-8c06-61778f55c020-device-dir\") pod \"aws-ebs-csi-driver-node-chlrt\" (UID: \"f1e1cf7f-6a08-455e-8c06-61778f55c020\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt"
May 29 13:42:07.145387 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144652    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-host-var-lib-cni-multus\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.145387 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144689    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-log-socket\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.145387 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144715    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t78rs\" (UniqueName: \"kubernetes.io/projected/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-kube-api-access-t78rs\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.145387 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144741    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-sys\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.145387 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144760    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-log-socket\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.145387 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144766    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bm24k\" (UniqueName: \"kubernetes.io/projected/dda729c3-39e3-45ac-ba23-1628f3874144-kube-api-access-bm24k\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.145387 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144821    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-sys\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.145387 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144826    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-host-cni-netd\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.145387 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144830    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/82e3dbfa-9712-4124-8031-c9f9a0f45abb-iptables-alerter-script\") pod \"iptables-alerter-k7qwn\" (UID: \"82e3dbfa-9712-4124-8031-c9f9a0f45abb\") " pod="openshift-network-operator/iptables-alerter-k7qwn"
May 29 13:42:07.145387 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144861    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-host-cni-netd\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.145387 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144876    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs\") pod \"network-metrics-daemon-n48xk\" (UID: \"2a114752-df85-41bc-b828-17308b5311de\") " pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:07.146104 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144902    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk2fh\" (UniqueName: \"kubernetes.io/projected/2a114752-df85-41bc-b828-17308b5311de-kube-api-access-jk2fh\") pod \"network-metrics-daemon-n48xk\" (UID: \"2a114752-df85-41bc-b828-17308b5311de\") " pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:07.146104 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144935    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-system-cni-dir\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.146104 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.144978    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-host-var-lib-cni-bin\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.146104 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145006    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-etc-kubernetes\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.146104 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145024    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d022803-adf3-4686-8136-a57ab660afad-konnectivity-ca\") pod \"konnectivity-agent-bz56r\" (UID: \"2d022803-adf3-4686-8136-a57ab660afad\") " pod="kube-system/konnectivity-agent-bz56r"
May 29 13:42:07.146104 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145040    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-etc-modprobe-d\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.146104 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145077    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dda729c3-39e3-45ac-ba23-1628f3874144-tmp\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.146104 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145147    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-etc-modprobe-d\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.146104 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145184    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-var-lib-openvswitch\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.146104 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145218    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f1e1cf7f-6a08-455e-8c06-61778f55c020-etc-selinux\") pod \"aws-ebs-csi-driver-node-chlrt\" (UID: \"f1e1cf7f-6a08-455e-8c06-61778f55c020\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt"
May 29 13:42:07.146104 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145245    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f1e1cf7f-6a08-455e-8c06-61778f55c020-sys-fs\") pod \"aws-ebs-csi-driver-node-chlrt\" (UID: \"f1e1cf7f-6a08-455e-8c06-61778f55c020\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt"
May 29 13:42:07.146104 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145263    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-var-lib-openvswitch\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.146104 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145273    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-multus-socket-dir-parent\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.146104 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145300    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-lib-modules\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.146104 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145302    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f1e1cf7f-6a08-455e-8c06-61778f55c020-sys-fs\") pod \"aws-ebs-csi-driver-node-chlrt\" (UID: \"f1e1cf7f-6a08-455e-8c06-61778f55c020\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt"
May 29 13:42:07.146104 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145325    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dda729c3-39e3-45ac-ba23-1628f3874144-etc-tuned\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.146104 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145346    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f1e1cf7f-6a08-455e-8c06-61778f55c020-etc-selinux\") pod \"aws-ebs-csi-driver-node-chlrt\" (UID: \"f1e1cf7f-6a08-455e-8c06-61778f55c020\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt"
May 29 13:42:07.146865 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145358    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-host-run-netns\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.146865 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145395    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-lib-modules\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.146865 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145396    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-run-ovn\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.146865 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145411    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-host-run-netns\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.146865 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145426    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-run-ovn\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.146865 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145433    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-host-run-ovn-kubernetes\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.146865 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145470    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-ovn-node-metrics-cert\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.146865 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145475    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-host-run-ovn-kubernetes\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.146865 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145481    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d022803-adf3-4686-8136-a57ab660afad-konnectivity-ca\") pod \"konnectivity-agent-bz56r\" (UID: \"2d022803-adf3-4686-8136-a57ab660afad\") " pod="kube-system/konnectivity-agent-bz56r"
May 29 13:42:07.146865 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145510    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-etc-systemd\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.146865 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145593    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-etc-systemd\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.146865 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145602    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-host-run-netns\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.146865 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145641    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pj84r\" (UniqueName: \"kubernetes.io/projected/5ded8d51-ee6a-492b-a1d4-5afefbf29dec-kube-api-access-pj84r\") pod \"node-resolver-xqvkm\" (UID: \"5ded8d51-ee6a-492b-a1d4-5afefbf29dec\") " pod="openshift-dns/node-resolver-xqvkm"
May 29 13:42:07.146865 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145666    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/82e3dbfa-9712-4124-8031-c9f9a0f45abb-host-slash\") pod \"iptables-alerter-k7qwn\" (UID: \"82e3dbfa-9712-4124-8031-c9f9a0f45abb\") " pod="openshift-network-operator/iptables-alerter-k7qwn"
May 29 13:42:07.146865 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145691    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-etc-openvswitch\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.146865 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145714    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-ovnkube-config\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.146865 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145717    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/82e3dbfa-9712-4124-8031-c9f9a0f45abb-host-slash\") pod \"iptables-alerter-k7qwn\" (UID: \"82e3dbfa-9712-4124-8031-c9f9a0f45abb\") " pod="openshift-network-operator/iptables-alerter-k7qwn"
May 29 13:42:07.147722 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145751    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-multus-conf-dir\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.147722 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145787    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-etc-sysctl-conf\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.147722 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145910    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-etc-openvswitch\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.147722 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.145941    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dda729c3-39e3-45ac-ba23-1628f3874144-etc-sysctl-conf\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.147722 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.146271    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-ovnkube-config\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.147722 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.147282    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dda729c3-39e3-45ac-ba23-1628f3874144-tmp\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.147722 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.147431    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d022803-adf3-4686-8136-a57ab660afad-agent-certs\") pod \"konnectivity-agent-bz56r\" (UID: \"2d022803-adf3-4686-8136-a57ab660afad\") " pod="kube-system/konnectivity-agent-bz56r"
May 29 13:42:07.147722 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.147525    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dda729c3-39e3-45ac-ba23-1628f3874144-etc-tuned\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.147722 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.147630    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-ovn-node-metrics-cert\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.164640 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.164606    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57nkb\" (UniqueName: \"kubernetes.io/projected/82e3dbfa-9712-4124-8031-c9f9a0f45abb-kube-api-access-57nkb\") pod \"iptables-alerter-k7qwn\" (UID: \"82e3dbfa-9712-4124-8031-c9f9a0f45abb\") " pod="openshift-network-operator/iptables-alerter-k7qwn"
May 29 13:42:07.164763 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.164742    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm24k\" (UniqueName: \"kubernetes.io/projected/dda729c3-39e3-45ac-ba23-1628f3874144-kube-api-access-bm24k\") pod \"tuned-xqvgg\" (UID: \"dda729c3-39e3-45ac-ba23-1628f3874144\") " pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.164813 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.164759    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj84r\" (UniqueName: \"kubernetes.io/projected/5ded8d51-ee6a-492b-a1d4-5afefbf29dec-kube-api-access-pj84r\") pod \"node-resolver-xqvkm\" (UID: \"5ded8d51-ee6a-492b-a1d4-5afefbf29dec\") " pod="openshift-dns/node-resolver-xqvkm"
May 29 13:42:07.164990 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.164966    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrh2l\" (UniqueName: \"kubernetes.io/projected/f1e1cf7f-6a08-455e-8c06-61778f55c020-kube-api-access-lrh2l\") pod \"aws-ebs-csi-driver-node-chlrt\" (UID: \"f1e1cf7f-6a08-455e-8c06-61778f55c020\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt"
May 29 13:42:07.165812 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.165792    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsq74\" (UniqueName: \"kubernetes.io/projected/4f4cffa7-97f7-4149-8118-e3fd4edc15a7-kube-api-access-lsq74\") pod \"ovnkube-node-p2drl\" (UID: \"4f4cffa7-97f7-4149-8118-e3fd4edc15a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.210224 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:07.210187    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf64b7f1c8724ca5ea45aa00abb85e2.slice/crio-c189628b77a28ea5353bdfb6bc3861048acb0a4e6fd8580a4d9af21b95cba256 WatchSource:0}: Error finding container c189628b77a28ea5353bdfb6bc3861048acb0a4e6fd8580a4d9af21b95cba256: Status 404 returned error can't find the container with id c189628b77a28ea5353bdfb6bc3861048acb0a4e6fd8580a4d9af21b95cba256
May 29 13:42:07.210434 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:07.210415    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36931e78e4f753d131caf513c9d0151.slice/crio-4da7fbac9ce45497e4a3ec4d0b21356762de7ccdc1a88abbb71fbdce4ffcd9e1 WatchSource:0}: Error finding container 4da7fbac9ce45497e4a3ec4d0b21356762de7ccdc1a88abbb71fbdce4ffcd9e1: Status 404 returned error can't find the container with id 4da7fbac9ce45497e4a3ec4d0b21356762de7ccdc1a88abbb71fbdce4ffcd9e1
May 29 13:42:07.213094 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.213075    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 29 13:42:07.246471 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246443    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jk2fh\" (UniqueName: \"kubernetes.io/projected/2a114752-df85-41bc-b828-17308b5311de-kube-api-access-jk2fh\") pod \"network-metrics-daemon-n48xk\" (UID: \"2a114752-df85-41bc-b828-17308b5311de\") " pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:07.246471 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246474    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-system-cni-dir\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.246711 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246489    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-host-var-lib-cni-bin\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.246711 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246503    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-etc-kubernetes\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.246711 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246520    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-multus-socket-dir-parent\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.246711 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246539    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-host-run-netns\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.246711 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246563    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-multus-conf-dir\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.246711 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246564    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-host-var-lib-cni-bin\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.246711 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246588    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qfcwf\" (UniqueName: \"kubernetes.io/projected/1b2d1d27-e735-4a58-a93f-5d0bf642aa78-kube-api-access-qfcwf\") pod \"multus-additional-cni-plugins-4vtqj\" (UID: \"1b2d1d27-e735-4a58-a93f-5d0bf642aa78\") " pod="openshift-multus/multus-additional-cni-plugins-4vtqj"
May 29 13:42:07.246711 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246585    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-etc-kubernetes\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.246711 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246611    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-host-run-netns\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.246711 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246619    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-multus-socket-dir-parent\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.246711 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246624    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-59xl8\" (UniqueName: \"kubernetes.io/projected/af79b7f2-4958-48fc-ad8d-ef0c731dc642-kube-api-access-59xl8\") pod \"node-ca-9wzn2\" (UID: \"af79b7f2-4958-48fc-ad8d-ef0c731dc642\") " pod="openshift-image-registry/node-ca-9wzn2"
May 29 13:42:07.246711 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246646    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-multus-conf-dir\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.246711 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246664    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-cni-binary-copy\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.247294 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246728    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b2d1d27-e735-4a58-a93f-5d0bf642aa78-os-release\") pod \"multus-additional-cni-plugins-4vtqj\" (UID: \"1b2d1d27-e735-4a58-a93f-5d0bf642aa78\") " pod="openshift-multus/multus-additional-cni-plugins-4vtqj"
May 29 13:42:07.247294 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246758    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-host-run-k8s-cni-cncf-io\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.247294 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246757    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-system-cni-dir\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.247294 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246786    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-cnibin\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.247294 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246806    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-host-run-k8s-cni-cncf-io\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.247294 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246810    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1b2d1d27-e735-4a58-a93f-5d0bf642aa78-os-release\") pod \"multus-additional-cni-plugins-4vtqj\" (UID: \"1b2d1d27-e735-4a58-a93f-5d0bf642aa78\") " pod="openshift-multus/multus-additional-cni-plugins-4vtqj"
May 29 13:42:07.247294 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246812    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-host-var-lib-kubelet\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.247294 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246844    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-cnibin\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.247294 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246855    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-host-var-lib-kubelet\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.247294 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246872    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-multus-daemon-config\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.247294 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246900    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b2d1d27-e735-4a58-a93f-5d0bf642aa78-system-cni-dir\") pod \"multus-additional-cni-plugins-4vtqj\" (UID: \"1b2d1d27-e735-4a58-a93f-5d0bf642aa78\") " pod="openshift-multus/multus-additional-cni-plugins-4vtqj"
May 29 13:42:07.247294 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246920    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b2d1d27-e735-4a58-a93f-5d0bf642aa78-cni-binary-copy\") pod \"multus-additional-cni-plugins-4vtqj\" (UID: \"1b2d1d27-e735-4a58-a93f-5d0bf642aa78\") " pod="openshift-multus/multus-additional-cni-plugins-4vtqj"
May 29 13:42:07.247294 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246950    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b2d1d27-e735-4a58-a93f-5d0bf642aa78-cnibin\") pod \"multus-additional-cni-plugins-4vtqj\" (UID: \"1b2d1d27-e735-4a58-a93f-5d0bf642aa78\") " pod="openshift-multus/multus-additional-cni-plugins-4vtqj"
May 29 13:42:07.247294 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246965    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1b2d1d27-e735-4a58-a93f-5d0bf642aa78-system-cni-dir\") pod \"multus-additional-cni-plugins-4vtqj\" (UID: \"1b2d1d27-e735-4a58-a93f-5d0bf642aa78\") " pod="openshift-multus/multus-additional-cni-plugins-4vtqj"
May 29 13:42:07.247294 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.246994    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1b2d1d27-e735-4a58-a93f-5d0bf642aa78-cnibin\") pod \"multus-additional-cni-plugins-4vtqj\" (UID: \"1b2d1d27-e735-4a58-a93f-5d0bf642aa78\") " pod="openshift-multus/multus-additional-cni-plugins-4vtqj"
May 29 13:42:07.247294 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.247014    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/af79b7f2-4958-48fc-ad8d-ef0c731dc642-serviceca\") pod \"node-ca-9wzn2\" (UID: \"af79b7f2-4958-48fc-ad8d-ef0c731dc642\") " pod="openshift-image-registry/node-ca-9wzn2"
May 29 13:42:07.247294 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.247030    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-multus-cni-dir\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.247880 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.247074    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b2d1d27-e735-4a58-a93f-5d0bf642aa78-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4vtqj\" (UID: \"1b2d1d27-e735-4a58-a93f-5d0bf642aa78\") " pod="openshift-multus/multus-additional-cni-plugins-4vtqj"
May 29 13:42:07.247880 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.247099    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b2d1d27-e735-4a58-a93f-5d0bf642aa78-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4vtqj\" (UID: \"1b2d1d27-e735-4a58-a93f-5d0bf642aa78\") " pod="openshift-multus/multus-additional-cni-plugins-4vtqj"
May 29 13:42:07.247880 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.247118    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-multus-cni-dir\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.247880 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.247125    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b2d1d27-e735-4a58-a93f-5d0bf642aa78-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4vtqj\" (UID: \"1b2d1d27-e735-4a58-a93f-5d0bf642aa78\") " pod="openshift-multus/multus-additional-cni-plugins-4vtqj"
May 29 13:42:07.247880 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.247154    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-host-run-multus-certs\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.247880 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.247181    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nlmkx\" (UniqueName: \"kubernetes.io/projected/81333a26-7a37-4277-9753-ebd412eb34aa-kube-api-access-nlmkx\") pod \"network-check-target-vhmzh\" (UID: \"81333a26-7a37-4277-9753-ebd412eb34aa\") " pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:42:07.247880 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.247205    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-os-release\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.247880 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.247229    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-hostroot\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.247880 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.247247    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-host-run-multus-certs\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.247880 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.247259    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af79b7f2-4958-48fc-ad8d-ef0c731dc642-host\") pod \"node-ca-9wzn2\" (UID: \"af79b7f2-4958-48fc-ad8d-ef0c731dc642\") " pod="openshift-image-registry/node-ca-9wzn2"
May 29 13:42:07.247880 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.247283    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-host-var-lib-cni-multus\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.247880 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.247301    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-cni-binary-copy\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.247880 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.247311    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t78rs\" (UniqueName: \"kubernetes.io/projected/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-kube-api-access-t78rs\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.247880 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.247338    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs\") pod \"network-metrics-daemon-n48xk\" (UID: \"2a114752-df85-41bc-b828-17308b5311de\") " pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:07.247880 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.247367    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-host-var-lib-cni-multus\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.247880 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.247283    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1b2d1d27-e735-4a58-a93f-5d0bf642aa78-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4vtqj\" (UID: \"1b2d1d27-e735-4a58-a93f-5d0bf642aa78\") " pod="openshift-multus/multus-additional-cni-plugins-4vtqj"
May 29 13:42:07.247880 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.247403    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-multus-daemon-config\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.247880 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:07.247445    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 13:42:07.248382 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.247434    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-os-release\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.248382 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.247472    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/af79b7f2-4958-48fc-ad8d-ef0c731dc642-serviceca\") pod \"node-ca-9wzn2\" (UID: \"af79b7f2-4958-48fc-ad8d-ef0c731dc642\") " pod="openshift-image-registry/node-ca-9wzn2"
May 29 13:42:07.248382 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.247503    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-hostroot\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.248382 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:07.247521    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs podName:2a114752-df85-41bc-b828-17308b5311de nodeName:}" failed. No retries permitted until 2026-05-29 13:42:07.747499265 +0000 UTC m=+2.214569347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs") pod "network-metrics-daemon-n48xk" (UID: "2a114752-df85-41bc-b828-17308b5311de") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 13:42:07.248382 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.247529    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af79b7f2-4958-48fc-ad8d-ef0c731dc642-host\") pod \"node-ca-9wzn2\" (UID: \"af79b7f2-4958-48fc-ad8d-ef0c731dc642\") " pod="openshift-image-registry/node-ca-9wzn2"
May 29 13:42:07.248382 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.247563    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1b2d1d27-e735-4a58-a93f-5d0bf642aa78-cni-binary-copy\") pod \"multus-additional-cni-plugins-4vtqj\" (UID: \"1b2d1d27-e735-4a58-a93f-5d0bf642aa78\") " pod="openshift-multus/multus-additional-cni-plugins-4vtqj"
May 29 13:42:07.248382 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.247837    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1b2d1d27-e735-4a58-a93f-5d0bf642aa78-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4vtqj\" (UID: \"1b2d1d27-e735-4a58-a93f-5d0bf642aa78\") " pod="openshift-multus/multus-additional-cni-plugins-4vtqj"
May 29 13:42:07.248603 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.248452    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1b2d1d27-e735-4a58-a93f-5d0bf642aa78-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4vtqj\" (UID: \"1b2d1d27-e735-4a58-a93f-5d0bf642aa78\") " pod="openshift-multus/multus-additional-cni-plugins-4vtqj"
May 29 13:42:07.253276 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:07.253250    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 13:42:07.253276 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:07.253273    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 13:42:07.253423 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:07.253284    2571 projected.go:194] Error preparing data for projected volume kube-api-access-nlmkx for pod openshift-network-diagnostics/network-check-target-vhmzh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 13:42:07.253423 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:07.253336    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81333a26-7a37-4277-9753-ebd412eb34aa-kube-api-access-nlmkx podName:81333a26-7a37-4277-9753-ebd412eb34aa nodeName:}" failed. No retries permitted until 2026-05-29 13:42:07.753324958 +0000 UTC m=+2.220395031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nlmkx" (UniqueName: "kubernetes.io/projected/81333a26-7a37-4277-9753-ebd412eb34aa-kube-api-access-nlmkx") pod "network-check-target-vhmzh" (UID: "81333a26-7a37-4277-9753-ebd412eb34aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 13:42:07.255440 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.255414    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59xl8\" (UniqueName: \"kubernetes.io/projected/af79b7f2-4958-48fc-ad8d-ef0c731dc642-kube-api-access-59xl8\") pod \"node-ca-9wzn2\" (UID: \"af79b7f2-4958-48fc-ad8d-ef0c731dc642\") " pod="openshift-image-registry/node-ca-9wzn2"
May 29 13:42:07.255556 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.255536    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk2fh\" (UniqueName: \"kubernetes.io/projected/2a114752-df85-41bc-b828-17308b5311de-kube-api-access-jk2fh\") pod \"network-metrics-daemon-n48xk\" (UID: \"2a114752-df85-41bc-b828-17308b5311de\") " pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:07.255891 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.255872    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t78rs\" (UniqueName: \"kubernetes.io/projected/758a6d2f-ef64-4995-9a3b-d66b09cc98f1-kube-api-access-t78rs\") pod \"multus-46jjq\" (UID: \"758a6d2f-ef64-4995-9a3b-d66b09cc98f1\") " pod="openshift-multus/multus-46jjq"
May 29 13:42:07.256109 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.256088    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfcwf\" (UniqueName: \"kubernetes.io/projected/1b2d1d27-e735-4a58-a93f-5d0bf642aa78-kube-api-access-qfcwf\") pod \"multus-additional-cni-plugins-4vtqj\" (UID: \"1b2d1d27-e735-4a58-a93f-5d0bf642aa78\") " pod="openshift-multus/multus-additional-cni-plugins-4vtqj"
May 29 13:42:07.355736 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.355666    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bz56r"
May 29 13:42:07.362508 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:07.362481    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d022803_adf3_4686_8136_a57ab660afad.slice/crio-053fbd49c25f326aae6f30de9a90bceeccd109a6a15a5e6ca48742e75f3b8f5a WatchSource:0}: Error finding container 053fbd49c25f326aae6f30de9a90bceeccd109a6a15a5e6ca48742e75f3b8f5a: Status 404 returned error can't find the container with id 053fbd49c25f326aae6f30de9a90bceeccd109a6a15a5e6ca48742e75f3b8f5a
May 29 13:42:07.383536 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.383504    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xqvgg"
May 29 13:42:07.389461 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:07.389423    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda729c3_39e3_45ac_ba23_1628f3874144.slice/crio-168e8b98541891560ddccc863bd3d418d1a4485de7325f33b085e20bbbbe3c24 WatchSource:0}: Error finding container 168e8b98541891560ddccc863bd3d418d1a4485de7325f33b085e20bbbbe3c24: Status 404 returned error can't find the container with id 168e8b98541891560ddccc863bd3d418d1a4485de7325f33b085e20bbbbe3c24
May 29 13:42:07.402142 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.402119    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k7qwn"
May 29 13:42:07.404148 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.404128    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:07.409423 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:07.409402    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e3dbfa_9712_4124_8031_c9f9a0f45abb.slice/crio-496efc1505a5dd8ad61fe2f38d5bb636a0c10aa0c7cce8dd0a71fd448e96aa01 WatchSource:0}: Error finding container 496efc1505a5dd8ad61fe2f38d5bb636a0c10aa0c7cce8dd0a71fd448e96aa01: Status 404 returned error can't find the container with id 496efc1505a5dd8ad61fe2f38d5bb636a0c10aa0c7cce8dd0a71fd448e96aa01
May 29 13:42:07.411463 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:07.411445    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f4cffa7_97f7_4149_8118_e3fd4edc15a7.slice/crio-da4c7165d3ee76c44161fa16a31f91534acf075cc8b3d5632279892f551b7157 WatchSource:0}: Error finding container da4c7165d3ee76c44161fa16a31f91534acf075cc8b3d5632279892f551b7157: Status 404 returned error can't find the container with id da4c7165d3ee76c44161fa16a31f91534acf075cc8b3d5632279892f551b7157
May 29 13:42:07.417536 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.417517    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt"
May 29 13:42:07.423359 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:07.423336    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e1cf7f_6a08_455e_8c06_61778f55c020.slice/crio-a127ff71dc0326375a6e61e9f58faf8be8c8c06bd4c64f38d1b74657b1a9868c WatchSource:0}: Error finding container a127ff71dc0326375a6e61e9f58faf8be8c8c06bd4c64f38d1b74657b1a9868c: Status 404 returned error can't find the container with id a127ff71dc0326375a6e61e9f58faf8be8c8c06bd4c64f38d1b74657b1a9868c
May 29 13:42:07.430597 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.430578    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xqvkm"
May 29 13:42:07.437265 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:07.437242    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ded8d51_ee6a_492b_a1d4_5afefbf29dec.slice/crio-0ca865904fc75e969736f94af3ed83841a70ff2eb0dfc58c05809f76ff8f7085 WatchSource:0}: Error finding container 0ca865904fc75e969736f94af3ed83841a70ff2eb0dfc58c05809f76ff8f7085: Status 404 returned error can't find the container with id 0ca865904fc75e969736f94af3ed83841a70ff2eb0dfc58c05809f76ff8f7085
May 29 13:42:07.448491 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.448469    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9wzn2"
May 29 13:42:07.455135 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:07.455115    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf79b7f2_4958_48fc_ad8d_ef0c731dc642.slice/crio-a75fc7cc7a7fadeff9b04996d30fb4a91e2a67eec4e3580aa9323b3aa5f5ae65 WatchSource:0}: Error finding container a75fc7cc7a7fadeff9b04996d30fb4a91e2a67eec4e3580aa9323b3aa5f5ae65: Status 404 returned error can't find the container with id a75fc7cc7a7fadeff9b04996d30fb4a91e2a67eec4e3580aa9323b3aa5f5ae65
May 29 13:42:07.460650 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.460632    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-46jjq"
May 29 13:42:07.466553 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:07.466526    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod758a6d2f_ef64_4995_9a3b_d66b09cc98f1.slice/crio-e07494f6fa3fe4ed9395e41ffd703f23161b72cee033b3e82b55c984f2aefb41 WatchSource:0}: Error finding container e07494f6fa3fe4ed9395e41ffd703f23161b72cee033b3e82b55c984f2aefb41: Status 404 returned error can't find the container with id e07494f6fa3fe4ed9395e41ffd703f23161b72cee033b3e82b55c984f2aefb41
May 29 13:42:07.478954 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.478933    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4vtqj"
May 29 13:42:07.486596 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:07.486576    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b2d1d27_e735_4a58_a93f_5d0bf642aa78.slice/crio-5c383e5aad58217e2fd90d298754491f76f540472044a38cb6cfc4196dcf0b0e WatchSource:0}: Error finding container 5c383e5aad58217e2fd90d298754491f76f540472044a38cb6cfc4196dcf0b0e: Status 404 returned error can't find the container with id 5c383e5aad58217e2fd90d298754491f76f540472044a38cb6cfc4196dcf0b0e
May 29 13:42:07.563848 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.563820    2571 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 29 13:42:07.750456 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.750373    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs\") pod \"network-metrics-daemon-n48xk\" (UID: \"2a114752-df85-41bc-b828-17308b5311de\") " pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:07.750697 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:07.750519    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 13:42:07.750697 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:07.750592    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs podName:2a114752-df85-41bc-b828-17308b5311de nodeName:}" failed. No retries permitted until 2026-05-29 13:42:08.750572812 +0000 UTC m=+3.217642874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs") pod "network-metrics-daemon-n48xk" (UID: "2a114752-df85-41bc-b828-17308b5311de") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 13:42:07.851801 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:07.851754    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nlmkx\" (UniqueName: \"kubernetes.io/projected/81333a26-7a37-4277-9753-ebd412eb34aa-kube-api-access-nlmkx\") pod \"network-check-target-vhmzh\" (UID: \"81333a26-7a37-4277-9753-ebd412eb34aa\") " pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:42:07.851975 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:07.851948    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 13:42:07.851975 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:07.851968    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 13:42:07.852113 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:07.851981    2571 projected.go:194] Error preparing data for projected volume kube-api-access-nlmkx for pod openshift-network-diagnostics/network-check-target-vhmzh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 13:42:07.852113 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:07.852069    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81333a26-7a37-4277-9753-ebd412eb34aa-kube-api-access-nlmkx podName:81333a26-7a37-4277-9753-ebd412eb34aa nodeName:}" failed. No retries permitted until 2026-05-29 13:42:08.85202395 +0000 UTC m=+3.319094023 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nlmkx" (UniqueName: "kubernetes.io/projected/81333a26-7a37-4277-9753-ebd412eb34aa-kube-api-access-nlmkx") pod "network-check-target-vhmzh" (UID: "81333a26-7a37-4277-9753-ebd412eb34aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 13:42:08.098092 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:08.097927    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-28 13:37:07 +0000 UTC" deadline="2027-10-31 00:27:38.314890167 +0000 UTC"
May 29 13:42:08.098092 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:08.097967    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12466h45m30.216927328s"
May 29 13:42:08.176065 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:08.175984    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-46jjq" event={"ID":"758a6d2f-ef64-4995-9a3b-d66b09cc98f1","Type":"ContainerStarted","Data":"e07494f6fa3fe4ed9395e41ffd703f23161b72cee033b3e82b55c984f2aefb41"}
May 29 13:42:08.189575 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:08.189536    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xqvkm" event={"ID":"5ded8d51-ee6a-492b-a1d4-5afefbf29dec","Type":"ContainerStarted","Data":"0ca865904fc75e969736f94af3ed83841a70ff2eb0dfc58c05809f76ff8f7085"}
May 29 13:42:08.205856 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:08.205809    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xqvgg" event={"ID":"dda729c3-39e3-45ac-ba23-1628f3874144","Type":"ContainerStarted","Data":"168e8b98541891560ddccc863bd3d418d1a4485de7325f33b085e20bbbbe3c24"}
May 29 13:42:08.222899 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:08.222853    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-1.ec2.internal" event={"ID":"f36931e78e4f753d131caf513c9d0151","Type":"ContainerStarted","Data":"4da7fbac9ce45497e4a3ec4d0b21356762de7ccdc1a88abbb71fbdce4ffcd9e1"}
May 29 13:42:08.228502 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:08.228423    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4vtqj" event={"ID":"1b2d1d27-e735-4a58-a93f-5d0bf642aa78","Type":"ContainerStarted","Data":"5c383e5aad58217e2fd90d298754491f76f540472044a38cb6cfc4196dcf0b0e"}
May 29 13:42:08.257715 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:08.257644    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9wzn2" event={"ID":"af79b7f2-4958-48fc-ad8d-ef0c731dc642","Type":"ContainerStarted","Data":"a75fc7cc7a7fadeff9b04996d30fb4a91e2a67eec4e3580aa9323b3aa5f5ae65"}
May 29 13:42:08.279500 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:08.279461    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt" event={"ID":"f1e1cf7f-6a08-455e-8c06-61778f55c020","Type":"ContainerStarted","Data":"a127ff71dc0326375a6e61e9f58faf8be8c8c06bd4c64f38d1b74657b1a9868c"}
May 29 13:42:08.298584 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:08.298546    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p2drl" event={"ID":"4f4cffa7-97f7-4149-8118-e3fd4edc15a7","Type":"ContainerStarted","Data":"da4c7165d3ee76c44161fa16a31f91534acf075cc8b3d5632279892f551b7157"}
May 29 13:42:08.317378 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:08.317336    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k7qwn" event={"ID":"82e3dbfa-9712-4124-8031-c9f9a0f45abb","Type":"ContainerStarted","Data":"496efc1505a5dd8ad61fe2f38d5bb636a0c10aa0c7cce8dd0a71fd448e96aa01"}
May 29 13:42:08.344163 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:08.344093    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bz56r" event={"ID":"2d022803-adf3-4686-8136-a57ab660afad","Type":"ContainerStarted","Data":"053fbd49c25f326aae6f30de9a90bceeccd109a6a15a5e6ca48742e75f3b8f5a"}
May 29 13:42:08.349437 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:08.349353    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-1.ec2.internal" event={"ID":"adf64b7f1c8724ca5ea45aa00abb85e2","Type":"ContainerStarted","Data":"c189628b77a28ea5353bdfb6bc3861048acb0a4e6fd8580a4d9af21b95cba256"}
May 29 13:42:08.479032 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:08.478995    2571 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 29 13:42:08.647120 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:08.647029    2571 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 29 13:42:08.762819 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:08.762236    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs\") pod \"network-metrics-daemon-n48xk\" (UID: \"2a114752-df85-41bc-b828-17308b5311de\") " pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:08.762819 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:08.762379    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 13:42:08.762819 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:08.762445    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs podName:2a114752-df85-41bc-b828-17308b5311de nodeName:}" failed. No retries permitted until 2026-05-29 13:42:10.762425339 +0000 UTC m=+5.229495411 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs") pod "network-metrics-daemon-n48xk" (UID: "2a114752-df85-41bc-b828-17308b5311de") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 13:42:08.863078 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:08.863028    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nlmkx\" (UniqueName: \"kubernetes.io/projected/81333a26-7a37-4277-9753-ebd412eb34aa-kube-api-access-nlmkx\") pod \"network-check-target-vhmzh\" (UID: \"81333a26-7a37-4277-9753-ebd412eb34aa\") " pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:42:08.863282 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:08.863262    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 13:42:08.863353 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:08.863290    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 13:42:08.863353 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:08.863303    2571 projected.go:194] Error preparing data for projected volume kube-api-access-nlmkx for pod openshift-network-diagnostics/network-check-target-vhmzh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 13:42:08.863474 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:08.863366    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81333a26-7a37-4277-9753-ebd412eb34aa-kube-api-access-nlmkx podName:81333a26-7a37-4277-9753-ebd412eb34aa nodeName:}" failed. No retries permitted until 2026-05-29 13:42:10.863347284 +0000 UTC m=+5.330417346 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nlmkx" (UniqueName: "kubernetes.io/projected/81333a26-7a37-4277-9753-ebd412eb34aa-kube-api-access-nlmkx") pod "network-check-target-vhmzh" (UID: "81333a26-7a37-4277-9753-ebd412eb34aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 13:42:09.098991 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:09.098899    2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-28 13:37:07 +0000 UTC" deadline="2027-11-11 15:24:20.674825644 +0000 UTC"
May 29 13:42:09.098991 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:09.098941    2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12745h42m11.575888725s"
May 29 13:42:09.143850 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:09.143823    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:42:09.143850 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:09.143843    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:09.144125 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:09.143949    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vhmzh" podUID="81333a26-7a37-4277-9753-ebd412eb34aa"
May 29 13:42:09.144262 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:09.144227    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n48xk" podUID="2a114752-df85-41bc-b828-17308b5311de"
May 29 13:42:09.178161 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:09.178124    2571 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 29 13:42:10.781062 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:10.781010    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs\") pod \"network-metrics-daemon-n48xk\" (UID: \"2a114752-df85-41bc-b828-17308b5311de\") " pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:10.781539 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:10.781249    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 13:42:10.781539 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:10.781323    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs podName:2a114752-df85-41bc-b828-17308b5311de nodeName:}" failed. No retries permitted until 2026-05-29 13:42:14.781303722 +0000 UTC m=+9.248373827 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs") pod "network-metrics-daemon-n48xk" (UID: "2a114752-df85-41bc-b828-17308b5311de") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 13:42:10.882814 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:10.882616    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nlmkx\" (UniqueName: \"kubernetes.io/projected/81333a26-7a37-4277-9753-ebd412eb34aa-kube-api-access-nlmkx\") pod \"network-check-target-vhmzh\" (UID: \"81333a26-7a37-4277-9753-ebd412eb34aa\") " pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:42:10.882814 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:10.882796    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 13:42:10.883023 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:10.882835    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 13:42:10.883023 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:10.882848    2571 projected.go:194] Error preparing data for projected volume kube-api-access-nlmkx for pod openshift-network-diagnostics/network-check-target-vhmzh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 13:42:10.883023 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:10.882913    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81333a26-7a37-4277-9753-ebd412eb34aa-kube-api-access-nlmkx podName:81333a26-7a37-4277-9753-ebd412eb34aa nodeName:}" failed. No retries permitted until 2026-05-29 13:42:14.882894488 +0000 UTC m=+9.349964552 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nlmkx" (UniqueName: "kubernetes.io/projected/81333a26-7a37-4277-9753-ebd412eb34aa-kube-api-access-nlmkx") pod "network-check-target-vhmzh" (UID: "81333a26-7a37-4277-9753-ebd412eb34aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 13:42:11.145411 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:11.144117    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:42:11.145411 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:11.144251    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vhmzh" podUID="81333a26-7a37-4277-9753-ebd412eb34aa"
May 29 13:42:11.145411 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:11.144647    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:11.145411 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:11.144751    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n48xk" podUID="2a114752-df85-41bc-b828-17308b5311de"
May 29 13:42:13.144645 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:13.144428    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:42:13.145108 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:13.144742    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vhmzh" podUID="81333a26-7a37-4277-9753-ebd412eb34aa"
May 29 13:42:13.145108 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:13.144428    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:13.145108 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:13.144852    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n48xk" podUID="2a114752-df85-41bc-b828-17308b5311de"
May 29 13:42:14.813369 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:14.813331    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs\") pod \"network-metrics-daemon-n48xk\" (UID: \"2a114752-df85-41bc-b828-17308b5311de\") " pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:14.813868 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:14.813495    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 13:42:14.813868 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:14.813564    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs podName:2a114752-df85-41bc-b828-17308b5311de nodeName:}" failed. No retries permitted until 2026-05-29 13:42:22.813543875 +0000 UTC m=+17.280613945 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs") pod "network-metrics-daemon-n48xk" (UID: "2a114752-df85-41bc-b828-17308b5311de") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 13:42:14.914769 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:14.914727    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nlmkx\" (UniqueName: \"kubernetes.io/projected/81333a26-7a37-4277-9753-ebd412eb34aa-kube-api-access-nlmkx\") pod \"network-check-target-vhmzh\" (UID: \"81333a26-7a37-4277-9753-ebd412eb34aa\") " pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:42:14.914939 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:14.914888    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 13:42:14.914939 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:14.914906    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 13:42:14.914939 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:14.914918    2571 projected.go:194] Error preparing data for projected volume kube-api-access-nlmkx for pod openshift-network-diagnostics/network-check-target-vhmzh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 13:42:14.915143 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:14.914983    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81333a26-7a37-4277-9753-ebd412eb34aa-kube-api-access-nlmkx podName:81333a26-7a37-4277-9753-ebd412eb34aa nodeName:}" failed. No retries permitted until 2026-05-29 13:42:22.914964103 +0000 UTC m=+17.382034167 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nlmkx" (UniqueName: "kubernetes.io/projected/81333a26-7a37-4277-9753-ebd412eb34aa-kube-api-access-nlmkx") pod "network-check-target-vhmzh" (UID: "81333a26-7a37-4277-9753-ebd412eb34aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 13:42:15.144333 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:15.144252    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:42:15.144509 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:15.144436    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vhmzh" podUID="81333a26-7a37-4277-9753-ebd412eb34aa"
May 29 13:42:15.144572 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:15.144558    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:15.144669 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:15.144651    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n48xk" podUID="2a114752-df85-41bc-b828-17308b5311de"
May 29 13:42:16.288555 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:16.288470    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-w4pw2"]
May 29 13:42:16.295038 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:16.294598    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:16.295038 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:16.294680    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4pw2" podUID="0d8451cb-e6fd-48aa-a319-bca020321b7a"
May 29 13:42:16.325983 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:16.325944    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d8451cb-e6fd-48aa-a319-bca020321b7a-original-pull-secret\") pod \"global-pull-secret-syncer-w4pw2\" (UID: \"0d8451cb-e6fd-48aa-a319-bca020321b7a\") " pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:16.325983 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:16.325995    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d8451cb-e6fd-48aa-a319-bca020321b7a-dbus\") pod \"global-pull-secret-syncer-w4pw2\" (UID: \"0d8451cb-e6fd-48aa-a319-bca020321b7a\") " pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:16.326236 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:16.326115    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d8451cb-e6fd-48aa-a319-bca020321b7a-kubelet-config\") pod \"global-pull-secret-syncer-w4pw2\" (UID: \"0d8451cb-e6fd-48aa-a319-bca020321b7a\") " pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:16.426861 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:16.426827    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d8451cb-e6fd-48aa-a319-bca020321b7a-kubelet-config\") pod \"global-pull-secret-syncer-w4pw2\" (UID: \"0d8451cb-e6fd-48aa-a319-bca020321b7a\") " pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:16.427029 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:16.426900    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d8451cb-e6fd-48aa-a319-bca020321b7a-original-pull-secret\") pod \"global-pull-secret-syncer-w4pw2\" (UID: \"0d8451cb-e6fd-48aa-a319-bca020321b7a\") " pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:16.427029 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:16.426930    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d8451cb-e6fd-48aa-a319-bca020321b7a-dbus\") pod \"global-pull-secret-syncer-w4pw2\" (UID: \"0d8451cb-e6fd-48aa-a319-bca020321b7a\") " pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:16.427164 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:16.427136    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0d8451cb-e6fd-48aa-a319-bca020321b7a-kubelet-config\") pod \"global-pull-secret-syncer-w4pw2\" (UID: \"0d8451cb-e6fd-48aa-a319-bca020321b7a\") " pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:16.427164 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:16.427147    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0d8451cb-e6fd-48aa-a319-bca020321b7a-dbus\") pod \"global-pull-secret-syncer-w4pw2\" (UID: \"0d8451cb-e6fd-48aa-a319-bca020321b7a\") " pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:16.427266 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:16.427235    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 29 13:42:16.427318 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:16.427287    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d8451cb-e6fd-48aa-a319-bca020321b7a-original-pull-secret podName:0d8451cb-e6fd-48aa-a319-bca020321b7a nodeName:}" failed. No retries permitted until 2026-05-29 13:42:16.92727171 +0000 UTC m=+11.394341769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d8451cb-e6fd-48aa-a319-bca020321b7a-original-pull-secret") pod "global-pull-secret-syncer-w4pw2" (UID: "0d8451cb-e6fd-48aa-a319-bca020321b7a") : object "kube-system"/"original-pull-secret" not registered
May 29 13:42:16.931243 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:16.931196    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d8451cb-e6fd-48aa-a319-bca020321b7a-original-pull-secret\") pod \"global-pull-secret-syncer-w4pw2\" (UID: \"0d8451cb-e6fd-48aa-a319-bca020321b7a\") " pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:16.931435 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:16.931400    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 29 13:42:16.931505 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:16.931489    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d8451cb-e6fd-48aa-a319-bca020321b7a-original-pull-secret podName:0d8451cb-e6fd-48aa-a319-bca020321b7a nodeName:}" failed. No retries permitted until 2026-05-29 13:42:17.931466632 +0000 UTC m=+12.398536702 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d8451cb-e6fd-48aa-a319-bca020321b7a-original-pull-secret") pod "global-pull-secret-syncer-w4pw2" (UID: "0d8451cb-e6fd-48aa-a319-bca020321b7a") : object "kube-system"/"original-pull-secret" not registered
May 29 13:42:17.144312 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:17.144277    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:42:17.144473 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:17.144403    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vhmzh" podUID="81333a26-7a37-4277-9753-ebd412eb34aa"
May 29 13:42:17.144829 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:17.144805    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:17.144951 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:17.144904    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n48xk" podUID="2a114752-df85-41bc-b828-17308b5311de"
May 29 13:42:17.939096 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:17.939062    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d8451cb-e6fd-48aa-a319-bca020321b7a-original-pull-secret\") pod \"global-pull-secret-syncer-w4pw2\" (UID: \"0d8451cb-e6fd-48aa-a319-bca020321b7a\") " pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:17.939433 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:17.939203    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 29 13:42:17.939433 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:17.939274    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d8451cb-e6fd-48aa-a319-bca020321b7a-original-pull-secret podName:0d8451cb-e6fd-48aa-a319-bca020321b7a nodeName:}" failed. No retries permitted until 2026-05-29 13:42:19.93925767 +0000 UTC m=+14.406327729 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d8451cb-e6fd-48aa-a319-bca020321b7a-original-pull-secret") pod "global-pull-secret-syncer-w4pw2" (UID: "0d8451cb-e6fd-48aa-a319-bca020321b7a") : object "kube-system"/"original-pull-secret" not registered
May 29 13:42:18.144173 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:18.144142    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:18.144315 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:18.144272    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4pw2" podUID="0d8451cb-e6fd-48aa-a319-bca020321b7a"
May 29 13:42:19.143977 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:19.143940    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:19.144401 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:19.143940    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:42:19.144401 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:19.144106    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n48xk" podUID="2a114752-df85-41bc-b828-17308b5311de"
May 29 13:42:19.144401 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:19.144131    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vhmzh" podUID="81333a26-7a37-4277-9753-ebd412eb34aa"
May 29 13:42:19.952078 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:19.952024    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d8451cb-e6fd-48aa-a319-bca020321b7a-original-pull-secret\") pod \"global-pull-secret-syncer-w4pw2\" (UID: \"0d8451cb-e6fd-48aa-a319-bca020321b7a\") " pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:19.952241 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:19.952189    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 29 13:42:19.952290 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:19.952268    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d8451cb-e6fd-48aa-a319-bca020321b7a-original-pull-secret podName:0d8451cb-e6fd-48aa-a319-bca020321b7a nodeName:}" failed. No retries permitted until 2026-05-29 13:42:23.952252491 +0000 UTC m=+18.419322550 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d8451cb-e6fd-48aa-a319-bca020321b7a-original-pull-secret") pod "global-pull-secret-syncer-w4pw2" (UID: "0d8451cb-e6fd-48aa-a319-bca020321b7a") : object "kube-system"/"original-pull-secret" not registered
May 29 13:42:20.144258 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:20.144227    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:20.144677 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:20.144357    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4pw2" podUID="0d8451cb-e6fd-48aa-a319-bca020321b7a"
May 29 13:42:21.144229 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:21.144194    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:21.144424 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:21.144192    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:42:21.144424 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:21.144331    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n48xk" podUID="2a114752-df85-41bc-b828-17308b5311de"
May 29 13:42:21.144424 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:21.144395    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vhmzh" podUID="81333a26-7a37-4277-9753-ebd412eb34aa"
May 29 13:42:22.144495 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:22.144411    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:22.144930 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:22.144528    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4pw2" podUID="0d8451cb-e6fd-48aa-a319-bca020321b7a"
May 29 13:42:22.875289 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:22.875254    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs\") pod \"network-metrics-daemon-n48xk\" (UID: \"2a114752-df85-41bc-b828-17308b5311de\") " pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:22.875489 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:22.875368    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 13:42:22.875489 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:22.875435    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs podName:2a114752-df85-41bc-b828-17308b5311de nodeName:}" failed. No retries permitted until 2026-05-29 13:42:38.875416788 +0000 UTC m=+33.342486850 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs") pod "network-metrics-daemon-n48xk" (UID: "2a114752-df85-41bc-b828-17308b5311de") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 13:42:22.976166 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:22.976131    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nlmkx\" (UniqueName: \"kubernetes.io/projected/81333a26-7a37-4277-9753-ebd412eb34aa-kube-api-access-nlmkx\") pod \"network-check-target-vhmzh\" (UID: \"81333a26-7a37-4277-9753-ebd412eb34aa\") " pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:42:22.976345 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:22.976327    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 13:42:22.976389 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:22.976352    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 13:42:22.976389 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:22.976367    2571 projected.go:194] Error preparing data for projected volume kube-api-access-nlmkx for pod openshift-network-diagnostics/network-check-target-vhmzh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 13:42:22.976455 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:22.976431    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81333a26-7a37-4277-9753-ebd412eb34aa-kube-api-access-nlmkx podName:81333a26-7a37-4277-9753-ebd412eb34aa nodeName:}" failed. No retries permitted until 2026-05-29 13:42:38.976412527 +0000 UTC m=+33.443482593 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nlmkx" (UniqueName: "kubernetes.io/projected/81333a26-7a37-4277-9753-ebd412eb34aa-kube-api-access-nlmkx") pod "network-check-target-vhmzh" (UID: "81333a26-7a37-4277-9753-ebd412eb34aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 13:42:23.143927 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:23.143852    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:42:23.144105 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:23.143852    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:23.144105 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:23.143959    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vhmzh" podUID="81333a26-7a37-4277-9753-ebd412eb34aa"
May 29 13:42:23.144105 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:23.144063    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n48xk" podUID="2a114752-df85-41bc-b828-17308b5311de"
May 29 13:42:23.983696 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:23.983663    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d8451cb-e6fd-48aa-a319-bca020321b7a-original-pull-secret\") pod \"global-pull-secret-syncer-w4pw2\" (UID: \"0d8451cb-e6fd-48aa-a319-bca020321b7a\") " pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:23.984113 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:23.983791    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 29 13:42:23.984113 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:23.983868    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d8451cb-e6fd-48aa-a319-bca020321b7a-original-pull-secret podName:0d8451cb-e6fd-48aa-a319-bca020321b7a nodeName:}" failed. No retries permitted until 2026-05-29 13:42:31.983851457 +0000 UTC m=+26.450921516 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d8451cb-e6fd-48aa-a319-bca020321b7a-original-pull-secret") pod "global-pull-secret-syncer-w4pw2" (UID: "0d8451cb-e6fd-48aa-a319-bca020321b7a") : object "kube-system"/"original-pull-secret" not registered
May 29 13:42:24.143795 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:24.143764    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:24.143957 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:24.143879    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4pw2" podUID="0d8451cb-e6fd-48aa-a319-bca020321b7a"
May 29 13:42:25.144292 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:25.144253    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:42:25.144760 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:25.144268    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:25.144760 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:25.144377    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vhmzh" podUID="81333a26-7a37-4277-9753-ebd412eb34aa"
May 29 13:42:25.144760 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:25.144451    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n48xk" podUID="2a114752-df85-41bc-b828-17308b5311de"
May 29 13:42:26.144019 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:26.143988    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:26.144166 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:26.144084    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4pw2" podUID="0d8451cb-e6fd-48aa-a319-bca020321b7a"
May 29 13:42:27.144628 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:27.144127    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:42:27.144628 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:27.144166    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:27.144628 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:27.144561    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vhmzh" podUID="81333a26-7a37-4277-9753-ebd412eb34aa"
May 29 13:42:27.144628 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:27.144585    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n48xk" podUID="2a114752-df85-41bc-b828-17308b5311de"
May 29 13:42:27.398561 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:27.398522    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-46jjq" event={"ID":"758a6d2f-ef64-4995-9a3b-d66b09cc98f1","Type":"ContainerStarted","Data":"9ca580207b144e401983917f3f51414fb74d3aebb7791025014d9d582e08b375"}
May 29 13:42:27.399898 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:27.399872    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xqvgg" event={"ID":"dda729c3-39e3-45ac-ba23-1628f3874144","Type":"ContainerStarted","Data":"9928e8e1fad7ae54c14f6dcebd4d1f5f0a115fd14e19491bb4ff66fdad34c7f1"}
May 29 13:42:27.407656 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:27.407501    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p2drl" event={"ID":"4f4cffa7-97f7-4149-8118-e3fd4edc15a7","Type":"ContainerStarted","Data":"bb51c94751fb7d381ef899ee9b2b1e444ed40f6ac23b18a0e53896e92ccdd816"}
May 29 13:42:27.407656 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:27.407539    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p2drl" event={"ID":"4f4cffa7-97f7-4149-8118-e3fd4edc15a7","Type":"ContainerStarted","Data":"a1aeca841dd2ef6c7783b015687d1b052915b253f7b4f4e50f729f98081d601e"}
May 29 13:42:27.407656 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:27.407552    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p2drl" event={"ID":"4f4cffa7-97f7-4149-8118-e3fd4edc15a7","Type":"ContainerStarted","Data":"830ea00cdc96772b44fd91c347330fd557e968f44327c4e51f4de1d4df7a4c3d"}
May 29 13:42:27.407656 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:27.407565    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p2drl" event={"ID":"4f4cffa7-97f7-4149-8118-e3fd4edc15a7","Type":"ContainerStarted","Data":"d4093ac0ceda6244a75e8439d3550d2dad9da2241ab3a39b3abec2709344d8da"}
May 29 13:42:27.409228 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:27.409170    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-1.ec2.internal" event={"ID":"adf64b7f1c8724ca5ea45aa00abb85e2","Type":"ContainerStarted","Data":"4aa00316e1aa6c59d170b31ca59e77229f92e37f7c08498da896feaff4dc9c56"}
May 29 13:42:27.412912 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:27.412861    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-46jjq" podStartSLOduration=2.306117416 podStartE2EDuration="21.412847633s" podCreationTimestamp="2026-05-29 13:42:06 +0000 UTC" firstStartedPulling="2026-05-29 13:42:07.467976292 +0000 UTC m=+1.935046351" lastFinishedPulling="2026-05-29 13:42:26.574706503 +0000 UTC m=+21.041776568" observedRunningTime="2026-05-29 13:42:27.412411602 +0000 UTC m=+21.879481684" watchObservedRunningTime="2026-05-29 13:42:27.412847633 +0000 UTC m=+21.879917774"
May 29 13:42:27.439087 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:27.438188    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-1.ec2.internal" podStartSLOduration=21.438170093 podStartE2EDuration="21.438170093s" podCreationTimestamp="2026-05-29 13:42:06 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 13:42:27.424626406 +0000 UTC m=+21.891696486" watchObservedRunningTime="2026-05-29 13:42:27.438170093 +0000 UTC m=+21.905240175"
May 29 13:42:27.439087 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:27.438835    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xqvgg" podStartSLOduration=2.254483764 podStartE2EDuration="21.43882389s" podCreationTimestamp="2026-05-29 13:42:06 +0000 UTC" firstStartedPulling="2026-05-29 13:42:07.390979333 +0000 UTC m=+1.858049398" lastFinishedPulling="2026-05-29 13:42:26.575319452 +0000 UTC m=+21.042389524" observedRunningTime="2026-05-29 13:42:27.438117618 +0000 UTC m=+21.905187705" watchObservedRunningTime="2026-05-29 13:42:27.43882389 +0000 UTC m=+21.905893971"
May 29 13:42:28.144219 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:28.143909    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:28.144430 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:28.144267    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4pw2" podUID="0d8451cb-e6fd-48aa-a319-bca020321b7a"
May 29 13:42:28.411693 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:28.411613    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9wzn2" event={"ID":"af79b7f2-4958-48fc-ad8d-ef0c731dc642","Type":"ContainerStarted","Data":"1eb979facf210379ad41e6d68a8feeda82bdb407b5be4dfa4040a5438b567c5a"}
May 29 13:42:28.413121 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:28.413091    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt" event={"ID":"f1e1cf7f-6a08-455e-8c06-61778f55c020","Type":"ContainerStarted","Data":"6f4f17b368299e9afed1cf9060c5e80e5eb7b7cb405fdaa2f7ff2523a17e3f7b"}
May 29 13:42:28.416022 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:28.415999    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p2drl" event={"ID":"4f4cffa7-97f7-4149-8118-e3fd4edc15a7","Type":"ContainerStarted","Data":"a0438ce2a7db34e18e8bcd8e79a3c55e830c4bae6fba5d171b6d912eda167880"}
May 29 13:42:28.416139 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:28.416027    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p2drl" event={"ID":"4f4cffa7-97f7-4149-8118-e3fd4edc15a7","Type":"ContainerStarted","Data":"5b48bd7d33651054cded322c27a4e3971d059bb2461d59717cbf52288c145274"}
May 29 13:42:28.417358 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:28.417337    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k7qwn" event={"ID":"82e3dbfa-9712-4124-8031-c9f9a0f45abb","Type":"ContainerStarted","Data":"7e8b2895cdb843d623a59a174587200419153f0b33bacc9325ae13683cd8fe57"}
May 29 13:42:28.418629 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:28.418596    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bz56r" event={"ID":"2d022803-adf3-4686-8136-a57ab660afad","Type":"ContainerStarted","Data":"469f9a12efce3744bedfaff1f5ed220beca029f687b223fdad8f7903b70135a5"}
May 29 13:42:28.419912 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:28.419891    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xqvkm" event={"ID":"5ded8d51-ee6a-492b-a1d4-5afefbf29dec","Type":"ContainerStarted","Data":"28170795643b286d59d9985f6915b27fc62eef326e8ada746f236d67e6827e44"}
May 29 13:42:28.421356 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:28.421333    2571 generic.go:358] "Generic (PLEG): container finished" podID="f36931e78e4f753d131caf513c9d0151" containerID="30dab36877a54df04d107e769db393f7bc7a9d88c40c5ccecb5b5eb36d3f3a17" exitCode=0
May 29 13:42:28.421450 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:28.421403    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-1.ec2.internal" event={"ID":"f36931e78e4f753d131caf513c9d0151","Type":"ContainerDied","Data":"30dab36877a54df04d107e769db393f7bc7a9d88c40c5ccecb5b5eb36d3f3a17"}
May 29 13:42:28.421526 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:28.421508    2571 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-1.ec2.internal"
May 29 13:42:28.422856 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:28.422830    2571 generic.go:358] "Generic (PLEG): container finished" podID="1b2d1d27-e735-4a58-a93f-5d0bf642aa78" containerID="358fd2a52a0a35de244a1f26433944daa58bf4b47516297ee0f8ba1613ae67c2" exitCode=0
May 29 13:42:28.422957 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:28.422927    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4vtqj" event={"ID":"1b2d1d27-e735-4a58-a93f-5d0bf642aa78","Type":"ContainerDied","Data":"358fd2a52a0a35de244a1f26433944daa58bf4b47516297ee0f8ba1613ae67c2"}
May 29 13:42:28.431700 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:28.431678    2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 29 13:42:28.432556 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:28.432527    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-1.ec2.internal"]
May 29 13:42:28.441803 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:28.441748    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9wzn2" podStartSLOduration=3.325326439 podStartE2EDuration="22.44173397s" podCreationTimestamp="2026-05-29 13:42:06 +0000 UTC" firstStartedPulling="2026-05-29 13:42:07.456478847 +0000 UTC m=+1.923548906" lastFinishedPulling="2026-05-29 13:42:26.572886378 +0000 UTC m=+21.039956437" observedRunningTime="2026-05-29 13:42:28.441627844 +0000 UTC m=+22.908697924" watchObservedRunningTime="2026-05-29 13:42:28.44173397 +0000 UTC m=+22.908804053"
May 29 13:42:28.457357 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:28.457300    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k7qwn" podStartSLOduration=3.32420507 podStartE2EDuration="22.457282221s" podCreationTimestamp="2026-05-29 13:42:06 +0000 UTC" firstStartedPulling="2026-05-29 13:42:07.411044879 +0000 UTC m=+1.878114939" lastFinishedPulling="2026-05-29 13:42:26.544122017 +0000 UTC m=+21.011192090" observedRunningTime="2026-05-29 13:42:28.457214933 +0000 UTC m=+22.924285015" watchObservedRunningTime="2026-05-29 13:42:28.457282221 +0000 UTC m=+22.924352304"
May 29 13:42:28.475757 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:28.475691    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-xqvkm" podStartSLOduration=3.557891528 podStartE2EDuration="22.475675168s" podCreationTimestamp="2026-05-29 13:42:06 +0000 UTC" firstStartedPulling="2026-05-29 13:42:07.438916972 +0000 UTC m=+1.905987031" lastFinishedPulling="2026-05-29 13:42:26.356700612 +0000 UTC m=+20.823770671" observedRunningTime="2026-05-29 13:42:28.475325918 +0000 UTC m=+22.942395999" watchObservedRunningTime="2026-05-29 13:42:28.475675168 +0000 UTC m=+22.942745249"
May 29 13:42:28.526363 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:28.526319    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bz56r" podStartSLOduration=3.317775106 podStartE2EDuration="22.526305162s" podCreationTimestamp="2026-05-29 13:42:06 +0000 UTC" firstStartedPulling="2026-05-29 13:42:07.363992541 +0000 UTC m=+1.831062599" lastFinishedPulling="2026-05-29 13:42:26.572522585 +0000 UTC m=+21.039592655" observedRunningTime="2026-05-29 13:42:28.526173119 +0000 UTC m=+22.993243199" watchObservedRunningTime="2026-05-29 13:42:28.526305162 +0000 UTC m=+22.993375242"
May 29 13:42:28.838751 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:28.838616    2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 29 13:42:29.116156 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:29.116002    2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-29T13:42:28.838630992Z","UUID":"a090d286-c5f2-4f84-a248-a728a7dae6bf","Handler":null,"Name":"","Endpoint":""}
May 29 13:42:29.119274 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:29.119248    2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 29 13:42:29.119274 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:29.119279    2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 29 13:42:29.144160 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:29.144128    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:29.144309 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:29.144128    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:42:29.144309 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:29.144270    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n48xk" podUID="2a114752-df85-41bc-b828-17308b5311de"
May 29 13:42:29.144411 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:29.144336    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vhmzh" podUID="81333a26-7a37-4277-9753-ebd412eb34aa"
May 29 13:42:29.426319 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:29.426240    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-1.ec2.internal" event={"ID":"f36931e78e4f753d131caf513c9d0151","Type":"ContainerStarted","Data":"14b17723a1147760d0af5adca5e1751453c8638800b3bfe44b14d22b4a5f7836"}
May 29 13:42:29.427954 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:29.427902    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt" event={"ID":"f1e1cf7f-6a08-455e-8c06-61778f55c020","Type":"ContainerStarted","Data":"c8ab2451706ebb6a446590331971a334ecdc6d0016ec6447e03aee92b4ab140b"}
May 29 13:42:29.442807 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:29.442757    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-1.ec2.internal" podStartSLOduration=1.44273846 podStartE2EDuration="1.44273846s" podCreationTimestamp="2026-05-29 13:42:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 13:42:29.441989716 +0000 UTC m=+23.909060019" watchObservedRunningTime="2026-05-29 13:42:29.44273846 +0000 UTC m=+23.909808540"
May 29 13:42:30.144362 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:30.144324    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:30.144534 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:30.144460    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4pw2" podUID="0d8451cb-e6fd-48aa-a319-bca020321b7a"
May 29 13:42:30.432229 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:30.432139    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt" event={"ID":"f1e1cf7f-6a08-455e-8c06-61778f55c020","Type":"ContainerStarted","Data":"37efb104aa368c544c14261d357f4c4eff8340799974f3bc2d9cd9816657a3b4"}
May 29 13:42:30.435562 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:30.435527    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p2drl" event={"ID":"4f4cffa7-97f7-4149-8118-e3fd4edc15a7","Type":"ContainerStarted","Data":"f64b5fac793332043c5fc702cc2b75ff9ec3f38326c55c613d9b6f0c3fb60701"}
May 29 13:42:30.451716 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:30.451661    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-chlrt" podStartSLOduration=1.929225983 podStartE2EDuration="24.451647165s" podCreationTimestamp="2026-05-29 13:42:06 +0000 UTC" firstStartedPulling="2026-05-29 13:42:07.425026181 +0000 UTC m=+1.892096243" lastFinishedPulling="2026-05-29 13:42:29.94744732 +0000 UTC m=+24.414517425" observedRunningTime="2026-05-29 13:42:30.451609011 +0000 UTC m=+24.918679086" watchObservedRunningTime="2026-05-29 13:42:30.451647165 +0000 UTC m=+24.918717246"
May 29 13:42:31.143634 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:31.143598    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:42:31.143805 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:31.143611    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:31.143805 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:31.143710    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vhmzh" podUID="81333a26-7a37-4277-9753-ebd412eb34aa"
May 29 13:42:31.143918 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:31.143824    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n48xk" podUID="2a114752-df85-41bc-b828-17308b5311de"
May 29 13:42:31.725809 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:31.725773    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bz56r"
May 29 13:42:32.040654 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:32.040614    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d8451cb-e6fd-48aa-a319-bca020321b7a-original-pull-secret\") pod \"global-pull-secret-syncer-w4pw2\" (UID: \"0d8451cb-e6fd-48aa-a319-bca020321b7a\") " pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:32.040837 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:32.040780    2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
May 29 13:42:32.040902 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:32.040861    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d8451cb-e6fd-48aa-a319-bca020321b7a-original-pull-secret podName:0d8451cb-e6fd-48aa-a319-bca020321b7a nodeName:}" failed. No retries permitted until 2026-05-29 13:42:48.040840268 +0000 UTC m=+42.507910349 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0d8451cb-e6fd-48aa-a319-bca020321b7a-original-pull-secret") pod "global-pull-secret-syncer-w4pw2" (UID: "0d8451cb-e6fd-48aa-a319-bca020321b7a") : object "kube-system"/"original-pull-secret" not registered
May 29 13:42:32.144042 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:32.144003    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:32.144256 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:32.144168    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4pw2" podUID="0d8451cb-e6fd-48aa-a319-bca020321b7a"
May 29 13:42:32.681571 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:32.681526    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bz56r"
May 29 13:42:32.682346 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:32.682308    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bz56r"
May 29 13:42:33.144283 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:33.144200    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:33.144283 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:33.144248    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:42:33.144775 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:33.144345    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n48xk" podUID="2a114752-df85-41bc-b828-17308b5311de"
May 29 13:42:33.144775 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:33.144475    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vhmzh" podUID="81333a26-7a37-4277-9753-ebd412eb34aa"
May 29 13:42:33.441957 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:33.441799    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bz56r"
May 29 13:42:34.143917 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:34.143828    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:34.144041 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:34.143969    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4pw2" podUID="0d8451cb-e6fd-48aa-a319-bca020321b7a"
May 29 13:42:34.444288 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:34.444184    2571 generic.go:358] "Generic (PLEG): container finished" podID="1b2d1d27-e735-4a58-a93f-5d0bf642aa78" containerID="cd7127ebf742dfe19e478e29e803507d9e5f1e9c4cb262560f47b9f0871434eb" exitCode=0
May 29 13:42:34.445178 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:34.444284    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4vtqj" event={"ID":"1b2d1d27-e735-4a58-a93f-5d0bf642aa78","Type":"ContainerDied","Data":"cd7127ebf742dfe19e478e29e803507d9e5f1e9c4cb262560f47b9f0871434eb"}
May 29 13:42:34.449848 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:34.449820    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p2drl" event={"ID":"4f4cffa7-97f7-4149-8118-e3fd4edc15a7","Type":"ContainerStarted","Data":"e0217ae37f912c900eac43e47530b39496386b71d464b7e320c5f7386e2de6f7"}
May 29 13:42:34.450097 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:34.450072    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:34.450097 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:34.450101    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:34.465585 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:34.465559    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:34.493474 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:34.493418    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-p2drl" podStartSLOduration=8.95450594 podStartE2EDuration="28.493404459s" podCreationTimestamp="2026-05-29 13:42:06 +0000 UTC" firstStartedPulling="2026-05-29 13:42:07.413107917 +0000 UTC m=+1.880177977" lastFinishedPulling="2026-05-29 13:42:26.952006422 +0000 UTC m=+21.419076496" observedRunningTime="2026-05-29 13:42:34.492323177 +0000 UTC m=+28.959393252" watchObservedRunningTime="2026-05-29 13:42:34.493404459 +0000 UTC m=+28.960474570"
May 29 13:42:35.143904 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:35.143697    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:42:35.144091 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:35.143701    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:35.144091 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:35.144012    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vhmzh" podUID="81333a26-7a37-4277-9753-ebd412eb34aa"
May 29 13:42:35.144215 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:35.144114    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n48xk" podUID="2a114752-df85-41bc-b828-17308b5311de"
May 29 13:42:35.446349 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:35.446323    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w4pw2"]
May 29 13:42:35.446665 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:35.446428    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:35.446665 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:35.446531    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4pw2" podUID="0d8451cb-e6fd-48aa-a319-bca020321b7a"
May 29 13:42:35.449021 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:35.448996    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vhmzh"]
May 29 13:42:35.451855 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:35.451838    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:42:35.451948 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:35.451929    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vhmzh" podUID="81333a26-7a37-4277-9753-ebd412eb34aa"
May 29 13:42:35.452399 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:35.452382    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:35.452491 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:35.452411    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n48xk"]
May 29 13:42:35.452491 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:35.452484    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:35.452618 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:35.452579    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n48xk" podUID="2a114752-df85-41bc-b828-17308b5311de"
May 29 13:42:35.468767 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:35.468742    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:42:36.456062 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:36.455973    2571 generic.go:358] "Generic (PLEG): container finished" podID="1b2d1d27-e735-4a58-a93f-5d0bf642aa78" containerID="4c0cec27d3bcb38909ea245a7a871eea58059adac01bef493549546998844fd2" exitCode=0
May 29 13:42:36.456441 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:36.456072    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4vtqj" event={"ID":"1b2d1d27-e735-4a58-a93f-5d0bf642aa78","Type":"ContainerDied","Data":"4c0cec27d3bcb38909ea245a7a871eea58059adac01bef493549546998844fd2"}
May 29 13:42:37.144557 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:37.144523    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:42:37.144557 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:37.144553    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:37.144831 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:37.144533    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:37.144831 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:37.144641    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vhmzh" podUID="81333a26-7a37-4277-9753-ebd412eb34aa"
May 29 13:42:37.144831 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:37.144725    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n48xk" podUID="2a114752-df85-41bc-b828-17308b5311de"
May 29 13:42:37.144831 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:37.144813    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4pw2" podUID="0d8451cb-e6fd-48aa-a319-bca020321b7a"
May 29 13:42:38.461432 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:38.461391    2571 generic.go:358] "Generic (PLEG): container finished" podID="1b2d1d27-e735-4a58-a93f-5d0bf642aa78" containerID="05cf69beecd077f476a2677cb7a692a847c3b963ed7a400bce54d4b36f63acf0" exitCode=0
May 29 13:42:38.461798 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:38.461462    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4vtqj" event={"ID":"1b2d1d27-e735-4a58-a93f-5d0bf642aa78","Type":"ContainerDied","Data":"05cf69beecd077f476a2677cb7a692a847c3b963ed7a400bce54d4b36f63acf0"}
May 29 13:42:38.895723 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:38.895687    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs\") pod \"network-metrics-daemon-n48xk\" (UID: \"2a114752-df85-41bc-b828-17308b5311de\") " pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:38.895870 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:38.895797    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 13:42:38.895870 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:38.895849    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs podName:2a114752-df85-41bc-b828-17308b5311de nodeName:}" failed. No retries permitted until 2026-05-29 13:43:10.895832706 +0000 UTC m=+65.362902765 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs") pod "network-metrics-daemon-n48xk" (UID: "2a114752-df85-41bc-b828-17308b5311de") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 29 13:42:38.996098 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:38.996062    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nlmkx\" (UniqueName: \"kubernetes.io/projected/81333a26-7a37-4277-9753-ebd412eb34aa-kube-api-access-nlmkx\") pod \"network-check-target-vhmzh\" (UID: \"81333a26-7a37-4277-9753-ebd412eb34aa\") " pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:42:38.996230 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:38.996158    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 29 13:42:38.996230 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:38.996174    2571 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 29 13:42:38.996230 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:38.996186    2571 projected.go:194] Error preparing data for projected volume kube-api-access-nlmkx for pod openshift-network-diagnostics/network-check-target-vhmzh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 13:42:38.996315 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:38.996235    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/81333a26-7a37-4277-9753-ebd412eb34aa-kube-api-access-nlmkx podName:81333a26-7a37-4277-9753-ebd412eb34aa nodeName:}" failed. No retries permitted until 2026-05-29 13:43:10.996220991 +0000 UTC m=+65.463291050 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nlmkx" (UniqueName: "kubernetes.io/projected/81333a26-7a37-4277-9753-ebd412eb34aa-kube-api-access-nlmkx") pod "network-check-target-vhmzh" (UID: "81333a26-7a37-4277-9753-ebd412eb34aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 29 13:42:39.143948 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.143908    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:39.144149 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.143908    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:42:39.144149 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:39.144047    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n48xk" podUID="2a114752-df85-41bc-b828-17308b5311de"
May 29 13:42:39.144149 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.143909    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:39.144149 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:39.144122    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vhmzh" podUID="81333a26-7a37-4277-9753-ebd412eb34aa"
May 29 13:42:39.144329 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:39.144203    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w4pw2" podUID="0d8451cb-e6fd-48aa-a319-bca020321b7a"
May 29 13:42:39.895553 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.895524    2571 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-141-1.ec2.internal" event="NodeReady"
May 29 13:42:39.895995 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.895673    2571 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 29 13:42:39.936251 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.936208    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c79dcd5cd-ggh69"]
May 29 13:42:39.948879 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.948848    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-555f88bbf7-5lxbw"]
May 29 13:42:39.949042 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.948997    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c79dcd5cd-ggh69"
May 29 13:42:39.951277 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.951038    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
May 29 13:42:39.951423 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.951320    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
May 29 13:42:39.951609 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.951562    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-bqlmj\""
May 29 13:42:39.951712 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.951604    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
May 29 13:42:39.951773 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.951588    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
May 29 13:42:39.959201 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.959176    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp"]
May 29 13:42:39.960161 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.959585    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:39.963562 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.963519    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 29 13:42:39.963861 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.963844    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zfktx\""
May 29 13:42:39.964665 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.964648    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 29 13:42:39.966287 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.966267    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 29 13:42:39.972007 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.971984    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 29 13:42:39.975434 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.975411    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6b486f46-5wxf8"]
May 29 13:42:39.975619 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.975603    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp"
May 29 13:42:39.978366 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.977729    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
May 29 13:42:39.978366 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.977763    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
May 29 13:42:39.978366 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.977828    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
May 29 13:42:39.978366 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.978029    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
May 29 13:42:39.987721 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.987697    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c79dcd5cd-ggh69"]
May 29 13:42:39.987721 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.987728    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6b486f46-5wxf8"]
May 29 13:42:39.987923 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.987740    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp"]
May 29 13:42:39.987923 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.987752    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-555f88bbf7-5lxbw"]
May 29 13:42:39.987923 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.987766    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7csjl"]
May 29 13:42:39.987923 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.987861    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6b486f46-5wxf8"
May 29 13:42:39.989870 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:39.989849    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
May 29 13:42:40.004039 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.004017    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dn82q"]
May 29 13:42:40.004170 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.004154    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7csjl"
May 29 13:42:40.006278 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.006260    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 29 13:42:40.006365 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.006339    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 29 13:42:40.006437 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.006421    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wdfdw\""
May 29 13:42:40.018360 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.018337    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7csjl"]
May 29 13:42:40.018360 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.018360    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dn82q"]
May 29 13:42:40.018510 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.018450    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dn82q"
May 29 13:42:40.020519 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.020500    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-25vw9\""
May 29 13:42:40.021190 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.021173    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 29 13:42:40.021274 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.021185    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 29 13:42:40.021274 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.021209    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 29 13:42:40.104357 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.104313    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-certificates\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:40.104357 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.104356    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhwxz\" (UniqueName: \"kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-kube-api-access-jhwxz\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:40.104595 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.104430    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlf77\" (UniqueName: \"kubernetes.io/projected/05bfd411-03f2-462d-bc97-568b19e3626d-kube-api-access-mlf77\") pod \"dns-default-7csjl\" (UID: \"05bfd411-03f2-462d-bc97-568b19e3626d\") " pod="openshift-dns/dns-default-7csjl"
May 29 13:42:40.104595 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.104461    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnzls\" (UniqueName: \"kubernetes.io/projected/29951f01-4535-401e-b698-8096a4fbc290-kube-api-access-pnzls\") pod \"ingress-canary-dn82q\" (UID: \"29951f01-4535-401e-b698-8096a4fbc290\") " pod="openshift-ingress-canary/ingress-canary-dn82q"
May 29 13:42:40.104595 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.104514    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2mbh\" (UniqueName: \"kubernetes.io/projected/35a676b1-9b9b-4c37-bbeb-2d61e14d69b6-kube-api-access-g2mbh\") pod \"managed-serviceaccount-addon-agent-7c79dcd5cd-ggh69\" (UID: \"35a676b1-9b9b-4c37-bbeb-2d61e14d69b6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c79dcd5cd-ggh69"
May 29 13:42:40.104802 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.104591    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/dafac8b7-3c8a-4dfd-b399-9988bbdca1f7-ca\") pod \"cluster-proxy-proxy-agent-7d7757b49f-zntlp\" (UID: \"dafac8b7-3c8a-4dfd-b399-9988bbdca1f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp"
May 29 13:42:40.104802 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.104668    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-bound-sa-token\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:40.104802 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.104704    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/dafac8b7-3c8a-4dfd-b399-9988bbdca1f7-hub\") pod \"cluster-proxy-proxy-agent-7d7757b49f-zntlp\" (UID: \"dafac8b7-3c8a-4dfd-b399-9988bbdca1f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp"
May 29 13:42:40.104802 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.104729    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/dafac8b7-3c8a-4dfd-b399-9988bbdca1f7-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-7d7757b49f-zntlp\" (UID: \"dafac8b7-3c8a-4dfd-b399-9988bbdca1f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp"
May 29 13:42:40.104802 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.104758    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/dafac8b7-3c8a-4dfd-b399-9988bbdca1f7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d7757b49f-zntlp\" (UID: \"dafac8b7-3c8a-4dfd-b399-9988bbdca1f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp"
May 29 13:42:40.105012 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.104816    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/05bfd411-03f2-462d-bc97-568b19e3626d-config-volume\") pod \"dns-default-7csjl\" (UID: \"05bfd411-03f2-462d-bc97-568b19e3626d\") " pod="openshift-dns/dns-default-7csjl"
May 29 13:42:40.105012 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.104844    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05bfd411-03f2-462d-bc97-568b19e3626d-tmp-dir\") pod \"dns-default-7csjl\" (UID: \"05bfd411-03f2-462d-bc97-568b19e3626d\") " pod="openshift-dns/dns-default-7csjl"
May 29 13:42:40.105012 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.104866    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29951f01-4535-401e-b698-8096a4fbc290-cert\") pod \"ingress-canary-dn82q\" (UID: \"29951f01-4535-401e-b698-8096a4fbc290\") " pod="openshift-ingress-canary/ingress-canary-dn82q"
May 29 13:42:40.105012 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.104921    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fc6f03b-90e5-442b-a552-c73faeb3a354-image-registry-private-configuration\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:40.105012 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.104957    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dafac8b7-3c8a-4dfd-b399-9988bbdca1f7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d7757b49f-zntlp\" (UID: \"dafac8b7-3c8a-4dfd-b399-9988bbdca1f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp"
May 29 13:42:40.105012 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.104985    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p9jh\" (UniqueName: \"kubernetes.io/projected/dafac8b7-3c8a-4dfd-b399-9988bbdca1f7-kube-api-access-2p9jh\") pod \"cluster-proxy-proxy-agent-7d7757b49f-zntlp\" (UID: \"dafac8b7-3c8a-4dfd-b399-9988bbdca1f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp"
May 29 13:42:40.105012 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.105008    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35a676b1-9b9b-4c37-bbeb-2d61e14d69b6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c79dcd5cd-ggh69\" (UID: \"35a676b1-9b9b-4c37-bbeb-2d61e14d69b6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c79dcd5cd-ggh69"
May 29 13:42:40.105360 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.105031    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:40.105360 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.105081    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fc6f03b-90e5-442b-a552-c73faeb3a354-installation-pull-secrets\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:40.105360 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.105122    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f6bd72e-60f1-4f97-9e7c-2cf694e17736-tmp\") pod \"klusterlet-addon-workmgr-5c6b486f46-5wxf8\" (UID: \"4f6bd72e-60f1-4f97-9e7c-2cf694e17736\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6b486f46-5wxf8"
May 29 13:42:40.105360 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.105148    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4f6bd72e-60f1-4f97-9e7c-2cf694e17736-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c6b486f46-5wxf8\" (UID: \"4f6bd72e-60f1-4f97-9e7c-2cf694e17736\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6b486f46-5wxf8"
May 29 13:42:40.105360 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.105172    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fc6f03b-90e5-442b-a552-c73faeb3a354-ca-trust-extracted\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:40.105360 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.105206    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fc6f03b-90e5-442b-a552-c73faeb3a354-trusted-ca\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:40.105360 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.105228    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05bfd411-03f2-462d-bc97-568b19e3626d-metrics-tls\") pod \"dns-default-7csjl\" (UID: \"05bfd411-03f2-462d-bc97-568b19e3626d\") " pod="openshift-dns/dns-default-7csjl"
May 29 13:42:40.105360 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.105256    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzjq6\" (UniqueName: \"kubernetes.io/projected/4f6bd72e-60f1-4f97-9e7c-2cf694e17736-kube-api-access-wzjq6\") pod \"klusterlet-addon-workmgr-5c6b486f46-5wxf8\" (UID: \"4f6bd72e-60f1-4f97-9e7c-2cf694e17736\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6b486f46-5wxf8"
May 29 13:42:40.205985 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.205905    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f6bd72e-60f1-4f97-9e7c-2cf694e17736-tmp\") pod \"klusterlet-addon-workmgr-5c6b486f46-5wxf8\" (UID: \"4f6bd72e-60f1-4f97-9e7c-2cf694e17736\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6b486f46-5wxf8"
May 29 13:42:40.205985 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.205941    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4f6bd72e-60f1-4f97-9e7c-2cf694e17736-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c6b486f46-5wxf8\" (UID: \"4f6bd72e-60f1-4f97-9e7c-2cf694e17736\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6b486f46-5wxf8"
May 29 13:42:40.205985 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.205961    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fc6f03b-90e5-442b-a552-c73faeb3a354-ca-trust-extracted\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:40.205985 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.205982    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fc6f03b-90e5-442b-a552-c73faeb3a354-trusted-ca\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:40.206328 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.206008    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05bfd411-03f2-462d-bc97-568b19e3626d-metrics-tls\") pod \"dns-default-7csjl\" (UID: \"05bfd411-03f2-462d-bc97-568b19e3626d\") " pod="openshift-dns/dns-default-7csjl"
May 29 13:42:40.206328 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.206038    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wzjq6\" (UniqueName: \"kubernetes.io/projected/4f6bd72e-60f1-4f97-9e7c-2cf694e17736-kube-api-access-wzjq6\") pod \"klusterlet-addon-workmgr-5c6b486f46-5wxf8\" (UID: \"4f6bd72e-60f1-4f97-9e7c-2cf694e17736\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6b486f46-5wxf8"
May 29 13:42:40.206328 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.206107    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-certificates\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:40.206328 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.206133    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jhwxz\" (UniqueName: \"kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-kube-api-access-jhwxz\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:40.206328 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:40.206159    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 29 13:42:40.206328 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:40.206247    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05bfd411-03f2-462d-bc97-568b19e3626d-metrics-tls podName:05bfd411-03f2-462d-bc97-568b19e3626d nodeName:}" failed. No retries permitted until 2026-05-29 13:42:40.70622075 +0000 UTC m=+35.173290814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05bfd411-03f2-462d-bc97-568b19e3626d-metrics-tls") pod "dns-default-7csjl" (UID: "05bfd411-03f2-462d-bc97-568b19e3626d") : secret "dns-default-metrics-tls" not found
May 29 13:42:40.206614 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.206464    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mlf77\" (UniqueName: \"kubernetes.io/projected/05bfd411-03f2-462d-bc97-568b19e3626d-kube-api-access-mlf77\") pod \"dns-default-7csjl\" (UID: \"05bfd411-03f2-462d-bc97-568b19e3626d\") " pod="openshift-dns/dns-default-7csjl"
May 29 13:42:40.206614 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.206507    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pnzls\" (UniqueName: \"kubernetes.io/projected/29951f01-4535-401e-b698-8096a4fbc290-kube-api-access-pnzls\") pod \"ingress-canary-dn82q\" (UID: \"29951f01-4535-401e-b698-8096a4fbc290\") " pod="openshift-ingress-canary/ingress-canary-dn82q"
May 29 13:42:40.206724 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.206698    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g2mbh\" (UniqueName: \"kubernetes.io/projected/35a676b1-9b9b-4c37-bbeb-2d61e14d69b6-kube-api-access-g2mbh\") pod \"managed-serviceaccount-addon-agent-7c79dcd5cd-ggh69\" (UID: \"35a676b1-9b9b-4c37-bbeb-2d61e14d69b6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c79dcd5cd-ggh69"
May 29 13:42:40.206779 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.206719    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fc6f03b-90e5-442b-a552-c73faeb3a354-ca-trust-extracted\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:40.206779 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.206747    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/dafac8b7-3c8a-4dfd-b399-9988bbdca1f7-ca\") pod \"cluster-proxy-proxy-agent-7d7757b49f-zntlp\" (UID: \"dafac8b7-3c8a-4dfd-b399-9988bbdca1f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp"
May 29 13:42:40.206779 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.206748    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4f6bd72e-60f1-4f97-9e7c-2cf694e17736-tmp\") pod \"klusterlet-addon-workmgr-5c6b486f46-5wxf8\" (UID: \"4f6bd72e-60f1-4f97-9e7c-2cf694e17736\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6b486f46-5wxf8"
May 29 13:42:40.206921 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.206809    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-bound-sa-token\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:40.206921 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.206846    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/dafac8b7-3c8a-4dfd-b399-9988bbdca1f7-hub\") pod \"cluster-proxy-proxy-agent-7d7757b49f-zntlp\" (UID: \"dafac8b7-3c8a-4dfd-b399-9988bbdca1f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp"
May 29 13:42:40.206921 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.206870    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/dafac8b7-3c8a-4dfd-b399-9988bbdca1f7-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-7d7757b49f-zntlp\" (UID: \"dafac8b7-3c8a-4dfd-b399-9988bbdca1f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp"
May 29 13:42:40.206921 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.206900    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/dafac8b7-3c8a-4dfd-b399-9988bbdca1f7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d7757b49f-zntlp\" (UID: \"dafac8b7-3c8a-4dfd-b399-9988bbdca1f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp"
May 29 13:42:40.207840 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.207349    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/05bfd411-03f2-462d-bc97-568b19e3626d-config-volume\") pod \"dns-default-7csjl\" (UID: \"05bfd411-03f2-462d-bc97-568b19e3626d\") " pod="openshift-dns/dns-default-7csjl"
May 29 13:42:40.207840 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.207410    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05bfd411-03f2-462d-bc97-568b19e3626d-tmp-dir\") pod \"dns-default-7csjl\" (UID: \"05bfd411-03f2-462d-bc97-568b19e3626d\") " pod="openshift-dns/dns-default-7csjl"
May 29 13:42:40.207840 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.207445    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29951f01-4535-401e-b698-8096a4fbc290-cert\") pod \"ingress-canary-dn82q\" (UID: \"29951f01-4535-401e-b698-8096a4fbc290\") " pod="openshift-ingress-canary/ingress-canary-dn82q"
May 29 13:42:40.207840 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.207488    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fc6f03b-90e5-442b-a552-c73faeb3a354-image-registry-private-configuration\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:40.207840 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.207505    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-certificates\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:40.207840 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.207525    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dafac8b7-3c8a-4dfd-b399-9988bbdca1f7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d7757b49f-zntlp\" (UID: \"dafac8b7-3c8a-4dfd-b399-9988bbdca1f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp"
May 29 13:42:40.207840 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.207572    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2p9jh\" (UniqueName: \"kubernetes.io/projected/dafac8b7-3c8a-4dfd-b399-9988bbdca1f7-kube-api-access-2p9jh\") pod \"cluster-proxy-proxy-agent-7d7757b49f-zntlp\" (UID: \"dafac8b7-3c8a-4dfd-b399-9988bbdca1f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp"
May 29 13:42:40.207840 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.207618    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35a676b1-9b9b-4c37-bbeb-2d61e14d69b6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c79dcd5cd-ggh69\" (UID: \"35a676b1-9b9b-4c37-bbeb-2d61e14d69b6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c79dcd5cd-ggh69"
May 29 13:42:40.207840 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.207713    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:40.207840 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.207761    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fc6f03b-90e5-442b-a552-c73faeb3a354-installation-pull-secrets\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:40.208363 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.207884    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ocpservice-ca\" (UniqueName: \"kubernetes.io/configmap/dafac8b7-3c8a-4dfd-b399-9988bbdca1f7-ocpservice-ca\") pod \"cluster-proxy-proxy-agent-7d7757b49f-zntlp\" (UID: \"dafac8b7-3c8a-4dfd-b399-9988bbdca1f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp"
May 29 13:42:40.208363 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.208192    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05bfd411-03f2-462d-bc97-568b19e3626d-tmp-dir\") pod \"dns-default-7csjl\" (UID: \"05bfd411-03f2-462d-bc97-568b19e3626d\") " pod="openshift-dns/dns-default-7csjl"
May 29 13:42:40.208783 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.208621    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/05bfd411-03f2-462d-bc97-568b19e3626d-config-volume\") pod \"dns-default-7csjl\" (UID: \"05bfd411-03f2-462d-bc97-568b19e3626d\") " pod="openshift-dns/dns-default-7csjl"
May 29 13:42:40.209175 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:40.209094    2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 29 13:42:40.209267 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:40.209192    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29951f01-4535-401e-b698-8096a4fbc290-cert podName:29951f01-4535-401e-b698-8096a4fbc290 nodeName:}" failed. No retries permitted until 2026-05-29 13:42:40.709172419 +0000 UTC m=+35.176242477 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/29951f01-4535-401e-b698-8096a4fbc290-cert") pod "ingress-canary-dn82q" (UID: "29951f01-4535-401e-b698-8096a4fbc290") : secret "canary-serving-cert" not found
May 29 13:42:40.210302 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.210277    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fc6f03b-90e5-442b-a552-c73faeb3a354-trusted-ca\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:40.212144 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.212115    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fc6f03b-90e5-442b-a552-c73faeb3a354-installation-pull-secrets\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:40.212259 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.212174    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/dafac8b7-3c8a-4dfd-b399-9988bbdca1f7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d7757b49f-zntlp\" (UID: \"dafac8b7-3c8a-4dfd-b399-9988bbdca1f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp"
May 29 13:42:40.212259 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:40.212229    2571 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 29 13:42:40.212259 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:40.212242    2571 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-555f88bbf7-5lxbw: secret "image-registry-tls" not found
May 29 13:42:40.212430 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:40.212304    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls podName:1fc6f03b-90e5-442b-a552-c73faeb3a354 nodeName:}" failed. No retries permitted until 2026-05-29 13:42:40.712286757 +0000 UTC m=+35.179356827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls") pod "image-registry-555f88bbf7-5lxbw" (UID: "1fc6f03b-90e5-442b-a552-c73faeb3a354") : secret "image-registry-tls" not found
May 29 13:42:40.212493 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.212440    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/dafac8b7-3c8a-4dfd-b399-9988bbdca1f7-ca\") pod \"cluster-proxy-proxy-agent-7d7757b49f-zntlp\" (UID: \"dafac8b7-3c8a-4dfd-b399-9988bbdca1f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp"
May 29 13:42:40.213237 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.213211    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/dafac8b7-3c8a-4dfd-b399-9988bbdca1f7-hub\") pod \"cluster-proxy-proxy-agent-7d7757b49f-zntlp\" (UID: \"dafac8b7-3c8a-4dfd-b399-9988bbdca1f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp"
May 29 13:42:40.214295 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.213645    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35a676b1-9b9b-4c37-bbeb-2d61e14d69b6-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7c79dcd5cd-ggh69\" (UID: \"35a676b1-9b9b-4c37-bbeb-2d61e14d69b6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c79dcd5cd-ggh69"
May 29 13:42:40.214295 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.213943    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4f6bd72e-60f1-4f97-9e7c-2cf694e17736-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c6b486f46-5wxf8\" (UID: \"4f6bd72e-60f1-4f97-9e7c-2cf694e17736\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6b486f46-5wxf8"
May 29 13:42:40.214915 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.214887    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fc6f03b-90e5-442b-a552-c73faeb3a354-image-registry-private-configuration\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:40.215465 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.215440    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhwxz\" (UniqueName: \"kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-kube-api-access-jhwxz\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:40.216308 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.216282    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlf77\" (UniqueName: \"kubernetes.io/projected/05bfd411-03f2-462d-bc97-568b19e3626d-kube-api-access-mlf77\") pod \"dns-default-7csjl\" (UID: \"05bfd411-03f2-462d-bc97-568b19e3626d\") " pod="openshift-dns/dns-default-7csjl"
May 29 13:42:40.216398 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.216286    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2mbh\" (UniqueName: \"kubernetes.io/projected/35a676b1-9b9b-4c37-bbeb-2d61e14d69b6-kube-api-access-g2mbh\") pod \"managed-serviceaccount-addon-agent-7c79dcd5cd-ggh69\" (UID: \"35a676b1-9b9b-4c37-bbeb-2d61e14d69b6\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c79dcd5cd-ggh69"
May 29 13:42:40.217360 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.217338    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-bound-sa-token\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:40.217571 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.217552    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p9jh\" (UniqueName: \"kubernetes.io/projected/dafac8b7-3c8a-4dfd-b399-9988bbdca1f7-kube-api-access-2p9jh\") pod \"cluster-proxy-proxy-agent-7d7757b49f-zntlp\" (UID: \"dafac8b7-3c8a-4dfd-b399-9988bbdca1f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp"
May 29 13:42:40.217875 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.217859    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzjq6\" (UniqueName: \"kubernetes.io/projected/4f6bd72e-60f1-4f97-9e7c-2cf694e17736-kube-api-access-wzjq6\") pod \"klusterlet-addon-workmgr-5c6b486f46-5wxf8\" (UID: \"4f6bd72e-60f1-4f97-9e7c-2cf694e17736\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6b486f46-5wxf8"
May 29 13:42:40.218872 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.218846    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnzls\" (UniqueName: \"kubernetes.io/projected/29951f01-4535-401e-b698-8096a4fbc290-kube-api-access-pnzls\") pod \"ingress-canary-dn82q\" (UID: \"29951f01-4535-401e-b698-8096a4fbc290\") " pod="openshift-ingress-canary/ingress-canary-dn82q"
May 29 13:42:40.219325 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.219307    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dafac8b7-3c8a-4dfd-b399-9988bbdca1f7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d7757b49f-zntlp\" (UID: \"dafac8b7-3c8a-4dfd-b399-9988bbdca1f7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp"
May 29 13:42:40.288806 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.288766    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c79dcd5cd-ggh69"
May 29 13:42:40.302021 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.301990    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp"
May 29 13:42:40.329693 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.329625    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6b486f46-5wxf8"
May 29 13:42:40.465471 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.465382    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c79dcd5cd-ggh69"]
May 29 13:42:40.469861 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.469803    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp"]
May 29 13:42:40.474197 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:40.474166    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddafac8b7_3c8a_4dfd_b399_9988bbdca1f7.slice/crio-e1753b60fa4f33c13aa6c363849a76a2dff7e3c92bdee9e6cf28fe30410d268e WatchSource:0}: Error finding container e1753b60fa4f33c13aa6c363849a76a2dff7e3c92bdee9e6cf28fe30410d268e: Status 404 returned error can't find the container with id e1753b60fa4f33c13aa6c363849a76a2dff7e3c92bdee9e6cf28fe30410d268e
May 29 13:42:40.495929 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.495900    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6b486f46-5wxf8"]
May 29 13:42:40.499924 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:40.499897    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f6bd72e_60f1_4f97_9e7c_2cf694e17736.slice/crio-902ccbc340f0b933050019b9ac660538d183e8f9cefc0eacf5fbd1e78a3fa919 WatchSource:0}: Error finding container 902ccbc340f0b933050019b9ac660538d183e8f9cefc0eacf5fbd1e78a3fa919: Status 404 returned error can't find the container with id 902ccbc340f0b933050019b9ac660538d183e8f9cefc0eacf5fbd1e78a3fa919
May 29 13:42:40.712068 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.712027    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29951f01-4535-401e-b698-8096a4fbc290-cert\") pod \"ingress-canary-dn82q\" (UID: \"29951f01-4535-401e-b698-8096a4fbc290\") " pod="openshift-ingress-canary/ingress-canary-dn82q"
May 29 13:42:40.712313 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.712127    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05bfd411-03f2-462d-bc97-568b19e3626d-metrics-tls\") pod \"dns-default-7csjl\" (UID: \"05bfd411-03f2-462d-bc97-568b19e3626d\") " pod="openshift-dns/dns-default-7csjl"
May 29 13:42:40.712313 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:40.712186    2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 29 13:42:40.712313 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:40.712213    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 29 13:42:40.712313 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:40.712249    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29951f01-4535-401e-b698-8096a4fbc290-cert podName:29951f01-4535-401e-b698-8096a4fbc290 nodeName:}" failed. No retries permitted until 2026-05-29 13:42:41.71223069 +0000 UTC m=+36.179300770 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/29951f01-4535-401e-b698-8096a4fbc290-cert") pod "ingress-canary-dn82q" (UID: "29951f01-4535-401e-b698-8096a4fbc290") : secret "canary-serving-cert" not found
May 29 13:42:40.712313 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:40.712268    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05bfd411-03f2-462d-bc97-568b19e3626d-metrics-tls podName:05bfd411-03f2-462d-bc97-568b19e3626d nodeName:}" failed. No retries permitted until 2026-05-29 13:42:41.712259623 +0000 UTC m=+36.179329683 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05bfd411-03f2-462d-bc97-568b19e3626d-metrics-tls") pod "dns-default-7csjl" (UID: "05bfd411-03f2-462d-bc97-568b19e3626d") : secret "dns-default-metrics-tls" not found
May 29 13:42:40.812822 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:40.812746    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:40.813018 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:40.812859    2571 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 29 13:42:40.813018 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:40.812871    2571 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-555f88bbf7-5lxbw: secret "image-registry-tls" not found
May 29 13:42:40.813018 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:40.812928    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls podName:1fc6f03b-90e5-442b-a552-c73faeb3a354 nodeName:}" failed. No retries permitted until 2026-05-29 13:42:41.812913828 +0000 UTC m=+36.279983887 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls") pod "image-registry-555f88bbf7-5lxbw" (UID: "1fc6f03b-90e5-442b-a552-c73faeb3a354") : secret "image-registry-tls" not found
May 29 13:42:41.144332 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:41.144259    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:41.145303 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:41.144259    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:42:41.145303 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:41.144270    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:42:41.147391 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:41.147370    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 29 13:42:41.147498 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:41.147373    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-km4z9\""
May 29 13:42:41.147498 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:41.147463    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 29 13:42:41.147680 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:41.147659    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-chjd5\""
May 29 13:42:41.147829 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:41.147707    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 29 13:42:41.147935 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:41.147921    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 29 13:42:41.472405 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:41.472327    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c79dcd5cd-ggh69" event={"ID":"35a676b1-9b9b-4c37-bbeb-2d61e14d69b6","Type":"ContainerStarted","Data":"2037d71f1fee8937927d62795742d396dfbee65a43a196b0bc580ac12a159bcf"}
May 29 13:42:41.474899 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:41.474866    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6b486f46-5wxf8" event={"ID":"4f6bd72e-60f1-4f97-9e7c-2cf694e17736","Type":"ContainerStarted","Data":"902ccbc340f0b933050019b9ac660538d183e8f9cefc0eacf5fbd1e78a3fa919"}
May 29 13:42:41.479162 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:41.479139    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp" event={"ID":"dafac8b7-3c8a-4dfd-b399-9988bbdca1f7","Type":"ContainerStarted","Data":"e1753b60fa4f33c13aa6c363849a76a2dff7e3c92bdee9e6cf28fe30410d268e"}
May 29 13:42:41.721749 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:41.721712    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29951f01-4535-401e-b698-8096a4fbc290-cert\") pod \"ingress-canary-dn82q\" (UID: \"29951f01-4535-401e-b698-8096a4fbc290\") " pod="openshift-ingress-canary/ingress-canary-dn82q"
May 29 13:42:41.721928 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:41.721799    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05bfd411-03f2-462d-bc97-568b19e3626d-metrics-tls\") pod \"dns-default-7csjl\" (UID: \"05bfd411-03f2-462d-bc97-568b19e3626d\") " pod="openshift-dns/dns-default-7csjl"
May 29 13:42:41.722006 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:41.721975    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 29 13:42:41.722080 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:41.722038    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05bfd411-03f2-462d-bc97-568b19e3626d-metrics-tls podName:05bfd411-03f2-462d-bc97-568b19e3626d nodeName:}" failed. No retries permitted until 2026-05-29 13:42:43.722019171 +0000 UTC m=+38.189089232 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05bfd411-03f2-462d-bc97-568b19e3626d-metrics-tls") pod "dns-default-7csjl" (UID: "05bfd411-03f2-462d-bc97-568b19e3626d") : secret "dns-default-metrics-tls" not found
May 29 13:42:41.722506 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:41.722442    2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 29 13:42:41.722506 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:41.722496    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29951f01-4535-401e-b698-8096a4fbc290-cert podName:29951f01-4535-401e-b698-8096a4fbc290 nodeName:}" failed. No retries permitted until 2026-05-29 13:42:43.722480942 +0000 UTC m=+38.189551019 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/29951f01-4535-401e-b698-8096a4fbc290-cert") pod "ingress-canary-dn82q" (UID: "29951f01-4535-401e-b698-8096a4fbc290") : secret "canary-serving-cert" not found
May 29 13:42:41.822820 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:41.822780    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:41.823167 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:41.823148    2571 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 29 13:42:41.823248 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:41.823172    2571 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-555f88bbf7-5lxbw: secret "image-registry-tls" not found
May 29 13:42:41.823248 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:41.823235    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls podName:1fc6f03b-90e5-442b-a552-c73faeb3a354 nodeName:}" failed. No retries permitted until 2026-05-29 13:42:43.823216396 +0000 UTC m=+38.290286459 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls") pod "image-registry-555f88bbf7-5lxbw" (UID: "1fc6f03b-90e5-442b-a552-c73faeb3a354") : secret "image-registry-tls" not found
May 29 13:42:43.743063 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:43.742795    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29951f01-4535-401e-b698-8096a4fbc290-cert\") pod \"ingress-canary-dn82q\" (UID: \"29951f01-4535-401e-b698-8096a4fbc290\") " pod="openshift-ingress-canary/ingress-canary-dn82q"
May 29 13:42:43.743530 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:43.743128    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05bfd411-03f2-462d-bc97-568b19e3626d-metrics-tls\") pod \"dns-default-7csjl\" (UID: \"05bfd411-03f2-462d-bc97-568b19e3626d\") " pod="openshift-dns/dns-default-7csjl"
May 29 13:42:43.743530 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:43.743303    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 29 13:42:43.743530 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:43.743366    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05bfd411-03f2-462d-bc97-568b19e3626d-metrics-tls podName:05bfd411-03f2-462d-bc97-568b19e3626d nodeName:}" failed. No retries permitted until 2026-05-29 13:42:47.743347453 +0000 UTC m=+42.210417520 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05bfd411-03f2-462d-bc97-568b19e3626d-metrics-tls") pod "dns-default-7csjl" (UID: "05bfd411-03f2-462d-bc97-568b19e3626d") : secret "dns-default-metrics-tls" not found
May 29 13:42:43.743530 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:43.743394    2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 29 13:42:43.743530 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:43.743431    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29951f01-4535-401e-b698-8096a4fbc290-cert podName:29951f01-4535-401e-b698-8096a4fbc290 nodeName:}" failed. No retries permitted until 2026-05-29 13:42:47.743418795 +0000 UTC m=+42.210488859 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/29951f01-4535-401e-b698-8096a4fbc290-cert") pod "ingress-canary-dn82q" (UID: "29951f01-4535-401e-b698-8096a4fbc290") : secret "canary-serving-cert" not found
May 29 13:42:43.844809 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:43.844072    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:43.844809 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:43.844355    2571 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 29 13:42:43.844809 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:43.844374    2571 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-555f88bbf7-5lxbw: secret "image-registry-tls" not found
May 29 13:42:43.844809 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:43.844437    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls podName:1fc6f03b-90e5-442b-a552-c73faeb3a354 nodeName:}" failed. No retries permitted until 2026-05-29 13:42:47.84441801 +0000 UTC m=+42.311488091 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls") pod "image-registry-555f88bbf7-5lxbw" (UID: "1fc6f03b-90e5-442b-a552-c73faeb3a354") : secret "image-registry-tls" not found
May 29 13:42:47.781122 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:47.781076    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29951f01-4535-401e-b698-8096a4fbc290-cert\") pod \"ingress-canary-dn82q\" (UID: \"29951f01-4535-401e-b698-8096a4fbc290\") " pod="openshift-ingress-canary/ingress-canary-dn82q"
May 29 13:42:47.781530 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:47.781157    2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 29 13:42:47.781530 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:47.781183    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05bfd411-03f2-462d-bc97-568b19e3626d-metrics-tls\") pod \"dns-default-7csjl\" (UID: \"05bfd411-03f2-462d-bc97-568b19e3626d\") " pod="openshift-dns/dns-default-7csjl"
May 29 13:42:47.781530 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:47.781220    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29951f01-4535-401e-b698-8096a4fbc290-cert podName:29951f01-4535-401e-b698-8096a4fbc290 nodeName:}" failed. No retries permitted until 2026-05-29 13:42:55.781202944 +0000 UTC m=+50.248273016 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/29951f01-4535-401e-b698-8096a4fbc290-cert") pod "ingress-canary-dn82q" (UID: "29951f01-4535-401e-b698-8096a4fbc290") : secret "canary-serving-cert" not found
May 29 13:42:47.781530 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:47.781266    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 29 13:42:47.781530 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:47.781308    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05bfd411-03f2-462d-bc97-568b19e3626d-metrics-tls podName:05bfd411-03f2-462d-bc97-568b19e3626d nodeName:}" failed. No retries permitted until 2026-05-29 13:42:55.781294703 +0000 UTC m=+50.248364772 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05bfd411-03f2-462d-bc97-568b19e3626d-metrics-tls") pod "dns-default-7csjl" (UID: "05bfd411-03f2-462d-bc97-568b19e3626d") : secret "dns-default-metrics-tls" not found
May 29 13:42:47.882541 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:47.882507    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:47.882703 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:47.882641    2571 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 29 13:42:47.882703 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:47.882659    2571 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-555f88bbf7-5lxbw: secret "image-registry-tls" not found
May 29 13:42:47.882792 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:47.882718    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls podName:1fc6f03b-90e5-442b-a552-c73faeb3a354 nodeName:}" failed. No retries permitted until 2026-05-29 13:42:55.882701091 +0000 UTC m=+50.349771164 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls") pod "image-registry-555f88bbf7-5lxbw" (UID: "1fc6f03b-90e5-442b-a552-c73faeb3a354") : secret "image-registry-tls" not found
May 29 13:42:48.084462 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:48.084392    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d8451cb-e6fd-48aa-a319-bca020321b7a-original-pull-secret\") pod \"global-pull-secret-syncer-w4pw2\" (UID: \"0d8451cb-e6fd-48aa-a319-bca020321b7a\") " pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:48.087618 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:48.087592    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0d8451cb-e6fd-48aa-a319-bca020321b7a-original-pull-secret\") pod \"global-pull-secret-syncer-w4pw2\" (UID: \"0d8451cb-e6fd-48aa-a319-bca020321b7a\") " pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:48.357749 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:48.357665    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w4pw2"
May 29 13:42:51.104308 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:51.104256    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w4pw2"]
May 29 13:42:51.107490 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:42:51.107415    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d8451cb_e6fd_48aa_a319_bca020321b7a.slice/crio-f32c9a1e45a477f45b94f93846f318487fd48fdc6a8624652730b59064ec3747 WatchSource:0}: Error finding container f32c9a1e45a477f45b94f93846f318487fd48fdc6a8624652730b59064ec3747: Status 404 returned error can't find the container with id f32c9a1e45a477f45b94f93846f318487fd48fdc6a8624652730b59064ec3747
May 29 13:42:51.504430 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:51.504338    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6b486f46-5wxf8" event={"ID":"4f6bd72e-60f1-4f97-9e7c-2cf694e17736","Type":"ContainerStarted","Data":"b4885b5abc8fb0945aa077b2f283ac3d621c70473d3f65f525cb05d31e6e217d"}
May 29 13:42:51.504598 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:51.504577    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6b486f46-5wxf8"
May 29 13:42:51.505690 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:51.505659    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c79dcd5cd-ggh69" event={"ID":"35a676b1-9b9b-4c37-bbeb-2d61e14d69b6","Type":"ContainerStarted","Data":"8174f6db8ab3096c9af7a26e17dae7e7980b2fb4c934ab0a1d0a9073ab337fd8"}
May 29 13:42:51.506407 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:51.506388    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6b486f46-5wxf8"
May 29 13:42:51.506712 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:51.506696    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w4pw2" event={"ID":"0d8451cb-e6fd-48aa-a319-bca020321b7a","Type":"ContainerStarted","Data":"f32c9a1e45a477f45b94f93846f318487fd48fdc6a8624652730b59064ec3747"}
May 29 13:42:51.507881 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:51.507859    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp" event={"ID":"dafac8b7-3c8a-4dfd-b399-9988bbdca1f7","Type":"ContainerStarted","Data":"7b187936de7dcd8f852edec1c3bb234b794ee62ff56c14a823bfea97f316d5c6"}
May 29 13:42:51.509871 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:51.509846    2571 generic.go:358] "Generic (PLEG): container finished" podID="1b2d1d27-e735-4a58-a93f-5d0bf642aa78" containerID="3f0a4c5d4953c7f833cd1c5aa897b3ff2c8f7b61baa805618e6fe5cef0870b23" exitCode=0
May 29 13:42:51.509997 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:51.509896    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4vtqj" event={"ID":"1b2d1d27-e735-4a58-a93f-5d0bf642aa78","Type":"ContainerDied","Data":"3f0a4c5d4953c7f833cd1c5aa897b3ff2c8f7b61baa805618e6fe5cef0870b23"}
May 29 13:42:51.523372 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:51.523335    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6b486f46-5wxf8" podStartSLOduration=32.04314982 podStartE2EDuration="42.523322989s" podCreationTimestamp="2026-05-29 13:42:09 +0000 UTC" firstStartedPulling="2026-05-29 13:42:40.50210599 +0000 UTC m=+34.969176050" lastFinishedPulling="2026-05-29 13:42:50.982279149 +0000 UTC m=+45.449349219" observedRunningTime="2026-05-29 13:42:51.522458517 +0000 UTC m=+45.989528600" watchObservedRunningTime="2026-05-29 13:42:51.523322989 +0000 UTC m=+45.990393069"
May 29 13:42:51.538249 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:51.538195    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c79dcd5cd-ggh69" podStartSLOduration=33.029355956 podStartE2EDuration="43.538180786s" podCreationTimestamp="2026-05-29 13:42:08 +0000 UTC" firstStartedPulling="2026-05-29 13:42:40.472376282 +0000 UTC m=+34.939446355" lastFinishedPulling="2026-05-29 13:42:50.981201117 +0000 UTC m=+45.448271185" observedRunningTime="2026-05-29 13:42:51.537816488 +0000 UTC m=+46.004886569" watchObservedRunningTime="2026-05-29 13:42:51.538180786 +0000 UTC m=+46.005250867"
May 29 13:42:52.515609 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:52.515573    2571 generic.go:358] "Generic (PLEG): container finished" podID="1b2d1d27-e735-4a58-a93f-5d0bf642aa78" containerID="4ae3dad9d5bda055b54e0d0eebfbae3ad9faa8b9c984cdfdfd35e8550796872c" exitCode=0
May 29 13:42:52.516107 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:52.515704    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4vtqj" event={"ID":"1b2d1d27-e735-4a58-a93f-5d0bf642aa78","Type":"ContainerDied","Data":"4ae3dad9d5bda055b54e0d0eebfbae3ad9faa8b9c984cdfdfd35e8550796872c"}
May 29 13:42:53.521288 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:53.521248    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4vtqj" event={"ID":"1b2d1d27-e735-4a58-a93f-5d0bf642aa78","Type":"ContainerStarted","Data":"48b1bbb54adc1bab5fbb92fdd3dcb6d839953c6bf68622595ae9d88e77f71a45"}
May 29 13:42:53.546980 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:53.546816    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4vtqj" podStartSLOduration=4.052807035 podStartE2EDuration="47.546801006s" podCreationTimestamp="2026-05-29 13:42:06 +0000 UTC" firstStartedPulling="2026-05-29 13:42:07.487896749 +0000 UTC m=+1.954966808" lastFinishedPulling="2026-05-29 13:42:50.981890706 +0000 UTC m=+45.448960779" observedRunningTime="2026-05-29 13:42:53.545333495 +0000 UTC m=+48.012403601" watchObservedRunningTime="2026-05-29 13:42:53.546801006 +0000 UTC m=+48.013871099"
May 29 13:42:54.525043 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:54.524990    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp" event={"ID":"dafac8b7-3c8a-4dfd-b399-9988bbdca1f7","Type":"ContainerStarted","Data":"d505218daa10ed11a831144be64648982ef4a8abc97621583baa3ddc88e5d493"}
May 29 13:42:54.525043 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:54.525047    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp" event={"ID":"dafac8b7-3c8a-4dfd-b399-9988bbdca1f7","Type":"ContainerStarted","Data":"91a6dea9ad718c31e74988a751c85075f2bd2179f0bedb9a20198485a0361ef6"}
May 29 13:42:54.545426 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:54.545360    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp" podStartSLOduration=32.937394063 podStartE2EDuration="46.545342683s" podCreationTimestamp="2026-05-29 13:42:08 +0000 UTC" firstStartedPulling="2026-05-29 13:42:40.476171294 +0000 UTC m=+34.943241358" lastFinishedPulling="2026-05-29 13:42:54.084119917 +0000 UTC m=+48.551189978" observedRunningTime="2026-05-29 13:42:54.544247853 +0000 UTC m=+49.011317935" watchObservedRunningTime="2026-05-29 13:42:54.545342683 +0000 UTC m=+49.012412765"
May 29 13:42:55.850821 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:55.850786    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05bfd411-03f2-462d-bc97-568b19e3626d-metrics-tls\") pod \"dns-default-7csjl\" (UID: \"05bfd411-03f2-462d-bc97-568b19e3626d\") " pod="openshift-dns/dns-default-7csjl"
May 29 13:42:55.851222 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:55.850933    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29951f01-4535-401e-b698-8096a4fbc290-cert\") pod \"ingress-canary-dn82q\" (UID: \"29951f01-4535-401e-b698-8096a4fbc290\") " pod="openshift-ingress-canary/ingress-canary-dn82q"
May 29 13:42:55.851222 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:55.850933    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 29 13:42:55.851222 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:55.850974    2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 29 13:42:55.851222 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:55.851002    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05bfd411-03f2-462d-bc97-568b19e3626d-metrics-tls podName:05bfd411-03f2-462d-bc97-568b19e3626d nodeName:}" failed. No retries permitted until 2026-05-29 13:43:11.850987952 +0000 UTC m=+66.318058011 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05bfd411-03f2-462d-bc97-568b19e3626d-metrics-tls") pod "dns-default-7csjl" (UID: "05bfd411-03f2-462d-bc97-568b19e3626d") : secret "dns-default-metrics-tls" not found
May 29 13:42:55.851222 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:55.851022    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29951f01-4535-401e-b698-8096a4fbc290-cert podName:29951f01-4535-401e-b698-8096a4fbc290 nodeName:}" failed. No retries permitted until 2026-05-29 13:43:11.851009486 +0000 UTC m=+66.318079544 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/29951f01-4535-401e-b698-8096a4fbc290-cert") pod "ingress-canary-dn82q" (UID: "29951f01-4535-401e-b698-8096a4fbc290") : secret "canary-serving-cert" not found
May 29 13:42:55.952138 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:55.952032    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:42:55.952277 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:55.952168    2571 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 29 13:42:55.952277 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:55.952180    2571 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-555f88bbf7-5lxbw: secret "image-registry-tls" not found
May 29 13:42:55.952277 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:42:55.952231    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls podName:1fc6f03b-90e5-442b-a552-c73faeb3a354 nodeName:}" failed. No retries permitted until 2026-05-29 13:43:11.952217206 +0000 UTC m=+66.419287265 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls") pod "image-registry-555f88bbf7-5lxbw" (UID: "1fc6f03b-90e5-442b-a552-c73faeb3a354") : secret "image-registry-tls" not found
May 29 13:42:56.534760 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:56.534725    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w4pw2" event={"ID":"0d8451cb-e6fd-48aa-a319-bca020321b7a","Type":"ContainerStarted","Data":"925eb37f5c03b097c07debdca8b8e5a2354a58c7997a102e1f56eca49a1b2351"}
May 29 13:42:56.550333 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:42:56.550294    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-w4pw2" podStartSLOduration=36.048115498 podStartE2EDuration="40.550281412s" podCreationTimestamp="2026-05-29 13:42:16 +0000 UTC" firstStartedPulling="2026-05-29 13:42:51.1096785 +0000 UTC m=+45.576748558" lastFinishedPulling="2026-05-29 13:42:55.611844406 +0000 UTC m=+50.078914472" observedRunningTime="2026-05-29 13:42:56.549499379 +0000 UTC m=+51.016569459" watchObservedRunningTime="2026-05-29 13:42:56.550281412 +0000 UTC m=+51.017351493"
May 29 13:43:07.468266 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:43:07.468234    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p2drl"
May 29 13:43:10.952975 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:43:10.952931    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs\") pod \"network-metrics-daemon-n48xk\" (UID: \"2a114752-df85-41bc-b828-17308b5311de\") " pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:43:10.955576 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:43:10.955555    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 29 13:43:10.963388 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:43:10.963369    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 29 13:43:10.963451 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:43:10.963441    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs podName:2a114752-df85-41bc-b828-17308b5311de nodeName:}" failed. No retries permitted until 2026-05-29 13:44:14.963420721 +0000 UTC m=+129.430490780 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs") pod "network-metrics-daemon-n48xk" (UID: "2a114752-df85-41bc-b828-17308b5311de") : secret "metrics-daemon-secret" not found
May 29 13:43:11.054208 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:43:11.054171    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nlmkx\" (UniqueName: \"kubernetes.io/projected/81333a26-7a37-4277-9753-ebd412eb34aa-kube-api-access-nlmkx\") pod \"network-check-target-vhmzh\" (UID: \"81333a26-7a37-4277-9753-ebd412eb34aa\") " pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:43:11.056511 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:43:11.056491    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 29 13:43:11.067159 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:43:11.067140    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 29 13:43:11.078246 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:43:11.078221    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlmkx\" (UniqueName: \"kubernetes.io/projected/81333a26-7a37-4277-9753-ebd412eb34aa-kube-api-access-nlmkx\") pod \"network-check-target-vhmzh\" (UID: \"81333a26-7a37-4277-9753-ebd412eb34aa\") " pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:43:11.173218 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:43:11.173190    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-chjd5\""
May 29 13:43:11.180854 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:43:11.180820    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:43:11.304697 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:43:11.304666    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vhmzh"]
May 29 13:43:11.307533 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:43:11.307506    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81333a26_7a37_4277_9753_ebd412eb34aa.slice/crio-4f771541c3675400baf794cf135aa7a1d55cb38a3f9a147490d0436e0b64f87b WatchSource:0}: Error finding container 4f771541c3675400baf794cf135aa7a1d55cb38a3f9a147490d0436e0b64f87b: Status 404 returned error can't find the container with id 4f771541c3675400baf794cf135aa7a1d55cb38a3f9a147490d0436e0b64f87b
May 29 13:43:11.576715 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:43:11.576637    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vhmzh" event={"ID":"81333a26-7a37-4277-9753-ebd412eb34aa","Type":"ContainerStarted","Data":"4f771541c3675400baf794cf135aa7a1d55cb38a3f9a147490d0436e0b64f87b"}
May 29 13:43:11.862915 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:43:11.862879    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05bfd411-03f2-462d-bc97-568b19e3626d-metrics-tls\") pod \"dns-default-7csjl\" (UID: \"05bfd411-03f2-462d-bc97-568b19e3626d\") " pod="openshift-dns/dns-default-7csjl"
May 29 13:43:11.863110 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:43:11.862954    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29951f01-4535-401e-b698-8096a4fbc290-cert\") pod \"ingress-canary-dn82q\" (UID: \"29951f01-4535-401e-b698-8096a4fbc290\") " pod="openshift-ingress-canary/ingress-canary-dn82q"
May 29 13:43:11.863110 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:43:11.863047    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 29 13:43:11.863244 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:43:11.863133    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05bfd411-03f2-462d-bc97-568b19e3626d-metrics-tls podName:05bfd411-03f2-462d-bc97-568b19e3626d nodeName:}" failed. No retries permitted until 2026-05-29 13:43:43.863118258 +0000 UTC m=+98.330188322 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05bfd411-03f2-462d-bc97-568b19e3626d-metrics-tls") pod "dns-default-7csjl" (UID: "05bfd411-03f2-462d-bc97-568b19e3626d") : secret "dns-default-metrics-tls" not found
May 29 13:43:11.863244 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:43:11.863068    2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 29 13:43:11.863244 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:43:11.863212    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29951f01-4535-401e-b698-8096a4fbc290-cert podName:29951f01-4535-401e-b698-8096a4fbc290 nodeName:}" failed. No retries permitted until 2026-05-29 13:43:43.863195008 +0000 UTC m=+98.330265071 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/29951f01-4535-401e-b698-8096a4fbc290-cert") pod "ingress-canary-dn82q" (UID: "29951f01-4535-401e-b698-8096a4fbc290") : secret "canary-serving-cert" not found
May 29 13:43:11.964008 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:43:11.963971    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:43:11.964403 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:43:11.964132    2571 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 29 13:43:11.964403 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:43:11.964152    2571 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-555f88bbf7-5lxbw: secret "image-registry-tls" not found
May 29 13:43:11.964403 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:43:11.964204    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls podName:1fc6f03b-90e5-442b-a552-c73faeb3a354 nodeName:}" failed. No retries permitted until 2026-05-29 13:43:43.964190458 +0000 UTC m=+98.431260517 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls") pod "image-registry-555f88bbf7-5lxbw" (UID: "1fc6f03b-90e5-442b-a552-c73faeb3a354") : secret "image-registry-tls" not found
May 29 13:43:15.587506 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:43:15.587472    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vhmzh" event={"ID":"81333a26-7a37-4277-9753-ebd412eb34aa","Type":"ContainerStarted","Data":"02052efecd11d8d906c55418ef3086e4c8b9070e70ecdc3ffb856a9797d0b6a7"}
May 29 13:43:15.587962 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:43:15.587614    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:43:15.605606 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:43:15.605565    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vhmzh" podStartSLOduration=65.919520378 podStartE2EDuration="1m9.605551979s" podCreationTimestamp="2026-05-29 13:42:06 +0000 UTC" firstStartedPulling="2026-05-29 13:43:11.309383509 +0000 UTC m=+65.776453582" lastFinishedPulling="2026-05-29 13:43:14.995415124 +0000 UTC m=+69.462485183" observedRunningTime="2026-05-29 13:43:15.604476916 +0000 UTC m=+70.071546998" watchObservedRunningTime="2026-05-29 13:43:15.605551979 +0000 UTC m=+70.072622057"
May 29 13:43:43.918491 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:43:43.918329    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29951f01-4535-401e-b698-8096a4fbc290-cert\") pod \"ingress-canary-dn82q\" (UID: \"29951f01-4535-401e-b698-8096a4fbc290\") " pod="openshift-ingress-canary/ingress-canary-dn82q"
May 29 13:43:43.918491 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:43:43.918403    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05bfd411-03f2-462d-bc97-568b19e3626d-metrics-tls\") pod \"dns-default-7csjl\" (UID: \"05bfd411-03f2-462d-bc97-568b19e3626d\") " pod="openshift-dns/dns-default-7csjl"
May 29 13:43:43.918491 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:43:43.918493    2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 29 13:43:43.919089 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:43:43.918573    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05bfd411-03f2-462d-bc97-568b19e3626d-metrics-tls podName:05bfd411-03f2-462d-bc97-568b19e3626d nodeName:}" failed. No retries permitted until 2026-05-29 13:44:47.918556514 +0000 UTC m=+162.385626572 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05bfd411-03f2-462d-bc97-568b19e3626d-metrics-tls") pod "dns-default-7csjl" (UID: "05bfd411-03f2-462d-bc97-568b19e3626d") : secret "dns-default-metrics-tls" not found
May 29 13:43:43.919089 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:43:43.918496    2571 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 29 13:43:43.919089 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:43:43.918671    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29951f01-4535-401e-b698-8096a4fbc290-cert podName:29951f01-4535-401e-b698-8096a4fbc290 nodeName:}" failed. No retries permitted until 2026-05-29 13:44:47.918653771 +0000 UTC m=+162.385723842 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/29951f01-4535-401e-b698-8096a4fbc290-cert") pod "ingress-canary-dn82q" (UID: "29951f01-4535-401e-b698-8096a4fbc290") : secret "canary-serving-cert" not found
May 29 13:43:44.019625 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:43:44.019581    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:43:44.019739 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:43:44.019721    2571 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
May 29 13:43:44.019779 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:43:44.019742    2571 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-555f88bbf7-5lxbw: secret "image-registry-tls" not found
May 29 13:43:44.019811 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:43:44.019792    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls podName:1fc6f03b-90e5-442b-a552-c73faeb3a354 nodeName:}" failed. No retries permitted until 2026-05-29 13:44:48.01977833 +0000 UTC m=+162.486848388 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls") pod "image-registry-555f88bbf7-5lxbw" (UID: "1fc6f03b-90e5-442b-a552-c73faeb3a354") : secret "image-registry-tls" not found
May 29 13:43:46.592322 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:43:46.592287    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vhmzh"
May 29 13:44:15.049916 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:15.049859    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs\") pod \"network-metrics-daemon-n48xk\" (UID: \"2a114752-df85-41bc-b828-17308b5311de\") " pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:44:15.050442 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:44:15.050011    2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 29 13:44:15.050442 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:44:15.050113    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs podName:2a114752-df85-41bc-b828-17308b5311de nodeName:}" failed. No retries permitted until 2026-05-29 13:46:17.050096957 +0000 UTC m=+251.517167016 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs") pod "network-metrics-daemon-n48xk" (UID: "2a114752-df85-41bc-b828-17308b5311de") : secret "metrics-daemon-secret" not found
May 29 13:44:27.906401 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:27.906373    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xqvkm_5ded8d51-ee6a-492b-a1d4-5afefbf29dec/dns-node-resolver/0.log"
May 29 13:44:28.506875 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:28.506847    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9wzn2_af79b7f2-4958-48fc-ad8d-ef0c731dc642/node-ca/0.log"
May 29 13:44:42.997248 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:44:42.997203    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw" podUID="1fc6f03b-90e5-442b-a552-c73faeb3a354"
May 29 13:44:43.036017 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:44:43.035979    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7csjl" podUID="05bfd411-03f2-462d-bc97-568b19e3626d"
May 29 13:44:43.040131 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:44:43.040109    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-dn82q" podUID="29951f01-4535-401e-b698-8096a4fbc290"
May 29 13:44:43.782338 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:43.782300    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:44:43.782538 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:43.782474    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7csjl"
May 29 13:44:44.165605 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:44:44.165569    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-n48xk" podUID="2a114752-df85-41bc-b828-17308b5311de"
May 29 13:44:48.000398 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:48.000359    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29951f01-4535-401e-b698-8096a4fbc290-cert\") pod \"ingress-canary-dn82q\" (UID: \"29951f01-4535-401e-b698-8096a4fbc290\") " pod="openshift-ingress-canary/ingress-canary-dn82q"
May 29 13:44:48.000764 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:48.000416    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05bfd411-03f2-462d-bc97-568b19e3626d-metrics-tls\") pod \"dns-default-7csjl\" (UID: \"05bfd411-03f2-462d-bc97-568b19e3626d\") " pod="openshift-dns/dns-default-7csjl"
May 29 13:44:48.002647 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:48.002619    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05bfd411-03f2-462d-bc97-568b19e3626d-metrics-tls\") pod \"dns-default-7csjl\" (UID: \"05bfd411-03f2-462d-bc97-568b19e3626d\") " pod="openshift-dns/dns-default-7csjl"
May 29 13:44:48.002750 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:48.002673    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29951f01-4535-401e-b698-8096a4fbc290-cert\") pod \"ingress-canary-dn82q\" (UID: \"29951f01-4535-401e-b698-8096a4fbc290\") " pod="openshift-ingress-canary/ingress-canary-dn82q"
May 29 13:44:48.100998 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:48.100966    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:44:48.103096 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:48.103071    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls\") pod \"image-registry-555f88bbf7-5lxbw\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") " pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:44:48.285762 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:48.285668    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wdfdw\""
May 29 13:44:48.285762 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:48.285682    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zfktx\""
May 29 13:44:48.293378 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:48.293352    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7csjl"
May 29 13:44:48.293449 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:48.293375    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:44:48.425966 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:48.425912    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-555f88bbf7-5lxbw"]
May 29 13:44:48.429407 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:44:48.429379    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fc6f03b_90e5_442b_a552_c73faeb3a354.slice/crio-2eda2a5fdfa1058e020606b85023dc4ce830287d379acbc5ebfe74c80cfa7cc3 WatchSource:0}: Error finding container 2eda2a5fdfa1058e020606b85023dc4ce830287d379acbc5ebfe74c80cfa7cc3: Status 404 returned error can't find the container with id 2eda2a5fdfa1058e020606b85023dc4ce830287d379acbc5ebfe74c80cfa7cc3
May 29 13:44:48.447712 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:48.447688    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7csjl"]
May 29 13:44:48.455979 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:44:48.455957    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05bfd411_03f2_462d_bc97_568b19e3626d.slice/crio-44f9c5bf23c44cfcf965fcb30e8b78806d5612d7321fa7e7d5bacaa1ab1dfc1e WatchSource:0}: Error finding container 44f9c5bf23c44cfcf965fcb30e8b78806d5612d7321fa7e7d5bacaa1ab1dfc1e: Status 404 returned error can't find the container with id 44f9c5bf23c44cfcf965fcb30e8b78806d5612d7321fa7e7d5bacaa1ab1dfc1e
May 29 13:44:48.794461 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:48.794420    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw" event={"ID":"1fc6f03b-90e5-442b-a552-c73faeb3a354","Type":"ContainerStarted","Data":"1379442ae3307c2a73d8035b9df4ca2c2344289eee66f7ab6161c1fc132e7912"}
May 29 13:44:48.794461 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:48.794464    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw" event={"ID":"1fc6f03b-90e5-442b-a552-c73faeb3a354","Type":"ContainerStarted","Data":"2eda2a5fdfa1058e020606b85023dc4ce830287d379acbc5ebfe74c80cfa7cc3"}
May 29 13:44:48.794706 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:48.794564    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:44:48.795600 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:48.795578    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7csjl" event={"ID":"05bfd411-03f2-462d-bc97-568b19e3626d","Type":"ContainerStarted","Data":"44f9c5bf23c44cfcf965fcb30e8b78806d5612d7321fa7e7d5bacaa1ab1dfc1e"}
May 29 13:44:48.819011 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:48.818962    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw" podStartSLOduration=162.818947232 podStartE2EDuration="2m42.818947232s" podCreationTimestamp="2026-05-29 13:42:06 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 13:44:48.818047113 +0000 UTC m=+163.285117195" watchObservedRunningTime="2026-05-29 13:44:48.818947232 +0000 UTC m=+163.286017322"
May 29 13:44:49.318816 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:49.318776    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xw9s8"]
May 29 13:44:49.321929 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:49.321910    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xw9s8"
May 29 13:44:49.323914 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:49.323889    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 29 13:44:49.324330 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:49.324307    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8ccsm\""
May 29 13:44:49.324439 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:49.324405    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 29 13:44:49.324645 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:49.324623    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 29 13:44:49.324818 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:49.324805    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 29 13:44:49.334481 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:49.334461    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xw9s8"]
May 29 13:44:49.409961 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:49.409932    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvk44\" (UniqueName: \"kubernetes.io/projected/88909f40-15e8-4baa-9365-c504b94066c4-kube-api-access-jvk44\") pod \"insights-runtime-extractor-xw9s8\" (UID: \"88909f40-15e8-4baa-9365-c504b94066c4\") " pod="openshift-insights/insights-runtime-extractor-xw9s8"
May 29 13:44:49.410155 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:49.409981    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88909f40-15e8-4baa-9365-c504b94066c4-data-volume\") pod \"insights-runtime-extractor-xw9s8\" (UID: \"88909f40-15e8-4baa-9365-c504b94066c4\") " pod="openshift-insights/insights-runtime-extractor-xw9s8"
May 29 13:44:49.410155 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:49.410132    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88909f40-15e8-4baa-9365-c504b94066c4-crio-socket\") pod \"insights-runtime-extractor-xw9s8\" (UID: \"88909f40-15e8-4baa-9365-c504b94066c4\") " pod="openshift-insights/insights-runtime-extractor-xw9s8"
May 29 13:44:49.410268 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:49.410173    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88909f40-15e8-4baa-9365-c504b94066c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xw9s8\" (UID: \"88909f40-15e8-4baa-9365-c504b94066c4\") " pod="openshift-insights/insights-runtime-extractor-xw9s8"
May 29 13:44:49.410268 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:49.410199    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88909f40-15e8-4baa-9365-c504b94066c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xw9s8\" (UID: \"88909f40-15e8-4baa-9365-c504b94066c4\") " pod="openshift-insights/insights-runtime-extractor-xw9s8"
May 29 13:44:49.510799 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:49.510765    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jvk44\" (UniqueName: \"kubernetes.io/projected/88909f40-15e8-4baa-9365-c504b94066c4-kube-api-access-jvk44\") pod \"insights-runtime-extractor-xw9s8\" (UID: \"88909f40-15e8-4baa-9365-c504b94066c4\") " pod="openshift-insights/insights-runtime-extractor-xw9s8"
May 29 13:44:49.510799 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:49.510804    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88909f40-15e8-4baa-9365-c504b94066c4-data-volume\") pod \"insights-runtime-extractor-xw9s8\" (UID: \"88909f40-15e8-4baa-9365-c504b94066c4\") " pod="openshift-insights/insights-runtime-extractor-xw9s8"
May 29 13:44:49.511023 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:49.510851    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88909f40-15e8-4baa-9365-c504b94066c4-crio-socket\") pod \"insights-runtime-extractor-xw9s8\" (UID: \"88909f40-15e8-4baa-9365-c504b94066c4\") " pod="openshift-insights/insights-runtime-extractor-xw9s8"
May 29 13:44:49.511023 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:49.510868    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88909f40-15e8-4baa-9365-c504b94066c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xw9s8\" (UID: \"88909f40-15e8-4baa-9365-c504b94066c4\") " pod="openshift-insights/insights-runtime-extractor-xw9s8"
May 29 13:44:49.511023 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:49.510884    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88909f40-15e8-4baa-9365-c504b94066c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xw9s8\" (UID: \"88909f40-15e8-4baa-9365-c504b94066c4\") " pod="openshift-insights/insights-runtime-extractor-xw9s8"
May 29 13:44:49.511023 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:49.510963    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/88909f40-15e8-4baa-9365-c504b94066c4-crio-socket\") pod \"insights-runtime-extractor-xw9s8\" (UID: \"88909f40-15e8-4baa-9365-c504b94066c4\") " pod="openshift-insights/insights-runtime-extractor-xw9s8"
May 29 13:44:49.511318 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:49.511207    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/88909f40-15e8-4baa-9365-c504b94066c4-data-volume\") pod \"insights-runtime-extractor-xw9s8\" (UID: \"88909f40-15e8-4baa-9365-c504b94066c4\") " pod="openshift-insights/insights-runtime-extractor-xw9s8"
May 29 13:44:49.511505 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:49.511483    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/88909f40-15e8-4baa-9365-c504b94066c4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xw9s8\" (UID: \"88909f40-15e8-4baa-9365-c504b94066c4\") " pod="openshift-insights/insights-runtime-extractor-xw9s8"
May 29 13:44:49.513244 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:49.513222    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/88909f40-15e8-4baa-9365-c504b94066c4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xw9s8\" (UID: \"88909f40-15e8-4baa-9365-c504b94066c4\") " pod="openshift-insights/insights-runtime-extractor-xw9s8"
May 29 13:44:49.526271 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:49.526191    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvk44\" (UniqueName: \"kubernetes.io/projected/88909f40-15e8-4baa-9365-c504b94066c4-kube-api-access-jvk44\") pod \"insights-runtime-extractor-xw9s8\" (UID: \"88909f40-15e8-4baa-9365-c504b94066c4\") " pod="openshift-insights/insights-runtime-extractor-xw9s8"
May 29 13:44:49.632725 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:49.632694    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xw9s8"
May 29 13:44:50.096740 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:50.096716    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xw9s8"]
May 29 13:44:50.099586 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:44:50.099561    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88909f40_15e8_4baa_9365_c504b94066c4.slice/crio-4de54ae79eec6abc51f165985f7d9526b4bf4fc839e89a819ff5c5925f6bd491 WatchSource:0}: Error finding container 4de54ae79eec6abc51f165985f7d9526b4bf4fc839e89a819ff5c5925f6bd491: Status 404 returned error can't find the container with id 4de54ae79eec6abc51f165985f7d9526b4bf4fc839e89a819ff5c5925f6bd491
May 29 13:44:50.802592 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:50.802560    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7csjl" event={"ID":"05bfd411-03f2-462d-bc97-568b19e3626d","Type":"ContainerStarted","Data":"19f5af1dc6a7208c23f046c249d6defa782e48242d23fb98c61629e98af9d968"}
May 29 13:44:50.802592 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:50.802596    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7csjl" event={"ID":"05bfd411-03f2-462d-bc97-568b19e3626d","Type":"ContainerStarted","Data":"763dd048a81691c03a3839fd8e7da41ce17886bcdf93fdf434b71a07a59163c2"}
May 29 13:44:50.803015 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:50.802666    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7csjl"
May 29 13:44:50.804027 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:50.804007    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xw9s8" event={"ID":"88909f40-15e8-4baa-9365-c504b94066c4","Type":"ContainerStarted","Data":"a1a66580a382a46d848aa8cfa99fc6c9ff086991ccbaa15bab8ac66417e8f139"}
May 29 13:44:50.804130 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:50.804030    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xw9s8" event={"ID":"88909f40-15e8-4baa-9365-c504b94066c4","Type":"ContainerStarted","Data":"d2a361b18ec95cb797f6e02c5a07424a4f3fc1dcad6f2a6a1c83cb3b82fad322"}
May 29 13:44:50.804130 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:50.804039    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xw9s8" event={"ID":"88909f40-15e8-4baa-9365-c504b94066c4","Type":"ContainerStarted","Data":"4de54ae79eec6abc51f165985f7d9526b4bf4fc839e89a819ff5c5925f6bd491"}
May 29 13:44:50.819026 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:50.818987    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-7csjl" podStartSLOduration=130.257295392 podStartE2EDuration="2m11.818975336s" podCreationTimestamp="2026-05-29 13:42:39 +0000 UTC" firstStartedPulling="2026-05-29 13:44:48.457673997 +0000 UTC m=+162.924744060" lastFinishedPulling="2026-05-29 13:44:50.019353934 +0000 UTC m=+164.486424004" observedRunningTime="2026-05-29 13:44:50.818138195 +0000 UTC m=+165.285208278" watchObservedRunningTime="2026-05-29 13:44:50.818975336 +0000 UTC m=+165.286045416"
May 29 13:44:51.504950 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:51.504849    2571 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6b486f46-5wxf8" podUID="4f6bd72e-60f1-4f97-9e7c-2cf694e17736" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.9:8000/readyz\": dial tcp 10.132.0.9:8000: connect: connection refused"
May 29 13:44:51.807896 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:51.807808    2571 generic.go:358] "Generic (PLEG): container finished" podID="4f6bd72e-60f1-4f97-9e7c-2cf694e17736" containerID="b4885b5abc8fb0945aa077b2f283ac3d621c70473d3f65f525cb05d31e6e217d" exitCode=1
May 29 13:44:51.808304 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:51.807887    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6b486f46-5wxf8" event={"ID":"4f6bd72e-60f1-4f97-9e7c-2cf694e17736","Type":"ContainerDied","Data":"b4885b5abc8fb0945aa077b2f283ac3d621c70473d3f65f525cb05d31e6e217d"}
May 29 13:44:51.808304 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:51.808275    2571 scope.go:117] "RemoveContainer" containerID="b4885b5abc8fb0945aa077b2f283ac3d621c70473d3f65f525cb05d31e6e217d"
May 29 13:44:51.809340 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:51.809320    2571 generic.go:358] "Generic (PLEG): container finished" podID="35a676b1-9b9b-4c37-bbeb-2d61e14d69b6" containerID="8174f6db8ab3096c9af7a26e17dae7e7980b2fb4c934ab0a1d0a9073ab337fd8" exitCode=255
May 29 13:44:51.809459 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:51.809399    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c79dcd5cd-ggh69" event={"ID":"35a676b1-9b9b-4c37-bbeb-2d61e14d69b6","Type":"ContainerDied","Data":"8174f6db8ab3096c9af7a26e17dae7e7980b2fb4c934ab0a1d0a9073ab337fd8"}
May 29 13:44:51.809796 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:51.809779    2571 scope.go:117] "RemoveContainer" containerID="8174f6db8ab3096c9af7a26e17dae7e7980b2fb4c934ab0a1d0a9073ab337fd8"
May 29 13:44:52.813098 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:52.813042    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6b486f46-5wxf8" event={"ID":"4f6bd72e-60f1-4f97-9e7c-2cf694e17736","Type":"ContainerStarted","Data":"916558f2bb4a13cd12030539be749b143d86c5bf1c5d2a81c397cc1788d30b6b"}
May 29 13:44:52.813549 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:52.813373    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6b486f46-5wxf8"
May 29 13:44:52.814042 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:52.814008    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6b486f46-5wxf8"
May 29 13:44:52.814593 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:52.814569    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7c79dcd5cd-ggh69" event={"ID":"35a676b1-9b9b-4c37-bbeb-2d61e14d69b6","Type":"ContainerStarted","Data":"819f87466783654c7b3f0fea8d41512f74341d96c40be66eb123d1f67c060914"}
May 29 13:44:52.816165 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:52.816146    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xw9s8" event={"ID":"88909f40-15e8-4baa-9365-c504b94066c4","Type":"ContainerStarted","Data":"92fc7ef6708a5b6378b36a19d6803627e2b13a5c126606da0b36be2eed9d8ce9"}
May 29 13:44:52.880220 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:52.880172    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xw9s8" podStartSLOduration=1.916621414 podStartE2EDuration="3.880158718s" podCreationTimestamp="2026-05-29 13:44:49 +0000 UTC" firstStartedPulling="2026-05-29 13:44:50.168641092 +0000 UTC m=+164.635711151" lastFinishedPulling="2026-05-29 13:44:52.132178395 +0000 UTC m=+166.599248455" observedRunningTime="2026-05-29 13:44:52.87970658 +0000 UTC m=+167.346776660" watchObservedRunningTime="2026-05-29 13:44:52.880158718 +0000 UTC m=+167.347228799"
May 29 13:44:57.143952 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:57.143864    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dn82q"
May 29 13:44:57.143952 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:57.143864    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:44:57.146957 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:57.146939    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-25vw9\""
May 29 13:44:57.155106 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:57.155091    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dn82q"
May 29 13:44:57.279414 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:57.279386    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dn82q"]
May 29 13:44:57.282335 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:44:57.282310    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29951f01_4535_401e_b698_8096a4fbc290.slice/crio-a223131caf0eff86ee247e81257e38bdafc1c3c6bcd5e6dee1aa658bbbc3de2a WatchSource:0}: Error finding container a223131caf0eff86ee247e81257e38bdafc1c3c6bcd5e6dee1aa658bbbc3de2a: Status 404 returned error can't find the container with id a223131caf0eff86ee247e81257e38bdafc1c3c6bcd5e6dee1aa658bbbc3de2a
May 29 13:44:57.830968 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:57.830934    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dn82q" event={"ID":"29951f01-4535-401e-b698-8096a4fbc290","Type":"ContainerStarted","Data":"a223131caf0eff86ee247e81257e38bdafc1c3c6bcd5e6dee1aa658bbbc3de2a"}
May 29 13:44:59.837802 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:59.837763    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dn82q" event={"ID":"29951f01-4535-401e-b698-8096a4fbc290","Type":"ContainerStarted","Data":"7d664717648888ea7a549da0d54aa7102743d934835097de02459c2962329fbf"}
May 29 13:44:59.856038 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:44:59.855996    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dn82q" podStartSLOduration=138.948986666 podStartE2EDuration="2m20.855980513s" podCreationTimestamp="2026-05-29 13:42:39 +0000 UTC" firstStartedPulling="2026-05-29 13:44:57.284214023 +0000 UTC m=+171.751284083" lastFinishedPulling="2026-05-29 13:44:59.19120787 +0000 UTC m=+173.658277930" observedRunningTime="2026-05-29 13:44:59.855221813 +0000 UTC m=+174.322291895" watchObservedRunningTime="2026-05-29 13:44:59.855980513 +0000 UTC m=+174.323050590"
May 29 13:45:00.811949 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:00.811921    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7csjl"
May 29 13:45:04.041619 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.041587    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nvrh8"]
May 29 13:45:04.044695 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.044679    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.049157 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.049138    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 29 13:45:04.051293 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.051278    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 29 13:45:04.051293 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.051284    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 29 13:45:04.052738 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.052721    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 29 13:45:04.052790 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.052764    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pjbq9\""
May 29 13:45:04.055800 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.055785    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 29 13:45:04.055875 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.055790    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 29 13:45:04.225579 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.225547    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49ce8cda-457b-40e1-9f39-1820887e3970-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nvrh8\" (UID: \"49ce8cda-457b-40e1-9f39-1820887e3970\") " pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.225748 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.225586    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/49ce8cda-457b-40e1-9f39-1820887e3970-node-exporter-wtmp\") pod \"node-exporter-nvrh8\" (UID: \"49ce8cda-457b-40e1-9f39-1820887e3970\") " pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.225748 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.225607    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/49ce8cda-457b-40e1-9f39-1820887e3970-node-exporter-tls\") pod \"node-exporter-nvrh8\" (UID: \"49ce8cda-457b-40e1-9f39-1820887e3970\") " pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.225748 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.225729    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hklrg\" (UniqueName: \"kubernetes.io/projected/49ce8cda-457b-40e1-9f39-1820887e3970-kube-api-access-hklrg\") pod \"node-exporter-nvrh8\" (UID: \"49ce8cda-457b-40e1-9f39-1820887e3970\") " pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.225891 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.225772    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49ce8cda-457b-40e1-9f39-1820887e3970-sys\") pod \"node-exporter-nvrh8\" (UID: \"49ce8cda-457b-40e1-9f39-1820887e3970\") " pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.225891 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.225803    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49ce8cda-457b-40e1-9f39-1820887e3970-metrics-client-ca\") pod \"node-exporter-nvrh8\" (UID: \"49ce8cda-457b-40e1-9f39-1820887e3970\") " pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.225891 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.225863    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/49ce8cda-457b-40e1-9f39-1820887e3970-node-exporter-accelerators-collector-config\") pod \"node-exporter-nvrh8\" (UID: \"49ce8cda-457b-40e1-9f39-1820887e3970\") " pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.225891 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.225888    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/49ce8cda-457b-40e1-9f39-1820887e3970-node-exporter-textfile\") pod \"node-exporter-nvrh8\" (UID: \"49ce8cda-457b-40e1-9f39-1820887e3970\") " pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.226023 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.225921    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/49ce8cda-457b-40e1-9f39-1820887e3970-root\") pod \"node-exporter-nvrh8\" (UID: \"49ce8cda-457b-40e1-9f39-1820887e3970\") " pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.327037 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.326942    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49ce8cda-457b-40e1-9f39-1820887e3970-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nvrh8\" (UID: \"49ce8cda-457b-40e1-9f39-1820887e3970\") " pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.327037 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.326986    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/49ce8cda-457b-40e1-9f39-1820887e3970-node-exporter-wtmp\") pod \"node-exporter-nvrh8\" (UID: \"49ce8cda-457b-40e1-9f39-1820887e3970\") " pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.327275 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.327104    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/49ce8cda-457b-40e1-9f39-1820887e3970-node-exporter-tls\") pod \"node-exporter-nvrh8\" (UID: \"49ce8cda-457b-40e1-9f39-1820887e3970\") " pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.327275 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.327141    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/49ce8cda-457b-40e1-9f39-1820887e3970-node-exporter-wtmp\") pod \"node-exporter-nvrh8\" (UID: \"49ce8cda-457b-40e1-9f39-1820887e3970\") " pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.327275 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.327162    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hklrg\" (UniqueName: \"kubernetes.io/projected/49ce8cda-457b-40e1-9f39-1820887e3970-kube-api-access-hklrg\") pod \"node-exporter-nvrh8\" (UID: \"49ce8cda-457b-40e1-9f39-1820887e3970\") " pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.327275 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.327199    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49ce8cda-457b-40e1-9f39-1820887e3970-sys\") pod \"node-exporter-nvrh8\" (UID: \"49ce8cda-457b-40e1-9f39-1820887e3970\") " pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.327275 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.327228    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49ce8cda-457b-40e1-9f39-1820887e3970-metrics-client-ca\") pod \"node-exporter-nvrh8\" (UID: \"49ce8cda-457b-40e1-9f39-1820887e3970\") " pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.327496 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.327279    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/49ce8cda-457b-40e1-9f39-1820887e3970-node-exporter-accelerators-collector-config\") pod \"node-exporter-nvrh8\" (UID: \"49ce8cda-457b-40e1-9f39-1820887e3970\") " pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.327496 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.327309    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/49ce8cda-457b-40e1-9f39-1820887e3970-node-exporter-textfile\") pod \"node-exporter-nvrh8\" (UID: \"49ce8cda-457b-40e1-9f39-1820887e3970\") " pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.327496 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.327321    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49ce8cda-457b-40e1-9f39-1820887e3970-sys\") pod \"node-exporter-nvrh8\" (UID: \"49ce8cda-457b-40e1-9f39-1820887e3970\") " pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.327651 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.327609    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/49ce8cda-457b-40e1-9f39-1820887e3970-root\") pod \"node-exporter-nvrh8\" (UID: \"49ce8cda-457b-40e1-9f39-1820887e3970\") " pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.327733 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.327701    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/49ce8cda-457b-40e1-9f39-1820887e3970-root\") pod \"node-exporter-nvrh8\" (UID: \"49ce8cda-457b-40e1-9f39-1820887e3970\") " pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.327929 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.327900    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/49ce8cda-457b-40e1-9f39-1820887e3970-node-exporter-accelerators-collector-config\") pod \"node-exporter-nvrh8\" (UID: \"49ce8cda-457b-40e1-9f39-1820887e3970\") " pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.327929 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.327921    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/49ce8cda-457b-40e1-9f39-1820887e3970-node-exporter-textfile\") pod \"node-exporter-nvrh8\" (UID: \"49ce8cda-457b-40e1-9f39-1820887e3970\") " pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.328340 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.328316    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/49ce8cda-457b-40e1-9f39-1820887e3970-metrics-client-ca\") pod \"node-exporter-nvrh8\" (UID: \"49ce8cda-457b-40e1-9f39-1820887e3970\") " pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.329415 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.329392    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/49ce8cda-457b-40e1-9f39-1820887e3970-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nvrh8\" (UID: \"49ce8cda-457b-40e1-9f39-1820887e3970\") " pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.329645 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.329625    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/49ce8cda-457b-40e1-9f39-1820887e3970-node-exporter-tls\") pod \"node-exporter-nvrh8\" (UID: \"49ce8cda-457b-40e1-9f39-1820887e3970\") " pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.335250 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.335230    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hklrg\" (UniqueName: \"kubernetes.io/projected/49ce8cda-457b-40e1-9f39-1820887e3970-kube-api-access-hklrg\") pod \"node-exporter-nvrh8\" (UID: \"49ce8cda-457b-40e1-9f39-1820887e3970\") " pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.352646 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.352624    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nvrh8"
May 29 13:45:04.360157 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:45:04.360132    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ce8cda_457b_40e1_9f39_1820887e3970.slice/crio-9005fb18c2ec3778d6c5be1eddbdc1042a4dc8e17658014c2c2d811fac636e6e WatchSource:0}: Error finding container 9005fb18c2ec3778d6c5be1eddbdc1042a4dc8e17658014c2c2d811fac636e6e: Status 404 returned error can't find the container with id 9005fb18c2ec3778d6c5be1eddbdc1042a4dc8e17658014c2c2d811fac636e6e
May 29 13:45:04.854026 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:04.853992    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nvrh8" event={"ID":"49ce8cda-457b-40e1-9f39-1820887e3970","Type":"ContainerStarted","Data":"9005fb18c2ec3778d6c5be1eddbdc1042a4dc8e17658014c2c2d811fac636e6e"}
May 29 13:45:05.857479 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:05.857448    2571 generic.go:358] "Generic (PLEG): container finished" podID="49ce8cda-457b-40e1-9f39-1820887e3970" containerID="7eaa2e97c856962adda8da65b9486d658df71b4c62a073883713d3a69f020e9b" exitCode=0
May 29 13:45:05.857839 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:05.857504    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nvrh8" event={"ID":"49ce8cda-457b-40e1-9f39-1820887e3970","Type":"ContainerDied","Data":"7eaa2e97c856962adda8da65b9486d658df71b4c62a073883713d3a69f020e9b"}
May 29 13:45:06.861722 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:06.861678    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nvrh8" event={"ID":"49ce8cda-457b-40e1-9f39-1820887e3970","Type":"ContainerStarted","Data":"87514df0766a5e47508636c0841c2f4d516c98410ce0c46fdfdcbce574f600b2"}
May 29 13:45:06.862112 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:06.861729    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nvrh8" event={"ID":"49ce8cda-457b-40e1-9f39-1820887e3970","Type":"ContainerStarted","Data":"869b7736a0868fb833b3978743af2b377faa6f802ecab275adff8e8435e46a61"}
May 29 13:45:06.882139 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:06.882031    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nvrh8" podStartSLOduration=1.815752411 podStartE2EDuration="2.882017066s" podCreationTimestamp="2026-05-29 13:45:04 +0000 UTC" firstStartedPulling="2026-05-29 13:45:04.362007206 +0000 UTC m=+178.829077275" lastFinishedPulling="2026-05-29 13:45:05.428271854 +0000 UTC m=+179.895341930" observedRunningTime="2026-05-29 13:45:06.881015031 +0000 UTC m=+181.348085124" watchObservedRunningTime="2026-05-29 13:45:06.882017066 +0000 UTC m=+181.349087146"
May 29 13:45:08.297715 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:08.297679    2571 patch_prober.go:28] interesting pod/image-registry-555f88bbf7-5lxbw container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 29 13:45:08.298121 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:08.297734    2571 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw" podUID="1fc6f03b-90e5-442b-a552-c73faeb3a354" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 29 13:45:09.803321 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:09.803292    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:45:11.557843 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:11.557812    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-555f88bbf7-5lxbw"]
May 29 13:45:20.303991 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:20.303948    2571 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp" podUID="dafac8b7-3c8a-4dfd-b399-9988bbdca1f7" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 29 13:45:30.303490 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:30.303451    2571 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp" podUID="dafac8b7-3c8a-4dfd-b399-9988bbdca1f7" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 29 13:45:36.576313 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.576264    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw" podUID="1fc6f03b-90e5-442b-a552-c73faeb3a354" containerName="registry" containerID="cri-o://1379442ae3307c2a73d8035b9df4ca2c2344289eee66f7ab6161c1fc132e7912" gracePeriod=30
May 29 13:45:36.814701 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.814678    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:45:36.867183 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.867150    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls\") pod \"1fc6f03b-90e5-442b-a552-c73faeb3a354\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") "
May 29 13:45:36.867364 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.867194    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhwxz\" (UniqueName: \"kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-kube-api-access-jhwxz\") pod \"1fc6f03b-90e5-442b-a552-c73faeb3a354\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") "
May 29 13:45:36.867364 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.867214    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-bound-sa-token\") pod \"1fc6f03b-90e5-442b-a552-c73faeb3a354\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") "
May 29 13:45:36.867364 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.867238    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fc6f03b-90e5-442b-a552-c73faeb3a354-installation-pull-secrets\") pod \"1fc6f03b-90e5-442b-a552-c73faeb3a354\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") "
May 29 13:45:36.867529 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.867367    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fc6f03b-90e5-442b-a552-c73faeb3a354-ca-trust-extracted\") pod \"1fc6f03b-90e5-442b-a552-c73faeb3a354\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") "
May 29 13:45:36.867529 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.867412    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-certificates\") pod \"1fc6f03b-90e5-442b-a552-c73faeb3a354\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") "
May 29 13:45:36.867529 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.867450    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fc6f03b-90e5-442b-a552-c73faeb3a354-image-registry-private-configuration\") pod \"1fc6f03b-90e5-442b-a552-c73faeb3a354\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") "
May 29 13:45:36.867529 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.867491    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fc6f03b-90e5-442b-a552-c73faeb3a354-trusted-ca\") pod \"1fc6f03b-90e5-442b-a552-c73faeb3a354\" (UID: \"1fc6f03b-90e5-442b-a552-c73faeb3a354\") "
May 29 13:45:36.868135 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.868035    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fc6f03b-90e5-442b-a552-c73faeb3a354-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "1fc6f03b-90e5-442b-a552-c73faeb3a354" (UID: "1fc6f03b-90e5-442b-a552-c73faeb3a354"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 13:45:36.868256 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.868137    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "1fc6f03b-90e5-442b-a552-c73faeb3a354" (UID: "1fc6f03b-90e5-442b-a552-c73faeb3a354"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 29 13:45:36.869756 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.869727    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "1fc6f03b-90e5-442b-a552-c73faeb3a354" (UID: "1fc6f03b-90e5-442b-a552-c73faeb3a354"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 13:45:36.869873 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.869843    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fc6f03b-90e5-442b-a552-c73faeb3a354-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "1fc6f03b-90e5-442b-a552-c73faeb3a354" (UID: "1fc6f03b-90e5-442b-a552-c73faeb3a354"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 13:45:36.869983 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.869845    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "1fc6f03b-90e5-442b-a552-c73faeb3a354" (UID: "1fc6f03b-90e5-442b-a552-c73faeb3a354"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 13:45:36.870031 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.869976    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-kube-api-access-jhwxz" (OuterVolumeSpecName: "kube-api-access-jhwxz") pod "1fc6f03b-90e5-442b-a552-c73faeb3a354" (UID: "1fc6f03b-90e5-442b-a552-c73faeb3a354"). InnerVolumeSpecName "kube-api-access-jhwxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 13:45:36.870125 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.870085    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fc6f03b-90e5-442b-a552-c73faeb3a354-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "1fc6f03b-90e5-442b-a552-c73faeb3a354" (UID: "1fc6f03b-90e5-442b-a552-c73faeb3a354"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 13:45:36.876245 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.876219    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fc6f03b-90e5-442b-a552-c73faeb3a354-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "1fc6f03b-90e5-442b-a552-c73faeb3a354" (UID: "1fc6f03b-90e5-442b-a552-c73faeb3a354"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 29 13:45:36.939320 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.939283    2571 generic.go:358] "Generic (PLEG): container finished" podID="1fc6f03b-90e5-442b-a552-c73faeb3a354" containerID="1379442ae3307c2a73d8035b9df4ca2c2344289eee66f7ab6161c1fc132e7912" exitCode=0
May 29 13:45:36.939486 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.939367    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw" event={"ID":"1fc6f03b-90e5-442b-a552-c73faeb3a354","Type":"ContainerDied","Data":"1379442ae3307c2a73d8035b9df4ca2c2344289eee66f7ab6161c1fc132e7912"}
May 29 13:45:36.939486 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.939390    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw" event={"ID":"1fc6f03b-90e5-442b-a552-c73faeb3a354","Type":"ContainerDied","Data":"2eda2a5fdfa1058e020606b85023dc4ce830287d379acbc5ebfe74c80cfa7cc3"}
May 29 13:45:36.939486 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.939397    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-555f88bbf7-5lxbw"
May 29 13:45:36.939486 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.939404    2571 scope.go:117] "RemoveContainer" containerID="1379442ae3307c2a73d8035b9df4ca2c2344289eee66f7ab6161c1fc132e7912"
May 29 13:45:36.951030 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.951009    2571 scope.go:117] "RemoveContainer" containerID="1379442ae3307c2a73d8035b9df4ca2c2344289eee66f7ab6161c1fc132e7912"
May 29 13:45:36.951306 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:45:36.951286    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1379442ae3307c2a73d8035b9df4ca2c2344289eee66f7ab6161c1fc132e7912\": container with ID starting with 1379442ae3307c2a73d8035b9df4ca2c2344289eee66f7ab6161c1fc132e7912 not found: ID does not exist" containerID="1379442ae3307c2a73d8035b9df4ca2c2344289eee66f7ab6161c1fc132e7912"
May 29 13:45:36.951400 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.951324    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1379442ae3307c2a73d8035b9df4ca2c2344289eee66f7ab6161c1fc132e7912"} err="failed to get container status \"1379442ae3307c2a73d8035b9df4ca2c2344289eee66f7ab6161c1fc132e7912\": rpc error: code = NotFound desc = could not find container \"1379442ae3307c2a73d8035b9df4ca2c2344289eee66f7ab6161c1fc132e7912\": container with ID starting with 1379442ae3307c2a73d8035b9df4ca2c2344289eee66f7ab6161c1fc132e7912 not found: ID does not exist"
May 29 13:45:36.961749 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.961729    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-555f88bbf7-5lxbw"]
May 29 13:45:36.968873 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.968853    2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-tls\") on node \"ip-10-0-141-1.ec2.internal\" DevicePath \"\""
May 29 13:45:36.968954 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.968877    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhwxz\" (UniqueName: \"kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-kube-api-access-jhwxz\") on node \"ip-10-0-141-1.ec2.internal\" DevicePath \"\""
May 29 13:45:36.968954 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.968887    2571 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1fc6f03b-90e5-442b-a552-c73faeb3a354-bound-sa-token\") on node \"ip-10-0-141-1.ec2.internal\" DevicePath \"\""
May 29 13:45:36.968954 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.968896    2571 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1fc6f03b-90e5-442b-a552-c73faeb3a354-installation-pull-secrets\") on node \"ip-10-0-141-1.ec2.internal\" DevicePath \"\""
May 29 13:45:36.968954 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.968905    2571 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1fc6f03b-90e5-442b-a552-c73faeb3a354-ca-trust-extracted\") on node \"ip-10-0-141-1.ec2.internal\" DevicePath \"\""
May 29 13:45:36.968954 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.968913    2571 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1fc6f03b-90e5-442b-a552-c73faeb3a354-registry-certificates\") on node \"ip-10-0-141-1.ec2.internal\" DevicePath \"\""
May 29 13:45:36.968954 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.968922    2571 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1fc6f03b-90e5-442b-a552-c73faeb3a354-image-registry-private-configuration\") on node \"ip-10-0-141-1.ec2.internal\" DevicePath \"\""
May 29 13:45:36.968954 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.968931    2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1fc6f03b-90e5-442b-a552-c73faeb3a354-trusted-ca\") on node \"ip-10-0-141-1.ec2.internal\" DevicePath \"\""
May 29 13:45:36.969801 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:36.969783    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-555f88bbf7-5lxbw"]
May 29 13:45:38.147007 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:38.146978    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fc6f03b-90e5-442b-a552-c73faeb3a354" path="/var/lib/kubelet/pods/1fc6f03b-90e5-442b-a552-c73faeb3a354/volumes"
May 29 13:45:40.303880 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:40.303841    2571 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp" podUID="dafac8b7-3c8a-4dfd-b399-9988bbdca1f7" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
May 29 13:45:40.304267 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:40.303914    2571 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp"
May 29 13:45:40.304398 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:40.304381    2571 kuberuntime_manager.go:1107] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"d505218daa10ed11a831144be64648982ef4a8abc97621583baa3ddc88e5d493"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp" containerMessage="Container service-proxy failed liveness probe, will be restarted"
May 29 13:45:40.304434 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:40.304417    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp" podUID="dafac8b7-3c8a-4dfd-b399-9988bbdca1f7" containerName="service-proxy" containerID="cri-o://d505218daa10ed11a831144be64648982ef4a8abc97621583baa3ddc88e5d493" gracePeriod=30
May 29 13:45:40.952615 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:40.952582    2571 generic.go:358] "Generic (PLEG): container finished" podID="dafac8b7-3c8a-4dfd-b399-9988bbdca1f7" containerID="d505218daa10ed11a831144be64648982ef4a8abc97621583baa3ddc88e5d493" exitCode=2
May 29 13:45:40.952787 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:40.952645    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp" event={"ID":"dafac8b7-3c8a-4dfd-b399-9988bbdca1f7","Type":"ContainerDied","Data":"d505218daa10ed11a831144be64648982ef4a8abc97621583baa3ddc88e5d493"}
May 29 13:45:40.952787 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:45:40.952680    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7757b49f-zntlp" event={"ID":"dafac8b7-3c8a-4dfd-b399-9988bbdca1f7","Type":"ContainerStarted","Data":"64f6262e654033e84f341d721c2010b4b0cd3b1d9a7aa4f0c7e661fde76d6f1d"}
May 29 13:46:17.076780 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:46:17.076734    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs\") pod \"network-metrics-daemon-n48xk\" (UID: \"2a114752-df85-41bc-b828-17308b5311de\") " pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:46:17.079015 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:46:17.078992    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a114752-df85-41bc-b828-17308b5311de-metrics-certs\") pod \"network-metrics-daemon-n48xk\" (UID: \"2a114752-df85-41bc-b828-17308b5311de\") " pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:46:17.247776 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:46:17.247746    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-km4z9\""
May 29 13:46:17.255681 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:46:17.255660    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n48xk"
May 29 13:46:17.377951 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:46:17.377916    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n48xk"]
May 29 13:46:17.381355 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:46:17.381328    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a114752_df85_41bc_b828_17308b5311de.slice/crio-8bec0c7fbf3cb3f44be33c41d34800122eb7e66480563fc91ee83db5110bef3b WatchSource:0}: Error finding container 8bec0c7fbf3cb3f44be33c41d34800122eb7e66480563fc91ee83db5110bef3b: Status 404 returned error can't find the container with id 8bec0c7fbf3cb3f44be33c41d34800122eb7e66480563fc91ee83db5110bef3b
May 29 13:46:18.048241 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:46:18.048197    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n48xk" event={"ID":"2a114752-df85-41bc-b828-17308b5311de","Type":"ContainerStarted","Data":"8bec0c7fbf3cb3f44be33c41d34800122eb7e66480563fc91ee83db5110bef3b"}
May 29 13:46:19.052279 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:46:19.052238    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n48xk" event={"ID":"2a114752-df85-41bc-b828-17308b5311de","Type":"ContainerStarted","Data":"6466546f6fcbcf3904e9ce3e4044aa8220635547ee0af094aed1b4df23a160b8"}
May 29 13:46:19.052279 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:46:19.052277    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n48xk" event={"ID":"2a114752-df85-41bc-b828-17308b5311de","Type":"ContainerStarted","Data":"38fe69a657ee8256909e97e4021a74c1ed37d44c8b38a93ff64121ef1d55f207"}
May 29 13:46:19.070296 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:46:19.070245    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-n48xk" podStartSLOduration=252.114968466 podStartE2EDuration="4m13.070230402s" podCreationTimestamp="2026-05-29 13:42:06 +0000 UTC" firstStartedPulling="2026-05-29 13:46:17.383140063 +0000 UTC m=+251.850210125" lastFinishedPulling="2026-05-29 13:46:18.338401998 +0000 UTC m=+252.805472061" observedRunningTime="2026-05-29 13:46:19.068188084 +0000 UTC m=+253.535258165" watchObservedRunningTime="2026-05-29 13:46:19.070230402 +0000 UTC m=+253.537300482"
May 29 13:47:06.058920 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:47:06.058886    2571 kubelet.go:1628] "Image garbage collection succeeded"
May 29 13:49:08.076233 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:08.076183    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-587ccfb98-5h6z4"]
May 29 13:49:08.076768 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:08.076515    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fc6f03b-90e5-442b-a552-c73faeb3a354" containerName="registry"
May 29 13:49:08.076768 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:08.076534    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fc6f03b-90e5-442b-a552-c73faeb3a354" containerName="registry"
May 29 13:49:08.076768 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:08.076588    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="1fc6f03b-90e5-442b-a552-c73faeb3a354" containerName="registry"
May 29 13:49:08.079363 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:08.079343    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-587ccfb98-5h6z4"
May 29 13:49:08.082245 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:08.082223    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-h2pbx\""
May 29 13:49:08.082356 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:08.082224    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
May 29 13:49:08.082811 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:08.082796    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
May 29 13:49:08.087552 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:08.087527    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-587ccfb98-5h6z4"]
May 29 13:49:08.133337 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:08.133294    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk9px\" (UniqueName: \"kubernetes.io/projected/2d61fe16-2ed2-4daa-b6e3-80206069f240-kube-api-access-kk9px\") pod \"cert-manager-webhook-587ccfb98-5h6z4\" (UID: \"2d61fe16-2ed2-4daa-b6e3-80206069f240\") " pod="cert-manager/cert-manager-webhook-587ccfb98-5h6z4"
May 29 13:49:08.133337 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:08.133338    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d61fe16-2ed2-4daa-b6e3-80206069f240-bound-sa-token\") pod \"cert-manager-webhook-587ccfb98-5h6z4\" (UID: \"2d61fe16-2ed2-4daa-b6e3-80206069f240\") " pod="cert-manager/cert-manager-webhook-587ccfb98-5h6z4"
May 29 13:49:08.234678 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:08.234642    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kk9px\" (UniqueName: \"kubernetes.io/projected/2d61fe16-2ed2-4daa-b6e3-80206069f240-kube-api-access-kk9px\") pod \"cert-manager-webhook-587ccfb98-5h6z4\" (UID: \"2d61fe16-2ed2-4daa-b6e3-80206069f240\") " pod="cert-manager/cert-manager-webhook-587ccfb98-5h6z4"
May 29 13:49:08.234678 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:08.234681    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d61fe16-2ed2-4daa-b6e3-80206069f240-bound-sa-token\") pod \"cert-manager-webhook-587ccfb98-5h6z4\" (UID: \"2d61fe16-2ed2-4daa-b6e3-80206069f240\") " pod="cert-manager/cert-manager-webhook-587ccfb98-5h6z4"
May 29 13:49:08.242997 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:08.242972    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2d61fe16-2ed2-4daa-b6e3-80206069f240-bound-sa-token\") pod \"cert-manager-webhook-587ccfb98-5h6z4\" (UID: \"2d61fe16-2ed2-4daa-b6e3-80206069f240\") " pod="cert-manager/cert-manager-webhook-587ccfb98-5h6z4"
May 29 13:49:08.243145 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:08.243128    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk9px\" (UniqueName: \"kubernetes.io/projected/2d61fe16-2ed2-4daa-b6e3-80206069f240-kube-api-access-kk9px\") pod \"cert-manager-webhook-587ccfb98-5h6z4\" (UID: \"2d61fe16-2ed2-4daa-b6e3-80206069f240\") " pod="cert-manager/cert-manager-webhook-587ccfb98-5h6z4"
May 29 13:49:08.388404 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:08.388372    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-587ccfb98-5h6z4"
May 29 13:49:08.508616 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:08.508587    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-587ccfb98-5h6z4"]
May 29 13:49:08.511599 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:49:08.511574    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d61fe16_2ed2_4daa_b6e3_80206069f240.slice/crio-94352b95a72a4ce686d7b1b4f271bb2510f998a8d7e8febfc95c2a7e25fc6396 WatchSource:0}: Error finding container 94352b95a72a4ce686d7b1b4f271bb2510f998a8d7e8febfc95c2a7e25fc6396: Status 404 returned error can't find the container with id 94352b95a72a4ce686d7b1b4f271bb2510f998a8d7e8febfc95c2a7e25fc6396
May 29 13:49:08.513368 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:08.513355    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 29 13:49:09.464808 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:09.464767    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-587ccfb98-5h6z4" event={"ID":"2d61fe16-2ed2-4daa-b6e3-80206069f240","Type":"ContainerStarted","Data":"94352b95a72a4ce686d7b1b4f271bb2510f998a8d7e8febfc95c2a7e25fc6396"}
May 29 13:49:11.471393 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:11.471292    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-587ccfb98-5h6z4" event={"ID":"2d61fe16-2ed2-4daa-b6e3-80206069f240","Type":"ContainerStarted","Data":"b41b308017fc1b0e534a901b8d34a7f3db96ec59b3907083de8b073b2d0a8911"}
May 29 13:49:11.471393 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:11.471351    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-587ccfb98-5h6z4"
May 29 13:49:11.488750 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:11.488698    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-587ccfb98-5h6z4" podStartSLOduration=0.782119298 podStartE2EDuration="3.488683655s" podCreationTimestamp="2026-05-29 13:49:08 +0000 UTC" firstStartedPulling="2026-05-29 13:49:08.513476245 +0000 UTC m=+422.980546304" lastFinishedPulling="2026-05-29 13:49:11.220040598 +0000 UTC m=+425.687110661" observedRunningTime="2026-05-29 13:49:11.487372251 +0000 UTC m=+425.954442331" watchObservedRunningTime="2026-05-29 13:49:11.488683655 +0000 UTC m=+425.955753736"
May 29 13:49:17.476930 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:17.476899    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-587ccfb98-5h6z4"
May 29 13:49:41.416345 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:41.416255    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cf69667cd-pglk6"]
May 29 13:49:41.420654 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:41.420630    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cf69667cd-pglk6"
May 29 13:49:41.423181 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:41.423158    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
May 29 13:49:41.423298 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:41.423222    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zrrlh\""
May 29 13:49:41.423298 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:41.423230    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
May 29 13:49:41.423561 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:41.423531    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
May 29 13:49:41.423638 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:41.423534    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
May 29 13:49:41.433271 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:41.433248    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cf69667cd-pglk6"]
May 29 13:49:41.464467 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:41.464440    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/326d47a7-e9cf-4834-9b3e-11812596c49e-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cf69667cd-pglk6\" (UID: \"326d47a7-e9cf-4834-9b3e-11812596c49e\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf69667cd-pglk6"
May 29 13:49:41.464597 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:41.464474    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/326d47a7-e9cf-4834-9b3e-11812596c49e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cf69667cd-pglk6\" (UID: \"326d47a7-e9cf-4834-9b3e-11812596c49e\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf69667cd-pglk6"
May 29 13:49:41.464597 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:41.464494    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/326d47a7-e9cf-4834-9b3e-11812596c49e-tmp\") pod \"opendatahub-operator-controller-manager-5cf69667cd-pglk6\" (UID: \"326d47a7-e9cf-4834-9b3e-11812596c49e\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf69667cd-pglk6"
May 29 13:49:41.464597 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:41.464542    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/326d47a7-e9cf-4834-9b3e-11812596c49e-manifests\") pod \"opendatahub-operator-controller-manager-5cf69667cd-pglk6\" (UID: \"326d47a7-e9cf-4834-9b3e-11812596c49e\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf69667cd-pglk6"
May 29 13:49:41.464597 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:41.464592    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgqp9\" (UniqueName: \"kubernetes.io/projected/326d47a7-e9cf-4834-9b3e-11812596c49e-kube-api-access-mgqp9\") pod \"opendatahub-operator-controller-manager-5cf69667cd-pglk6\" (UID: \"326d47a7-e9cf-4834-9b3e-11812596c49e\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf69667cd-pglk6"
May 29 13:49:41.565749 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:41.565712    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mgqp9\" (UniqueName: \"kubernetes.io/projected/326d47a7-e9cf-4834-9b3e-11812596c49e-kube-api-access-mgqp9\") pod \"opendatahub-operator-controller-manager-5cf69667cd-pglk6\" (UID: \"326d47a7-e9cf-4834-9b3e-11812596c49e\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf69667cd-pglk6"
May 29 13:49:41.565749 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:41.565751    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/326d47a7-e9cf-4834-9b3e-11812596c49e-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cf69667cd-pglk6\" (UID: \"326d47a7-e9cf-4834-9b3e-11812596c49e\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf69667cd-pglk6"
May 29 13:49:41.565999 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:41.565783    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/326d47a7-e9cf-4834-9b3e-11812596c49e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cf69667cd-pglk6\" (UID: \"326d47a7-e9cf-4834-9b3e-11812596c49e\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf69667cd-pglk6"
May 29 13:49:41.565999 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:41.565801    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/326d47a7-e9cf-4834-9b3e-11812596c49e-tmp\") pod \"opendatahub-operator-controller-manager-5cf69667cd-pglk6\" (UID: \"326d47a7-e9cf-4834-9b3e-11812596c49e\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf69667cd-pglk6"
May 29 13:49:41.565999 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:41.565824    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/326d47a7-e9cf-4834-9b3e-11812596c49e-manifests\") pod \"opendatahub-operator-controller-manager-5cf69667cd-pglk6\" (UID: \"326d47a7-e9cf-4834-9b3e-11812596c49e\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf69667cd-pglk6"
May 29 13:49:41.566313 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:41.566296    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/326d47a7-e9cf-4834-9b3e-11812596c49e-manifests\") pod \"opendatahub-operator-controller-manager-5cf69667cd-pglk6\" (UID: \"326d47a7-e9cf-4834-9b3e-11812596c49e\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf69667cd-pglk6"
May 29 13:49:41.568279 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:41.568251    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/326d47a7-e9cf-4834-9b3e-11812596c49e-tmp\") pod \"opendatahub-operator-controller-manager-5cf69667cd-pglk6\" (UID: \"326d47a7-e9cf-4834-9b3e-11812596c49e\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf69667cd-pglk6"
May 29 13:49:41.568390 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:41.568326    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/326d47a7-e9cf-4834-9b3e-11812596c49e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cf69667cd-pglk6\" (UID: \"326d47a7-e9cf-4834-9b3e-11812596c49e\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf69667cd-pglk6"
May 29 13:49:41.568452 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:41.568415    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/326d47a7-e9cf-4834-9b3e-11812596c49e-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cf69667cd-pglk6\" (UID: \"326d47a7-e9cf-4834-9b3e-11812596c49e\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf69667cd-pglk6"
May 29 13:49:41.574311 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:41.574288    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgqp9\" (UniqueName: \"kubernetes.io/projected/326d47a7-e9cf-4834-9b3e-11812596c49e-kube-api-access-mgqp9\") pod \"opendatahub-operator-controller-manager-5cf69667cd-pglk6\" (UID: \"326d47a7-e9cf-4834-9b3e-11812596c49e\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf69667cd-pglk6"
May 29 13:49:41.730286 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:41.730189    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cf69667cd-pglk6"
May 29 13:49:41.856555 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:41.856530    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cf69667cd-pglk6"]
May 29 13:49:41.859025 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:49:41.858998    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod326d47a7_e9cf_4834_9b3e_11812596c49e.slice/crio-5089a10aa935cddf9481b1880d36bbd5cd78d12bcbdf1f872b8ba5f49b5b5b2b WatchSource:0}: Error finding container 5089a10aa935cddf9481b1880d36bbd5cd78d12bcbdf1f872b8ba5f49b5b5b2b: Status 404 returned error can't find the container with id 5089a10aa935cddf9481b1880d36bbd5cd78d12bcbdf1f872b8ba5f49b5b5b2b
May 29 13:49:42.548671 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:42.548623    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cf69667cd-pglk6" event={"ID":"326d47a7-e9cf-4834-9b3e-11812596c49e","Type":"ContainerStarted","Data":"5089a10aa935cddf9481b1880d36bbd5cd78d12bcbdf1f872b8ba5f49b5b5b2b"}
May 29 13:49:45.559632 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:45.559601    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cf69667cd-pglk6" event={"ID":"326d47a7-e9cf-4834-9b3e-11812596c49e","Type":"ContainerStarted","Data":"b76e3b6b4c9e0c1308641013cdd790856df65042723ca29aab4b2bc54c8fadf0"}
May 29 13:49:46.563234 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:46.563192    2571 generic.go:358] "Generic (PLEG): container finished" podID="326d47a7-e9cf-4834-9b3e-11812596c49e" containerID="b76e3b6b4c9e0c1308641013cdd790856df65042723ca29aab4b2bc54c8fadf0" exitCode=0
May 29 13:49:46.563627 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:46.563278    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cf69667cd-pglk6" event={"ID":"326d47a7-e9cf-4834-9b3e-11812596c49e","Type":"ContainerDied","Data":"b76e3b6b4c9e0c1308641013cdd790856df65042723ca29aab4b2bc54c8fadf0"}
May 29 13:49:47.567124 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:47.567089    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cf69667cd-pglk6" event={"ID":"326d47a7-e9cf-4834-9b3e-11812596c49e","Type":"ContainerStarted","Data":"97e9dbc354f4b3336824958880e0c935ae196e3771e517dd95b522fc8fa400f8"}
May 29 13:49:47.567522 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:47.567238    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5cf69667cd-pglk6"
May 29 13:49:47.590486 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:47.590435    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5cf69667cd-pglk6" podStartSLOduration=1.47788776 podStartE2EDuration="6.590422295s" podCreationTimestamp="2026-05-29 13:49:41 +0000 UTC" firstStartedPulling="2026-05-29 13:49:41.860701137 +0000 UTC m=+456.327771197" lastFinishedPulling="2026-05-29 13:49:46.97323567 +0000 UTC m=+461.440305732" observedRunningTime="2026-05-29 13:49:47.589226489 +0000 UTC m=+462.056296571" watchObservedRunningTime="2026-05-29 13:49:47.590422295 +0000 UTC m=+462.057492376"
May 29 13:49:58.571227 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:49:58.571196    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5cf69667cd-pglk6"
May 29 13:50:00.900487 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:00.900457    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-76588b95-g42dw"]
May 29 13:50:00.907669 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:00.907647    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-76588b95-g42dw"
May 29 13:50:00.910034 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:00.910006    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-9w66d\""
May 29 13:50:00.910169 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:00.910151    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
May 29 13:50:00.919369 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:00.919341    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-76588b95-g42dw"]
May 29 13:50:01.012611 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:01.012578    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5ngs\" (UniqueName: \"kubernetes.io/projected/59222dfa-c8a8-4d60-91eb-467a6c42f78a-kube-api-access-r5ngs\") pod \"odh-model-controller-76588b95-g42dw\" (UID: \"59222dfa-c8a8-4d60-91eb-467a6c42f78a\") " pod="opendatahub/odh-model-controller-76588b95-g42dw"
May 29 13:50:01.012778 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:01.012632    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59222dfa-c8a8-4d60-91eb-467a6c42f78a-cert\") pod \"odh-model-controller-76588b95-g42dw\" (UID: \"59222dfa-c8a8-4d60-91eb-467a6c42f78a\") " pod="opendatahub/odh-model-controller-76588b95-g42dw"
May 29 13:50:01.113797 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:01.113757    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r5ngs\" (UniqueName: \"kubernetes.io/projected/59222dfa-c8a8-4d60-91eb-467a6c42f78a-kube-api-access-r5ngs\") pod \"odh-model-controller-76588b95-g42dw\" (UID: \"59222dfa-c8a8-4d60-91eb-467a6c42f78a\") " pod="opendatahub/odh-model-controller-76588b95-g42dw"
May 29 13:50:01.113982 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:01.113820    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59222dfa-c8a8-4d60-91eb-467a6c42f78a-cert\") pod \"odh-model-controller-76588b95-g42dw\" (UID: \"59222dfa-c8a8-4d60-91eb-467a6c42f78a\") " pod="opendatahub/odh-model-controller-76588b95-g42dw"
May 29 13:50:01.113982 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:50:01.113952    2571 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
May 29 13:50:01.114121 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:50:01.114032    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59222dfa-c8a8-4d60-91eb-467a6c42f78a-cert podName:59222dfa-c8a8-4d60-91eb-467a6c42f78a nodeName:}" failed. No retries permitted until 2026-05-29 13:50:01.614011493 +0000 UTC m=+476.081081553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/59222dfa-c8a8-4d60-91eb-467a6c42f78a-cert") pod "odh-model-controller-76588b95-g42dw" (UID: "59222dfa-c8a8-4d60-91eb-467a6c42f78a") : secret "odh-model-controller-webhook-cert" not found
May 29 13:50:01.122502 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:01.122480    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5ngs\" (UniqueName: \"kubernetes.io/projected/59222dfa-c8a8-4d60-91eb-467a6c42f78a-kube-api-access-r5ngs\") pod \"odh-model-controller-76588b95-g42dw\" (UID: \"59222dfa-c8a8-4d60-91eb-467a6c42f78a\") " pod="opendatahub/odh-model-controller-76588b95-g42dw"
May 29 13:50:01.617986 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:01.617946    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59222dfa-c8a8-4d60-91eb-467a6c42f78a-cert\") pod \"odh-model-controller-76588b95-g42dw\" (UID: \"59222dfa-c8a8-4d60-91eb-467a6c42f78a\") " pod="opendatahub/odh-model-controller-76588b95-g42dw"
May 29 13:50:01.620251 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:01.620225    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/59222dfa-c8a8-4d60-91eb-467a6c42f78a-cert\") pod \"odh-model-controller-76588b95-g42dw\" (UID: \"59222dfa-c8a8-4d60-91eb-467a6c42f78a\") " pod="opendatahub/odh-model-controller-76588b95-g42dw"
May 29 13:50:01.799087 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:01.799034    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-855f784d5b-5fqkq"]
May 29 13:50:01.802097 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:01.802082    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-855f784d5b-5fqkq"
May 29 13:50:01.806336 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:01.806318    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
May 29 13:50:01.806790 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:01.806774    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
May 29 13:50:01.806848 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:01.806779    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-sscw8\""
May 29 13:50:01.808179 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:01.808160    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
May 29 13:50:01.812148 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:01.812131    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
May 29 13:50:01.816971 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:01.816955    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-76588b95-g42dw"
May 29 13:50:01.835623 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:01.835595    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-855f784d5b-5fqkq"]
May 29 13:50:01.919559 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:01.919529    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xzg2\" (UniqueName: \"kubernetes.io/projected/deae1b19-73c8-40fc-a0cc-e7663d86c726-kube-api-access-2xzg2\") pod \"kube-auth-proxy-855f784d5b-5fqkq\" (UID: \"deae1b19-73c8-40fc-a0cc-e7663d86c726\") " pod="openshift-ingress/kube-auth-proxy-855f784d5b-5fqkq"
May 29 13:50:01.919929 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:01.919576    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/deae1b19-73c8-40fc-a0cc-e7663d86c726-tmp\") pod \"kube-auth-proxy-855f784d5b-5fqkq\" (UID: \"deae1b19-73c8-40fc-a0cc-e7663d86c726\") " pod="openshift-ingress/kube-auth-proxy-855f784d5b-5fqkq"
May 29 13:50:01.919929 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:01.919606    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deae1b19-73c8-40fc-a0cc-e7663d86c726-tls-certs\") pod \"kube-auth-proxy-855f784d5b-5fqkq\" (UID: \"deae1b19-73c8-40fc-a0cc-e7663d86c726\") " pod="openshift-ingress/kube-auth-proxy-855f784d5b-5fqkq"
May 29 13:50:01.946814 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:01.946778    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-76588b95-g42dw"]
May 29 13:50:01.950751 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:50:01.950717    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59222dfa_c8a8_4d60_91eb_467a6c42f78a.slice/crio-6b8314fd0b9fb0427209ed37018db3ddbabcab02ea373710ce8932356ed1b861 WatchSource:0}: Error finding container 6b8314fd0b9fb0427209ed37018db3ddbabcab02ea373710ce8932356ed1b861: Status 404 returned error can't find the container with id 6b8314fd0b9fb0427209ed37018db3ddbabcab02ea373710ce8932356ed1b861
May 29 13:50:01.963944 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:01.963919    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-6577596cd5-dfvjf"]
May 29 13:50:01.967933 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:01.967917    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-6577596cd5-dfvjf"
May 29 13:50:01.970614 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:01.970596    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-pq4nv\""
May 29 13:50:01.971339 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:01.971320    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
May 29 13:50:01.980127 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:01.980096    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-6577596cd5-dfvjf"]
May 29 13:50:02.020769 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:02.020734    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2xzg2\" (UniqueName: \"kubernetes.io/projected/deae1b19-73c8-40fc-a0cc-e7663d86c726-kube-api-access-2xzg2\") pod \"kube-auth-proxy-855f784d5b-5fqkq\" (UID: \"deae1b19-73c8-40fc-a0cc-e7663d86c726\") " pod="openshift-ingress/kube-auth-proxy-855f784d5b-5fqkq"
May 29 13:50:02.020926 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:02.020779    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/deae1b19-73c8-40fc-a0cc-e7663d86c726-tmp\") pod \"kube-auth-proxy-855f784d5b-5fqkq\" (UID: \"deae1b19-73c8-40fc-a0cc-e7663d86c726\") " pod="openshift-ingress/kube-auth-proxy-855f784d5b-5fqkq"
May 29 13:50:02.020926 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:02.020811    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deae1b19-73c8-40fc-a0cc-e7663d86c726-tls-certs\") pod \"kube-auth-proxy-855f784d5b-5fqkq\" (UID: \"deae1b19-73c8-40fc-a0cc-e7663d86c726\") " pod="openshift-ingress/kube-auth-proxy-855f784d5b-5fqkq"
May 29 13:50:02.021034 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:50:02.020928    2571 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
May 29 13:50:02.021034 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:50:02.021002    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/deae1b19-73c8-40fc-a0cc-e7663d86c726-tls-certs podName:deae1b19-73c8-40fc-a0cc-e7663d86c726 nodeName:}" failed. No retries permitted until 2026-05-29 13:50:02.520982441 +0000 UTC m=+476.988052514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/deae1b19-73c8-40fc-a0cc-e7663d86c726-tls-certs") pod "kube-auth-proxy-855f784d5b-5fqkq" (UID: "deae1b19-73c8-40fc-a0cc-e7663d86c726") : secret "kube-auth-proxy-tls" not found
May 29 13:50:02.022956 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:02.022939    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/deae1b19-73c8-40fc-a0cc-e7663d86c726-tmp\") pod \"kube-auth-proxy-855f784d5b-5fqkq\" (UID: \"deae1b19-73c8-40fc-a0cc-e7663d86c726\") " pod="openshift-ingress/kube-auth-proxy-855f784d5b-5fqkq"
May 29 13:50:02.034125 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:02.034106    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xzg2\" (UniqueName: \"kubernetes.io/projected/deae1b19-73c8-40fc-a0cc-e7663d86c726-kube-api-access-2xzg2\") pod \"kube-auth-proxy-855f784d5b-5fqkq\" (UID: \"deae1b19-73c8-40fc-a0cc-e7663d86c726\") " pod="openshift-ingress/kube-auth-proxy-855f784d5b-5fqkq"
May 29 13:50:02.121628 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:02.121595    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec7a7005-5d22-4311-87b9-2faf1e135e15-tls-certs\") pod \"model-serving-api-6577596cd5-dfvjf\" (UID: \"ec7a7005-5d22-4311-87b9-2faf1e135e15\") " pod="opendatahub/model-serving-api-6577596cd5-dfvjf"
May 29 13:50:02.121784 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:02.121657    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nddb6\" (UniqueName: \"kubernetes.io/projected/ec7a7005-5d22-4311-87b9-2faf1e135e15-kube-api-access-nddb6\") pod \"model-serving-api-6577596cd5-dfvjf\" (UID: \"ec7a7005-5d22-4311-87b9-2faf1e135e15\") " pod="opendatahub/model-serving-api-6577596cd5-dfvjf"
May 29 13:50:02.222559 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:02.222465    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nddb6\" (UniqueName: \"kubernetes.io/projected/ec7a7005-5d22-4311-87b9-2faf1e135e15-kube-api-access-nddb6\") pod \"model-serving-api-6577596cd5-dfvjf\" (UID: \"ec7a7005-5d22-4311-87b9-2faf1e135e15\") " pod="opendatahub/model-serving-api-6577596cd5-dfvjf"
May 29 13:50:02.222559 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:02.222527    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec7a7005-5d22-4311-87b9-2faf1e135e15-tls-certs\") pod \"model-serving-api-6577596cd5-dfvjf\" (UID: \"ec7a7005-5d22-4311-87b9-2faf1e135e15\") " pod="opendatahub/model-serving-api-6577596cd5-dfvjf"
May 29 13:50:02.224823 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:02.224802    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec7a7005-5d22-4311-87b9-2faf1e135e15-tls-certs\") pod \"model-serving-api-6577596cd5-dfvjf\" (UID: \"ec7a7005-5d22-4311-87b9-2faf1e135e15\") " pod="opendatahub/model-serving-api-6577596cd5-dfvjf"
May 29 13:50:02.229674 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:02.229649    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nddb6\" (UniqueName: \"kubernetes.io/projected/ec7a7005-5d22-4311-87b9-2faf1e135e15-kube-api-access-nddb6\") pod \"model-serving-api-6577596cd5-dfvjf\" (UID: \"ec7a7005-5d22-4311-87b9-2faf1e135e15\") " pod="opendatahub/model-serving-api-6577596cd5-dfvjf"
May 29 13:50:02.277372 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:02.277342    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-6577596cd5-dfvjf"
May 29 13:50:02.399168 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:02.399131    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-6577596cd5-dfvjf"]
May 29 13:50:02.402462 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:50:02.402432    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec7a7005_5d22_4311_87b9_2faf1e135e15.slice/crio-d292531df1a959b28abd420be0d6f9f223401a47054fdf57c037de454d923782 WatchSource:0}: Error finding container d292531df1a959b28abd420be0d6f9f223401a47054fdf57c037de454d923782: Status 404 returned error can't find the container with id d292531df1a959b28abd420be0d6f9f223401a47054fdf57c037de454d923782
May 29 13:50:02.525789 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:02.525717    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deae1b19-73c8-40fc-a0cc-e7663d86c726-tls-certs\") pod \"kube-auth-proxy-855f784d5b-5fqkq\" (UID: \"deae1b19-73c8-40fc-a0cc-e7663d86c726\") " pod="openshift-ingress/kube-auth-proxy-855f784d5b-5fqkq"
May 29 13:50:02.528130 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:02.528101    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deae1b19-73c8-40fc-a0cc-e7663d86c726-tls-certs\") pod \"kube-auth-proxy-855f784d5b-5fqkq\" (UID: \"deae1b19-73c8-40fc-a0cc-e7663d86c726\") " pod="openshift-ingress/kube-auth-proxy-855f784d5b-5fqkq"
May 29 13:50:02.609045 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:02.609005    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-6577596cd5-dfvjf" event={"ID":"ec7a7005-5d22-4311-87b9-2faf1e135e15","Type":"ContainerStarted","Data":"d292531df1a959b28abd420be0d6f9f223401a47054fdf57c037de454d923782"}
May 29 13:50:02.610388 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:02.610347    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-76588b95-g42dw" event={"ID":"59222dfa-c8a8-4d60-91eb-467a6c42f78a","Type":"ContainerStarted","Data":"6b8314fd0b9fb0427209ed37018db3ddbabcab02ea373710ce8932356ed1b861"}
May 29 13:50:02.711837 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:02.711797    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-855f784d5b-5fqkq"
May 29 13:50:02.858564 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:02.858317    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-855f784d5b-5fqkq"]
May 29 13:50:02.861103 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:50:02.861071    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeae1b19_73c8_40fc_a0cc_e7663d86c726.slice/crio-4b17073362069fbf51a2b45ef67cae6d4c8388c51b4997aeaabac2a36d2c1c67 WatchSource:0}: Error finding container 4b17073362069fbf51a2b45ef67cae6d4c8388c51b4997aeaabac2a36d2c1c67: Status 404 returned error can't find the container with id 4b17073362069fbf51a2b45ef67cae6d4c8388c51b4997aeaabac2a36d2c1c67
May 29 13:50:03.628992 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:03.628950    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-855f784d5b-5fqkq" event={"ID":"deae1b19-73c8-40fc-a0cc-e7663d86c726","Type":"ContainerStarted","Data":"4b17073362069fbf51a2b45ef67cae6d4c8388c51b4997aeaabac2a36d2c1c67"}
May 29 13:50:06.773186 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:06.773160    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
May 29 13:50:07.640728 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:07.640690    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-6577596cd5-dfvjf" event={"ID":"ec7a7005-5d22-4311-87b9-2faf1e135e15","Type":"ContainerStarted","Data":"2ecbf6e69e0bc9f12c60156a563b5e20ad185eff242189fcd5b02b69ec1e3de7"}
May 29 13:50:07.640903 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:07.640847    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-6577596cd5-dfvjf"
May 29 13:50:07.642001 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:07.641978    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-76588b95-g42dw" event={"ID":"59222dfa-c8a8-4d60-91eb-467a6c42f78a","Type":"ContainerStarted","Data":"ae5e38287c85fb787e3074bac0819f54cd82c132215315c90fe8958fd458aede"}
May 29 13:50:07.642129 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:07.642092    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-76588b95-g42dw"
May 29 13:50:07.643187 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:07.643163    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-855f784d5b-5fqkq" event={"ID":"deae1b19-73c8-40fc-a0cc-e7663d86c726","Type":"ContainerStarted","Data":"8e02d3a4ecdc30e7725e90238fafde6929a980f0339955051f37596f352ace77"}
May 29 13:50:07.669388 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:07.669343    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="opendatahub/model-serving-api-6577596cd5-dfvjf" podStartSLOduration=2.307439183 podStartE2EDuration="6.669332954s" podCreationTimestamp="2026-05-29 13:50:01 +0000 UTC" firstStartedPulling="2026-05-29 13:50:02.404173258 +0000 UTC m=+476.871243317" lastFinishedPulling="2026-05-29 13:50:06.766067024 +0000 UTC m=+481.233137088" observedRunningTime="2026-05-29 13:50:07.668784314 +0000 UTC m=+482.135854394" watchObservedRunningTime="2026-05-29 13:50:07.669332954 +0000 UTC m=+482.136403035"
May 29 13:50:07.700160 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:07.700118    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="opendatahub/odh-model-controller-76588b95-g42dw" podStartSLOduration=2.894122636 podStartE2EDuration="7.70010185s" podCreationTimestamp="2026-05-29 13:50:00 +0000 UTC" firstStartedPulling="2026-05-29 13:50:01.951922624 +0000 UTC m=+476.418992684" lastFinishedPulling="2026-05-29 13:50:06.757901839 +0000 UTC m=+481.224971898" observedRunningTime="2026-05-29 13:50:07.699300334 +0000 UTC m=+482.166370415" watchObservedRunningTime="2026-05-29 13:50:07.70010185 +0000 UTC m=+482.167171930"
May 29 13:50:07.724965 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:07.724918    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-855f784d5b-5fqkq" podStartSLOduration=2.817428146 podStartE2EDuration="6.724905229s" podCreationTimestamp="2026-05-29 13:50:01 +0000 UTC" firstStartedPulling="2026-05-29 13:50:02.862962974 +0000 UTC m=+477.330033039" lastFinishedPulling="2026-05-29 13:50:06.770440054 +0000 UTC m=+481.237510122" observedRunningTime="2026-05-29 13:50:07.72471673 +0000 UTC m=+482.191786810" watchObservedRunningTime="2026-05-29 13:50:07.724905229 +0000 UTC m=+482.191975309"
May 29 13:50:10.518689 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:10.518655    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-699cb996b8-lc86g"]
May 29 13:50:10.521996 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:10.521974    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-699cb996b8-lc86g"
May 29 13:50:10.524561 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:10.524540    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
May 29 13:50:10.525306 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:10.525273    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
May 29 13:50:10.525404 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:10.525388    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
May 29 13:50:10.526318 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:10.526302    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-6zxgb\""
May 29 13:50:10.526400 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:10.526347    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
May 29 13:50:10.526539 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:10.526523    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
May 29 13:50:10.541181 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:10.541157    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-699cb996b8-lc86g"]
May 29 13:50:10.691236 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:10.691207    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqv6g\" (UniqueName: \"kubernetes.io/projected/cb007466-d172-40b4-b1e8-087723079292-kube-api-access-dqv6g\") pod \"lws-controller-manager-699cb996b8-lc86g\" (UID: \"cb007466-d172-40b4-b1e8-087723079292\") " pod="openshift-lws-operator/lws-controller-manager-699cb996b8-lc86g"
May 29 13:50:10.691402 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:10.691250    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cb007466-d172-40b4-b1e8-087723079292-manager-config\") pod \"lws-controller-manager-699cb996b8-lc86g\" (UID: \"cb007466-d172-40b4-b1e8-087723079292\") " pod="openshift-lws-operator/lws-controller-manager-699cb996b8-lc86g"
May 29 13:50:10.691402 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:10.691309    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb007466-d172-40b4-b1e8-087723079292-metrics-cert\") pod \"lws-controller-manager-699cb996b8-lc86g\" (UID: \"cb007466-d172-40b4-b1e8-087723079292\") " pod="openshift-lws-operator/lws-controller-manager-699cb996b8-lc86g"
May 29 13:50:10.691402 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:10.691329    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb007466-d172-40b4-b1e8-087723079292-cert\") pod \"lws-controller-manager-699cb996b8-lc86g\" (UID: \"cb007466-d172-40b4-b1e8-087723079292\") " pod="openshift-lws-operator/lws-controller-manager-699cb996b8-lc86g"
May 29 13:50:10.792120 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:10.792006    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb007466-d172-40b4-b1e8-087723079292-cert\") pod \"lws-controller-manager-699cb996b8-lc86g\" (UID: \"cb007466-d172-40b4-b1e8-087723079292\") " pod="openshift-lws-operator/lws-controller-manager-699cb996b8-lc86g"
May 29 13:50:10.792120 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:10.792088    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dqv6g\" (UniqueName: \"kubernetes.io/projected/cb007466-d172-40b4-b1e8-087723079292-kube-api-access-dqv6g\") pod \"lws-controller-manager-699cb996b8-lc86g\" (UID: \"cb007466-d172-40b4-b1e8-087723079292\") " pod="openshift-lws-operator/lws-controller-manager-699cb996b8-lc86g"
May 29 13:50:10.792354 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:10.792136    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cb007466-d172-40b4-b1e8-087723079292-manager-config\") pod \"lws-controller-manager-699cb996b8-lc86g\" (UID: \"cb007466-d172-40b4-b1e8-087723079292\") " pod="openshift-lws-operator/lws-controller-manager-699cb996b8-lc86g"
May 29 13:50:10.792354 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:10.792184    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb007466-d172-40b4-b1e8-087723079292-metrics-cert\") pod \"lws-controller-manager-699cb996b8-lc86g\" (UID: \"cb007466-d172-40b4-b1e8-087723079292\") " pod="openshift-lws-operator/lws-controller-manager-699cb996b8-lc86g"
May 29 13:50:10.792933 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:10.792906    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cb007466-d172-40b4-b1e8-087723079292-manager-config\") pod \"lws-controller-manager-699cb996b8-lc86g\" (UID: \"cb007466-d172-40b4-b1e8-087723079292\") " pod="openshift-lws-operator/lws-controller-manager-699cb996b8-lc86g"
May 29 13:50:10.794688 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:10.794668    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb007466-d172-40b4-b1e8-087723079292-cert\") pod \"lws-controller-manager-699cb996b8-lc86g\" (UID: \"cb007466-d172-40b4-b1e8-087723079292\") " pod="openshift-lws-operator/lws-controller-manager-699cb996b8-lc86g"
May 29 13:50:10.794819 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:10.794801    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb007466-d172-40b4-b1e8-087723079292-metrics-cert\") pod \"lws-controller-manager-699cb996b8-lc86g\" (UID: \"cb007466-d172-40b4-b1e8-087723079292\") " pod="openshift-lws-operator/lws-controller-manager-699cb996b8-lc86g"
May 29 13:50:10.808018 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:10.807992    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqv6g\" (UniqueName: \"kubernetes.io/projected/cb007466-d172-40b4-b1e8-087723079292-kube-api-access-dqv6g\") pod \"lws-controller-manager-699cb996b8-lc86g\" (UID: \"cb007466-d172-40b4-b1e8-087723079292\") " pod="openshift-lws-operator/lws-controller-manager-699cb996b8-lc86g"
May 29 13:50:10.830811 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:10.830777    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-699cb996b8-lc86g"
May 29 13:50:10.979953 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:10.979929    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-699cb996b8-lc86g"]
May 29 13:50:10.982321 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:50:10.982297    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb007466_d172_40b4_b1e8_087723079292.slice/crio-97d36c23faa59f79c8d533aeadff9025a63664ebe9efd4b1507438eec654818a WatchSource:0}: Error finding container 97d36c23faa59f79c8d533aeadff9025a63664ebe9efd4b1507438eec654818a: Status 404 returned error can't find the container with id 97d36c23faa59f79c8d533aeadff9025a63664ebe9efd4b1507438eec654818a
May 29 13:50:11.654751 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:11.654714    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-699cb996b8-lc86g" event={"ID":"cb007466-d172-40b4-b1e8-087723079292","Type":"ContainerStarted","Data":"97d36c23faa59f79c8d533aeadff9025a63664ebe9efd4b1507438eec654818a"}
May 29 13:50:14.665843 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:14.665809    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-699cb996b8-lc86g" event={"ID":"cb007466-d172-40b4-b1e8-087723079292","Type":"ContainerStarted","Data":"c9619c33d04f75dc8c9c85112a9e46b643ad1f09fdaa9bd8ea221375741118e7"}
May 29 13:50:14.666234 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:14.665935    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-699cb996b8-lc86g"
May 29 13:50:14.685502 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:14.685451    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-699cb996b8-lc86g" podStartSLOduration=1.977846614 podStartE2EDuration="4.685435733s" podCreationTimestamp="2026-05-29 13:50:10 +0000 UTC" firstStartedPulling="2026-05-29 13:50:10.984009236 +0000 UTC m=+485.451079295" lastFinishedPulling="2026-05-29 13:50:13.691598348 +0000 UTC m=+488.158668414" observedRunningTime="2026-05-29 13:50:14.683940456 +0000 UTC m=+489.151010537" watchObservedRunningTime="2026-05-29 13:50:14.685435733 +0000 UTC m=+489.152505845"
May 29 13:50:18.647981 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:18.647951    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-76588b95-g42dw"
May 29 13:50:18.649894 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:18.649873    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-6577596cd5-dfvjf"
May 29 13:50:25.671137 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:50:25.671102    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-699cb996b8-lc86g"
May 29 13:51:43.758471 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:51:43.758438    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6cb54b5c86-zxjxr"]
May 29 13:51:43.760312 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:51:43.760297    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-zxjxr"
May 29 13:51:43.762600 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:51:43.762577    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
May 29 13:51:43.762725 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:51:43.762624    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
May 29 13:51:43.762871 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:51:43.762856    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
May 29 13:51:43.763242 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:51:43.763224    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mxrd4\""
May 29 13:51:43.763339 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:51:43.763253    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
May 29 13:51:43.769591 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:51:43.769570    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6cb54b5c86-zxjxr"]
May 29 13:51:43.887734 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:51:43.887701    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d9ede80d-22ba-4e81-81c1-15b8b5631ab7-nginx-conf\") pod \"kuadrant-console-plugin-6cb54b5c86-zxjxr\" (UID: \"d9ede80d-22ba-4e81-81c1-15b8b5631ab7\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-zxjxr"
May 29 13:51:43.887913 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:51:43.887757    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9ede80d-22ba-4e81-81c1-15b8b5631ab7-plugin-serving-cert\") pod \"kuadrant-console-plugin-6cb54b5c86-zxjxr\" (UID: \"d9ede80d-22ba-4e81-81c1-15b8b5631ab7\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-zxjxr"
May 29 13:51:43.887913 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:51:43.887778    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnbmj\" (UniqueName: \"kubernetes.io/projected/d9ede80d-22ba-4e81-81c1-15b8b5631ab7-kube-api-access-bnbmj\") pod \"kuadrant-console-plugin-6cb54b5c86-zxjxr\" (UID: \"d9ede80d-22ba-4e81-81c1-15b8b5631ab7\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-zxjxr"
May 29 13:51:43.988812 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:51:43.988771    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9ede80d-22ba-4e81-81c1-15b8b5631ab7-plugin-serving-cert\") pod \"kuadrant-console-plugin-6cb54b5c86-zxjxr\" (UID: \"d9ede80d-22ba-4e81-81c1-15b8b5631ab7\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-zxjxr"
May 29 13:51:43.988812 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:51:43.988814    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bnbmj\" (UniqueName: \"kubernetes.io/projected/d9ede80d-22ba-4e81-81c1-15b8b5631ab7-kube-api-access-bnbmj\") pod \"kuadrant-console-plugin-6cb54b5c86-zxjxr\" (UID: \"d9ede80d-22ba-4e81-81c1-15b8b5631ab7\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-zxjxr"
May 29 13:51:43.989015 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:51:43.988849    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d9ede80d-22ba-4e81-81c1-15b8b5631ab7-nginx-conf\") pod \"kuadrant-console-plugin-6cb54b5c86-zxjxr\" (UID: \"d9ede80d-22ba-4e81-81c1-15b8b5631ab7\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-zxjxr"
May 29 13:51:43.989015 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:51:43.988971    2571 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
May 29 13:51:43.989119 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:51:43.989040    2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9ede80d-22ba-4e81-81c1-15b8b5631ab7-plugin-serving-cert podName:d9ede80d-22ba-4e81-81c1-15b8b5631ab7 nodeName:}" failed. No retries permitted until 2026-05-29 13:51:44.489023754 +0000 UTC m=+578.956093812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/d9ede80d-22ba-4e81-81c1-15b8b5631ab7-plugin-serving-cert") pod "kuadrant-console-plugin-6cb54b5c86-zxjxr" (UID: "d9ede80d-22ba-4e81-81c1-15b8b5631ab7") : secret "plugin-serving-cert" not found
May 29 13:51:43.989438 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:51:43.989421    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d9ede80d-22ba-4e81-81c1-15b8b5631ab7-nginx-conf\") pod \"kuadrant-console-plugin-6cb54b5c86-zxjxr\" (UID: \"d9ede80d-22ba-4e81-81c1-15b8b5631ab7\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-zxjxr"
May 29 13:51:44.001120 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:51:44.001091    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnbmj\" (UniqueName: \"kubernetes.io/projected/d9ede80d-22ba-4e81-81c1-15b8b5631ab7-kube-api-access-bnbmj\") pod \"kuadrant-console-plugin-6cb54b5c86-zxjxr\" (UID: \"d9ede80d-22ba-4e81-81c1-15b8b5631ab7\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-zxjxr"
May 29 13:51:44.493618 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:51:44.493583    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9ede80d-22ba-4e81-81c1-15b8b5631ab7-plugin-serving-cert\") pod \"kuadrant-console-plugin-6cb54b5c86-zxjxr\" (UID: \"d9ede80d-22ba-4e81-81c1-15b8b5631ab7\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-zxjxr"
May 29 13:51:44.495818 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:51:44.495786    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9ede80d-22ba-4e81-81c1-15b8b5631ab7-plugin-serving-cert\") pod \"kuadrant-console-plugin-6cb54b5c86-zxjxr\" (UID: \"d9ede80d-22ba-4e81-81c1-15b8b5631ab7\") " pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-zxjxr"
May 29 13:51:44.668852 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:51:44.668805    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-zxjxr"
May 29 13:51:44.793763 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:51:44.793625    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-6cb54b5c86-zxjxr"]
May 29 13:51:44.796377 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:51:44.796346    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9ede80d_22ba_4e81_81c1_15b8b5631ab7.slice/crio-9c632edddcddc28cf00f85f84956f33724aa2149e5b34fcbd1695e7e329f3b2c WatchSource:0}: Error finding container 9c632edddcddc28cf00f85f84956f33724aa2149e5b34fcbd1695e7e329f3b2c: Status 404 returned error can't find the container with id 9c632edddcddc28cf00f85f84956f33724aa2149e5b34fcbd1695e7e329f3b2c
May 29 13:51:44.905305 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:51:44.905271    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-zxjxr" event={"ID":"d9ede80d-22ba-4e81-81c1-15b8b5631ab7","Type":"ContainerStarted","Data":"9c632edddcddc28cf00f85f84956f33724aa2149e5b34fcbd1695e7e329f3b2c"}
May 29 13:52:07.983584 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:07.983493    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-zxjxr" event={"ID":"d9ede80d-22ba-4e81-81c1-15b8b5631ab7","Type":"ContainerStarted","Data":"abdb27af373b025ad2b95bb4e0943a2ea740586e8fe624f60b166e33c41cd4cf"}
May 29 13:52:08.004582 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:08.004468    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-6cb54b5c86-zxjxr" podStartSLOduration=2.185962168 podStartE2EDuration="25.004450943s" podCreationTimestamp="2026-05-29 13:51:43 +0000 UTC" firstStartedPulling="2026-05-29 13:51:44.79770849 +0000 UTC m=+579.264778552" lastFinishedPulling="2026-05-29 13:52:07.616197265 +0000 UTC m=+602.083267327" observedRunningTime="2026-05-29 13:52:08.003755082 +0000 UTC m=+602.470825164" watchObservedRunningTime="2026-05-29 13:52:08.004450943 +0000 UTC m=+602.471521025"
May 29 13:52:27.642824 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:27.642789    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-f99f4b5cd-9zn89"]
May 29 13:52:27.645186 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:27.645166    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f99f4b5cd-9zn89"
May 29 13:52:27.648253 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:27.648224    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-74wbz\""
May 29 13:52:27.660727 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:27.660696    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f99f4b5cd-9zn89"]
May 29 13:52:27.739037 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:27.738998    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkbwb\" (UniqueName: \"kubernetes.io/projected/91928791-d429-4475-9dc1-6a70abb3cbff-kube-api-access-jkbwb\") pod \"authorino-f99f4b5cd-9zn89\" (UID: \"91928791-d429-4475-9dc1-6a70abb3cbff\") " pod="kuadrant-system/authorino-f99f4b5cd-9zn89"
May 29 13:52:27.839714 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:27.839677    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jkbwb\" (UniqueName: \"kubernetes.io/projected/91928791-d429-4475-9dc1-6a70abb3cbff-kube-api-access-jkbwb\") pod \"authorino-f99f4b5cd-9zn89\" (UID: \"91928791-d429-4475-9dc1-6a70abb3cbff\") " pod="kuadrant-system/authorino-f99f4b5cd-9zn89"
May 29 13:52:27.848879 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:27.848846    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkbwb\" (UniqueName: \"kubernetes.io/projected/91928791-d429-4475-9dc1-6a70abb3cbff-kube-api-access-jkbwb\") pod \"authorino-f99f4b5cd-9zn89\" (UID: \"91928791-d429-4475-9dc1-6a70abb3cbff\") " pod="kuadrant-system/authorino-f99f4b5cd-9zn89"
May 29 13:52:27.882438 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:27.882401    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7498df8756-mcf8l"]
May 29 13:52:27.884521 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:27.884503    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7498df8756-mcf8l"
May 29 13:52:27.892753 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:27.892732    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7498df8756-mcf8l"]
May 29 13:52:27.940331 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:27.940233    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rtzp\" (UniqueName: \"kubernetes.io/projected/c46df769-ee08-4695-85f7-3b42ba3736dd-kube-api-access-7rtzp\") pod \"authorino-7498df8756-mcf8l\" (UID: \"c46df769-ee08-4695-85f7-3b42ba3736dd\") " pod="kuadrant-system/authorino-7498df8756-mcf8l"
May 29 13:52:27.953226 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:27.953196    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f99f4b5cd-9zn89"
May 29 13:52:28.040815 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:28.040779    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7rtzp\" (UniqueName: \"kubernetes.io/projected/c46df769-ee08-4695-85f7-3b42ba3736dd-kube-api-access-7rtzp\") pod \"authorino-7498df8756-mcf8l\" (UID: \"c46df769-ee08-4695-85f7-3b42ba3736dd\") " pod="kuadrant-system/authorino-7498df8756-mcf8l"
May 29 13:52:28.049747 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:28.049716    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rtzp\" (UniqueName: \"kubernetes.io/projected/c46df769-ee08-4695-85f7-3b42ba3736dd-kube-api-access-7rtzp\") pod \"authorino-7498df8756-mcf8l\" (UID: \"c46df769-ee08-4695-85f7-3b42ba3736dd\") " pod="kuadrant-system/authorino-7498df8756-mcf8l"
May 29 13:52:28.076022 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:28.075988    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f99f4b5cd-9zn89"]
May 29 13:52:28.078922 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:52:28.078896    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91928791_d429_4475_9dc1_6a70abb3cbff.slice/crio-3bdf9893c8650e3cf004236131ff0ae9b7a7cee0522a44c9ffcc57c24354d9cb WatchSource:0}: Error finding container 3bdf9893c8650e3cf004236131ff0ae9b7a7cee0522a44c9ffcc57c24354d9cb: Status 404 returned error can't find the container with id 3bdf9893c8650e3cf004236131ff0ae9b7a7cee0522a44c9ffcc57c24354d9cb
May 29 13:52:28.194618 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:28.194523    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7498df8756-mcf8l"
May 29 13:52:28.312821 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:28.312739    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7498df8756-mcf8l"]
May 29 13:52:28.315260 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:52:28.315233    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc46df769_ee08_4695_85f7_3b42ba3736dd.slice/crio-812d904accb76690d56fbeadd6427efd6c478d6d3eb3505f15cb2e0cecc7bc55 WatchSource:0}: Error finding container 812d904accb76690d56fbeadd6427efd6c478d6d3eb3505f15cb2e0cecc7bc55: Status 404 returned error can't find the container with id 812d904accb76690d56fbeadd6427efd6c478d6d3eb3505f15cb2e0cecc7bc55
May 29 13:52:29.052673 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:29.052633    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f99f4b5cd-9zn89" event={"ID":"91928791-d429-4475-9dc1-6a70abb3cbff","Type":"ContainerStarted","Data":"3bdf9893c8650e3cf004236131ff0ae9b7a7cee0522a44c9ffcc57c24354d9cb"}
May 29 13:52:29.055157 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:29.055130    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7498df8756-mcf8l" event={"ID":"c46df769-ee08-4695-85f7-3b42ba3736dd","Type":"ContainerStarted","Data":"812d904accb76690d56fbeadd6427efd6c478d6d3eb3505f15cb2e0cecc7bc55"}
May 29 13:52:32.067698 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:32.067659    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f99f4b5cd-9zn89" event={"ID":"91928791-d429-4475-9dc1-6a70abb3cbff","Type":"ContainerStarted","Data":"1205daae3609167c3de599a1e1b005946951fda7323635d06c7d83f045e41c91"}
May 29 13:52:32.069540 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:32.069488    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7498df8756-mcf8l" event={"ID":"c46df769-ee08-4695-85f7-3b42ba3736dd","Type":"ContainerStarted","Data":"9193fc8b29a23ddd44624bb38a7a3dc6dad19666707d729b5eafc858cc12b938"}
May 29 13:52:32.086205 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:32.086145    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-f99f4b5cd-9zn89" podStartSLOduration=1.187510788 podStartE2EDuration="5.086129267s" podCreationTimestamp="2026-05-29 13:52:27 +0000 UTC" firstStartedPulling="2026-05-29 13:52:28.080132965 +0000 UTC m=+622.547203024" lastFinishedPulling="2026-05-29 13:52:31.978751444 +0000 UTC m=+626.445821503" observedRunningTime="2026-05-29 13:52:32.085234084 +0000 UTC m=+626.552304166" watchObservedRunningTime="2026-05-29 13:52:32.086129267 +0000 UTC m=+626.553199347"
May 29 13:52:32.106473 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:32.106413    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-7498df8756-mcf8l" podStartSLOduration=1.454116685 podStartE2EDuration="5.106392096s" podCreationTimestamp="2026-05-29 13:52:27 +0000 UTC" firstStartedPulling="2026-05-29 13:52:28.316542941 +0000 UTC m=+622.783613001" lastFinishedPulling="2026-05-29 13:52:31.968818353 +0000 UTC m=+626.435888412" observedRunningTime="2026-05-29 13:52:32.103660907 +0000 UTC m=+626.570730989" watchObservedRunningTime="2026-05-29 13:52:32.106392096 +0000 UTC m=+626.573462181"
May 29 13:52:32.156016 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:32.155984    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f99f4b5cd-9zn89"]
May 29 13:52:34.075677 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:34.075635    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/authorino-f99f4b5cd-9zn89" podUID="91928791-d429-4475-9dc1-6a70abb3cbff" containerName="authorino" containerID="cri-o://1205daae3609167c3de599a1e1b005946951fda7323635d06c7d83f045e41c91" gracePeriod=30
May 29 13:52:34.313457 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:34.313435    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f99f4b5cd-9zn89"
May 29 13:52:34.393119 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:34.393079    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkbwb\" (UniqueName: \"kubernetes.io/projected/91928791-d429-4475-9dc1-6a70abb3cbff-kube-api-access-jkbwb\") pod \"91928791-d429-4475-9dc1-6a70abb3cbff\" (UID: \"91928791-d429-4475-9dc1-6a70abb3cbff\") "
May 29 13:52:34.395263 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:34.395239    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91928791-d429-4475-9dc1-6a70abb3cbff-kube-api-access-jkbwb" (OuterVolumeSpecName: "kube-api-access-jkbwb") pod "91928791-d429-4475-9dc1-6a70abb3cbff" (UID: "91928791-d429-4475-9dc1-6a70abb3cbff"). InnerVolumeSpecName "kube-api-access-jkbwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 13:52:34.493650 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:34.493602    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkbwb\" (UniqueName: \"kubernetes.io/projected/91928791-d429-4475-9dc1-6a70abb3cbff-kube-api-access-jkbwb\") on node \"ip-10-0-141-1.ec2.internal\" DevicePath \"\""
May 29 13:52:35.080067 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:35.080030    2571 generic.go:358] "Generic (PLEG): container finished" podID="91928791-d429-4475-9dc1-6a70abb3cbff" containerID="1205daae3609167c3de599a1e1b005946951fda7323635d06c7d83f045e41c91" exitCode=0
May 29 13:52:35.080509 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:35.080088    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f99f4b5cd-9zn89" event={"ID":"91928791-d429-4475-9dc1-6a70abb3cbff","Type":"ContainerDied","Data":"1205daae3609167c3de599a1e1b005946951fda7323635d06c7d83f045e41c91"}
May 29 13:52:35.080509 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:35.080102    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f99f4b5cd-9zn89"
May 29 13:52:35.080509 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:35.080126    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f99f4b5cd-9zn89" event={"ID":"91928791-d429-4475-9dc1-6a70abb3cbff","Type":"ContainerDied","Data":"3bdf9893c8650e3cf004236131ff0ae9b7a7cee0522a44c9ffcc57c24354d9cb"}
May 29 13:52:35.080509 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:35.080142    2571 scope.go:117] "RemoveContainer" containerID="1205daae3609167c3de599a1e1b005946951fda7323635d06c7d83f045e41c91"
May 29 13:52:35.088972 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:35.088927    2571 scope.go:117] "RemoveContainer" containerID="1205daae3609167c3de599a1e1b005946951fda7323635d06c7d83f045e41c91"
May 29 13:52:35.089239 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:52:35.089213    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1205daae3609167c3de599a1e1b005946951fda7323635d06c7d83f045e41c91\": container with ID starting with 1205daae3609167c3de599a1e1b005946951fda7323635d06c7d83f045e41c91 not found: ID does not exist" containerID="1205daae3609167c3de599a1e1b005946951fda7323635d06c7d83f045e41c91"
May 29 13:52:35.089308 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:35.089251    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1205daae3609167c3de599a1e1b005946951fda7323635d06c7d83f045e41c91"} err="failed to get container status \"1205daae3609167c3de599a1e1b005946951fda7323635d06c7d83f045e41c91\": rpc error: code = NotFound desc = could not find container \"1205daae3609167c3de599a1e1b005946951fda7323635d06c7d83f045e41c91\": container with ID starting with 1205daae3609167c3de599a1e1b005946951fda7323635d06c7d83f045e41c91 not found: ID does not exist"
May 29 13:52:35.103356 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:35.103328    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f99f4b5cd-9zn89"]
May 29 13:52:35.110688 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:35.110667    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-f99f4b5cd-9zn89"]
May 29 13:52:36.148659 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:52:36.148625    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91928791-d429-4475-9dc1-6a70abb3cbff" path="/var/lib/kubelet/pods/91928791-d429-4475-9dc1-6a70abb3cbff/volumes"
May 29 13:53:46.925636 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:46.925595    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8b475cf9f-nfslg"]
May 29 13:53:46.926080 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:46.925884    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91928791-d429-4475-9dc1-6a70abb3cbff" containerName="authorino"
May 29 13:53:46.926080 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:46.925895    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="91928791-d429-4475-9dc1-6a70abb3cbff" containerName="authorino"
May 29 13:53:46.926080 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:46.925947    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="91928791-d429-4475-9dc1-6a70abb3cbff" containerName="authorino"
May 29 13:53:46.928727 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:46.928710    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8b475cf9f-nfslg"
May 29 13:53:46.938581 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:46.938556    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8b475cf9f-nfslg"]
May 29 13:53:46.948930 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:46.948898    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq5nk\" (UniqueName: \"kubernetes.io/projected/18eab09d-c6e0-4167-871f-00fa241c70e3-kube-api-access-rq5nk\") pod \"authorino-8b475cf9f-nfslg\" (UID: \"18eab09d-c6e0-4167-871f-00fa241c70e3\") " pod="kuadrant-system/authorino-8b475cf9f-nfslg"
May 29 13:53:47.049726 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:47.049672    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rq5nk\" (UniqueName: \"kubernetes.io/projected/18eab09d-c6e0-4167-871f-00fa241c70e3-kube-api-access-rq5nk\") pod \"authorino-8b475cf9f-nfslg\" (UID: \"18eab09d-c6e0-4167-871f-00fa241c70e3\") " pod="kuadrant-system/authorino-8b475cf9f-nfslg"
May 29 13:53:47.059747 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:47.059717    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq5nk\" (UniqueName: \"kubernetes.io/projected/18eab09d-c6e0-4167-871f-00fa241c70e3-kube-api-access-rq5nk\") pod \"authorino-8b475cf9f-nfslg\" (UID: \"18eab09d-c6e0-4167-871f-00fa241c70e3\") " pod="kuadrant-system/authorino-8b475cf9f-nfslg"
May 29 13:53:47.177026 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:47.176943    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8b475cf9f-nfslg"]
May 29 13:53:47.177181 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:47.177165    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8b475cf9f-nfslg"
May 29 13:53:47.216832 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:47.216802    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-84bfbc579c-bph78"]
May 29 13:53:47.220907 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:47.220888    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84bfbc579c-bph78"
May 29 13:53:47.231322 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:47.231298    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-84bfbc579c-bph78"]
May 29 13:53:47.251373 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:47.251337    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s75wn\" (UniqueName: \"kubernetes.io/projected/5e54cfde-b13d-4153-a65e-78ed39490caa-kube-api-access-s75wn\") pod \"authorino-84bfbc579c-bph78\" (UID: \"5e54cfde-b13d-4153-a65e-78ed39490caa\") " pod="kuadrant-system/authorino-84bfbc579c-bph78"
May 29 13:53:47.308265 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:47.308232    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-84bfbc579c-bph78"]
May 29 13:53:47.308472 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:53:47.308453    2571 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-s75wn], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-84bfbc579c-bph78" podUID="5e54cfde-b13d-4153-a65e-78ed39490caa"
May 29 13:53:47.310710 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:47.310684    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8b475cf9f-nfslg"]
May 29 13:53:47.314830 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:53:47.314804    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18eab09d_c6e0_4167_871f_00fa241c70e3.slice/crio-a3ece046c31479666e3430ac2d12e207f001b85f0622a7771c806dd88f2fd979 WatchSource:0}: Error finding container a3ece046c31479666e3430ac2d12e207f001b85f0622a7771c806dd88f2fd979: Status 404 returned error can't find the container with id a3ece046c31479666e3430ac2d12e207f001b85f0622a7771c806dd88f2fd979
May 29 13:53:47.345399 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:47.345366    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5957664df-xcqcb"]
May 29 13:53:47.348549 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:47.348532    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5957664df-xcqcb"
May 29 13:53:47.350785 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:47.350764    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
May 29 13:53:47.352444 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:47.352426    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-s75wn\" (UniqueName: \"kubernetes.io/projected/5e54cfde-b13d-4153-a65e-78ed39490caa-kube-api-access-s75wn\") pod \"authorino-84bfbc579c-bph78\" (UID: \"5e54cfde-b13d-4153-a65e-78ed39490caa\") " pod="kuadrant-system/authorino-84bfbc579c-bph78"
May 29 13:53:47.356734 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:47.356710    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5957664df-xcqcb"]
May 29 13:53:47.366355 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:47.366331    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s75wn\" (UniqueName: \"kubernetes.io/projected/5e54cfde-b13d-4153-a65e-78ed39490caa-kube-api-access-s75wn\") pod \"authorino-84bfbc579c-bph78\" (UID: \"5e54cfde-b13d-4153-a65e-78ed39490caa\") " pod="kuadrant-system/authorino-84bfbc579c-bph78"
May 29 13:53:47.453756 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:47.453672    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4rg4\" (UniqueName: \"kubernetes.io/projected/6e606b8c-8816-4f71-8bac-c0fd558da6e7-kube-api-access-f4rg4\") pod \"authorino-5957664df-xcqcb\" (UID: \"6e606b8c-8816-4f71-8bac-c0fd558da6e7\") " pod="kuadrant-system/authorino-5957664df-xcqcb"
May 29 13:53:47.453756 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:47.453739    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e606b8c-8816-4f71-8bac-c0fd558da6e7-tls-cert\") pod \"authorino-5957664df-xcqcb\" (UID: \"6e606b8c-8816-4f71-8bac-c0fd558da6e7\") " pod="kuadrant-system/authorino-5957664df-xcqcb"
May 29 13:53:47.554417 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:47.554381    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f4rg4\" (UniqueName: \"kubernetes.io/projected/6e606b8c-8816-4f71-8bac-c0fd558da6e7-kube-api-access-f4rg4\") pod \"authorino-5957664df-xcqcb\" (UID: \"6e606b8c-8816-4f71-8bac-c0fd558da6e7\") " pod="kuadrant-system/authorino-5957664df-xcqcb"
May 29 13:53:47.554582 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:47.554443    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e606b8c-8816-4f71-8bac-c0fd558da6e7-tls-cert\") pod \"authorino-5957664df-xcqcb\" (UID: \"6e606b8c-8816-4f71-8bac-c0fd558da6e7\") " pod="kuadrant-system/authorino-5957664df-xcqcb"
May 29 13:53:47.556756 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:47.556736    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e606b8c-8816-4f71-8bac-c0fd558da6e7-tls-cert\") pod \"authorino-5957664df-xcqcb\" (UID: \"6e606b8c-8816-4f71-8bac-c0fd558da6e7\") " pod="kuadrant-system/authorino-5957664df-xcqcb"
May 29 13:53:47.563311 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:47.563288    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4rg4\" (UniqueName: \"kubernetes.io/projected/6e606b8c-8816-4f71-8bac-c0fd558da6e7-kube-api-access-f4rg4\") pod \"authorino-5957664df-xcqcb\" (UID: \"6e606b8c-8816-4f71-8bac-c0fd558da6e7\") " pod="kuadrant-system/authorino-5957664df-xcqcb"
May 29 13:53:47.661858 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:47.661825    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5957664df-xcqcb"
May 29 13:53:47.789197 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:47.788982    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5957664df-xcqcb"]
May 29 13:53:47.791205 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:53:47.791181    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e606b8c_8816_4f71_8bac_c0fd558da6e7.slice/crio-d8f0d7df5626d847ad68ce5df09704c1f9d16c3b6b5d5424f990500ae13320e5 WatchSource:0}: Error finding container d8f0d7df5626d847ad68ce5df09704c1f9d16c3b6b5d5424f990500ae13320e5: Status 404 returned error can't find the container with id d8f0d7df5626d847ad68ce5df09704c1f9d16c3b6b5d5424f990500ae13320e5
May 29 13:53:48.295673 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:48.295634    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8b475cf9f-nfslg" event={"ID":"18eab09d-c6e0-4167-871f-00fa241c70e3","Type":"ContainerStarted","Data":"7838426e84fa4f7c3ad03b5f520cc2d6d08f7b6dbb883a9c4c22c4a609836767"}
May 29 13:53:48.296163 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:48.295682    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8b475cf9f-nfslg" event={"ID":"18eab09d-c6e0-4167-871f-00fa241c70e3","Type":"ContainerStarted","Data":"a3ece046c31479666e3430ac2d12e207f001b85f0622a7771c806dd88f2fd979"}
May 29 13:53:48.296163 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:48.295659    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/authorino-8b475cf9f-nfslg" podUID="18eab09d-c6e0-4167-871f-00fa241c70e3" containerName="authorino" containerID="cri-o://7838426e84fa4f7c3ad03b5f520cc2d6d08f7b6dbb883a9c4c22c4a609836767" gracePeriod=30
May 29 13:53:48.297210 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:48.297187    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5957664df-xcqcb" event={"ID":"6e606b8c-8816-4f71-8bac-c0fd558da6e7","Type":"ContainerStarted","Data":"f2a6a4f4b02e12805c2ea0483e65434097fbda21f92157b9066de85915752b17"}
May 29 13:53:48.297318 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:48.297222    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5957664df-xcqcb" event={"ID":"6e606b8c-8816-4f71-8bac-c0fd558da6e7","Type":"ContainerStarted","Data":"d8f0d7df5626d847ad68ce5df09704c1f9d16c3b6b5d5424f990500ae13320e5"}
May 29 13:53:48.297318 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:48.297286    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84bfbc579c-bph78"
May 29 13:53:48.302760 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:48.302742    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84bfbc579c-bph78"
May 29 13:53:48.313796 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:48.313758    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-8b475cf9f-nfslg" podStartSLOduration=1.926430851 podStartE2EDuration="2.313745766s" podCreationTimestamp="2026-05-29 13:53:46 +0000 UTC" firstStartedPulling="2026-05-29 13:53:47.316125458 +0000 UTC m=+701.783195517" lastFinishedPulling="2026-05-29 13:53:47.703440361 +0000 UTC m=+702.170510432" observedRunningTime="2026-05-29 13:53:48.312704702 +0000 UTC m=+702.779774807" watchObservedRunningTime="2026-05-29 13:53:48.313745766 +0000 UTC m=+702.780815846"
May 29 13:53:48.335606 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:48.335558    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-5957664df-xcqcb" podStartSLOduration=0.978713277 podStartE2EDuration="1.335540564s" podCreationTimestamp="2026-05-29 13:53:47 +0000 UTC" firstStartedPulling="2026-05-29 13:53:47.792331693 +0000 UTC m=+702.259401753" lastFinishedPulling="2026-05-29 13:53:48.149158978 +0000 UTC m=+702.616229040" observedRunningTime="2026-05-29 13:53:48.335300504 +0000 UTC m=+702.802370586" watchObservedRunningTime="2026-05-29 13:53:48.335540564 +0000 UTC m=+702.802610646"
May 29 13:53:48.360707 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:48.360679    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s75wn\" (UniqueName: \"kubernetes.io/projected/5e54cfde-b13d-4153-a65e-78ed39490caa-kube-api-access-s75wn\") pod \"5e54cfde-b13d-4153-a65e-78ed39490caa\" (UID: \"5e54cfde-b13d-4153-a65e-78ed39490caa\") "
May 29 13:53:48.362823 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:48.362792    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e54cfde-b13d-4153-a65e-78ed39490caa-kube-api-access-s75wn" (OuterVolumeSpecName: "kube-api-access-s75wn") pod "5e54cfde-b13d-4153-a65e-78ed39490caa" (UID: "5e54cfde-b13d-4153-a65e-78ed39490caa"). InnerVolumeSpecName "kube-api-access-s75wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 13:53:48.367294 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:48.367266    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7498df8756-mcf8l"]
May 29 13:53:48.367474 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:48.367453    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/authorino-7498df8756-mcf8l" podUID="c46df769-ee08-4695-85f7-3b42ba3736dd" containerName="authorino" containerID="cri-o://9193fc8b29a23ddd44624bb38a7a3dc6dad19666707d729b5eafc858cc12b938" gracePeriod=30
May 29 13:53:48.461642 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:48.461557    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s75wn\" (UniqueName: \"kubernetes.io/projected/5e54cfde-b13d-4153-a65e-78ed39490caa-kube-api-access-s75wn\") on node \"ip-10-0-141-1.ec2.internal\" DevicePath \"\""
May 29 13:53:48.605115 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:48.605091    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8b475cf9f-nfslg"
May 29 13:53:48.612691 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:48.612672    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7498df8756-mcf8l"
May 29 13:53:48.663351 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:48.663317    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rq5nk\" (UniqueName: \"kubernetes.io/projected/18eab09d-c6e0-4167-871f-00fa241c70e3-kube-api-access-rq5nk\") pod \"18eab09d-c6e0-4167-871f-00fa241c70e3\" (UID: \"18eab09d-c6e0-4167-871f-00fa241c70e3\") "
May 29 13:53:48.663523 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:48.663373    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rtzp\" (UniqueName: \"kubernetes.io/projected/c46df769-ee08-4695-85f7-3b42ba3736dd-kube-api-access-7rtzp\") pod \"c46df769-ee08-4695-85f7-3b42ba3736dd\" (UID: \"c46df769-ee08-4695-85f7-3b42ba3736dd\") "
May 29 13:53:48.665423 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:48.665400    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c46df769-ee08-4695-85f7-3b42ba3736dd-kube-api-access-7rtzp" (OuterVolumeSpecName: "kube-api-access-7rtzp") pod "c46df769-ee08-4695-85f7-3b42ba3736dd" (UID: "c46df769-ee08-4695-85f7-3b42ba3736dd"). InnerVolumeSpecName "kube-api-access-7rtzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 13:53:48.665490 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:48.665433    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18eab09d-c6e0-4167-871f-00fa241c70e3-kube-api-access-rq5nk" (OuterVolumeSpecName: "kube-api-access-rq5nk") pod "18eab09d-c6e0-4167-871f-00fa241c70e3" (UID: "18eab09d-c6e0-4167-871f-00fa241c70e3"). InnerVolumeSpecName "kube-api-access-rq5nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 13:53:48.764323 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:48.764241    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq5nk\" (UniqueName: \"kubernetes.io/projected/18eab09d-c6e0-4167-871f-00fa241c70e3-kube-api-access-rq5nk\") on node \"ip-10-0-141-1.ec2.internal\" DevicePath \"\""
May 29 13:53:48.764323 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:48.764270    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rtzp\" (UniqueName: \"kubernetes.io/projected/c46df769-ee08-4695-85f7-3b42ba3736dd-kube-api-access-7rtzp\") on node \"ip-10-0-141-1.ec2.internal\" DevicePath \"\""
May 29 13:53:49.300506 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:49.300472    2571 generic.go:358] "Generic (PLEG): container finished" podID="c46df769-ee08-4695-85f7-3b42ba3736dd" containerID="9193fc8b29a23ddd44624bb38a7a3dc6dad19666707d729b5eafc858cc12b938" exitCode=0
May 29 13:53:49.300950 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:49.300519    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7498df8756-mcf8l"
May 29 13:53:49.300950 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:49.300541    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7498df8756-mcf8l" event={"ID":"c46df769-ee08-4695-85f7-3b42ba3736dd","Type":"ContainerDied","Data":"9193fc8b29a23ddd44624bb38a7a3dc6dad19666707d729b5eafc858cc12b938"}
May 29 13:53:49.300950 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:49.300574    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7498df8756-mcf8l" event={"ID":"c46df769-ee08-4695-85f7-3b42ba3736dd","Type":"ContainerDied","Data":"812d904accb76690d56fbeadd6427efd6c478d6d3eb3505f15cb2e0cecc7bc55"}
May 29 13:53:49.300950 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:49.300594    2571 scope.go:117] "RemoveContainer" containerID="9193fc8b29a23ddd44624bb38a7a3dc6dad19666707d729b5eafc858cc12b938"
May 29 13:53:49.301797 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:49.301707    2571 generic.go:358] "Generic (PLEG): container finished" podID="18eab09d-c6e0-4167-871f-00fa241c70e3" containerID="7838426e84fa4f7c3ad03b5f520cc2d6d08f7b6dbb883a9c4c22c4a609836767" exitCode=0
May 29 13:53:49.301797 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:49.301741    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8b475cf9f-nfslg"
May 29 13:53:49.301906 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:49.301795    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8b475cf9f-nfslg" event={"ID":"18eab09d-c6e0-4167-871f-00fa241c70e3","Type":"ContainerDied","Data":"7838426e84fa4f7c3ad03b5f520cc2d6d08f7b6dbb883a9c4c22c4a609836767"}
May 29 13:53:49.301906 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:49.301827    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8b475cf9f-nfslg" event={"ID":"18eab09d-c6e0-4167-871f-00fa241c70e3","Type":"ContainerDied","Data":"a3ece046c31479666e3430ac2d12e207f001b85f0622a7771c806dd88f2fd979"}
May 29 13:53:49.301906 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:49.301828    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-84bfbc579c-bph78"
May 29 13:53:49.308665 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:49.308650    2571 scope.go:117] "RemoveContainer" containerID="9193fc8b29a23ddd44624bb38a7a3dc6dad19666707d729b5eafc858cc12b938"
May 29 13:53:49.308955 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:53:49.308929    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9193fc8b29a23ddd44624bb38a7a3dc6dad19666707d729b5eafc858cc12b938\": container with ID starting with 9193fc8b29a23ddd44624bb38a7a3dc6dad19666707d729b5eafc858cc12b938 not found: ID does not exist" containerID="9193fc8b29a23ddd44624bb38a7a3dc6dad19666707d729b5eafc858cc12b938"
May 29 13:53:49.309030 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:49.308965    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9193fc8b29a23ddd44624bb38a7a3dc6dad19666707d729b5eafc858cc12b938"} err="failed to get container status \"9193fc8b29a23ddd44624bb38a7a3dc6dad19666707d729b5eafc858cc12b938\": rpc error: code = NotFound desc = could not find container \"9193fc8b29a23ddd44624bb38a7a3dc6dad19666707d729b5eafc858cc12b938\": container with ID starting with 9193fc8b29a23ddd44624bb38a7a3dc6dad19666707d729b5eafc858cc12b938 not found: ID does not exist"
May 29 13:53:49.309030 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:49.308990    2571 scope.go:117] "RemoveContainer" containerID="7838426e84fa4f7c3ad03b5f520cc2d6d08f7b6dbb883a9c4c22c4a609836767"
May 29 13:53:49.316958 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:49.316937    2571 scope.go:117] "RemoveContainer" containerID="7838426e84fa4f7c3ad03b5f520cc2d6d08f7b6dbb883a9c4c22c4a609836767"
May 29 13:53:49.317244 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:53:49.317225    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7838426e84fa4f7c3ad03b5f520cc2d6d08f7b6dbb883a9c4c22c4a609836767\": container with ID starting with 7838426e84fa4f7c3ad03b5f520cc2d6d08f7b6dbb883a9c4c22c4a609836767 not found: ID does not exist" containerID="7838426e84fa4f7c3ad03b5f520cc2d6d08f7b6dbb883a9c4c22c4a609836767"
May 29 13:53:49.317331 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:49.317249    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7838426e84fa4f7c3ad03b5f520cc2d6d08f7b6dbb883a9c4c22c4a609836767"} err="failed to get container status \"7838426e84fa4f7c3ad03b5f520cc2d6d08f7b6dbb883a9c4c22c4a609836767\": rpc error: code = NotFound desc = could not find container \"7838426e84fa4f7c3ad03b5f520cc2d6d08f7b6dbb883a9c4c22c4a609836767\": container with ID starting with 7838426e84fa4f7c3ad03b5f520cc2d6d08f7b6dbb883a9c4c22c4a609836767 not found: ID does not exist"
May 29 13:53:49.326540 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:49.326520    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8b475cf9f-nfslg"]
May 29 13:53:49.331578 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:49.331559    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8b475cf9f-nfslg"]
May 29 13:53:49.352691 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:49.352666    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-84bfbc579c-bph78"]
May 29 13:53:49.359513 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:49.359491    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-84bfbc579c-bph78"]
May 29 13:53:49.369860 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:49.369839    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7498df8756-mcf8l"]
May 29 13:53:49.373385 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:49.373369    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7498df8756-mcf8l"]
May 29 13:53:50.148790 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:50.148761    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18eab09d-c6e0-4167-871f-00fa241c70e3" path="/var/lib/kubelet/pods/18eab09d-c6e0-4167-871f-00fa241c70e3/volumes"
May 29 13:53:50.149124 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:50.149109    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e54cfde-b13d-4153-a65e-78ed39490caa" path="/var/lib/kubelet/pods/5e54cfde-b13d-4153-a65e-78ed39490caa/volumes"
May 29 13:53:50.149341 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:50.149329    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c46df769-ee08-4695-85f7-3b42ba3736dd" path="/var/lib/kubelet/pods/c46df769-ee08-4695-85f7-3b42ba3736dd/volumes"
May 29 13:53:51.597215 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:51.597182    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-59b894889c-cb865"]
May 29 13:53:51.598232 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:51.598207    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c46df769-ee08-4695-85f7-3b42ba3736dd" containerName="authorino"
May 29 13:53:51.598368 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:51.598357    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c46df769-ee08-4695-85f7-3b42ba3736dd" containerName="authorino"
May 29 13:53:51.598492 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:51.598482    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18eab09d-c6e0-4167-871f-00fa241c70e3" containerName="authorino"
May 29 13:53:51.598564 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:51.598555    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="18eab09d-c6e0-4167-871f-00fa241c70e3" containerName="authorino"
May 29 13:53:51.598743 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:51.598732    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="18eab09d-c6e0-4167-871f-00fa241c70e3" containerName="authorino"
May 29 13:53:51.598827 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:51.598818    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="c46df769-ee08-4695-85f7-3b42ba3736dd" containerName="authorino"
May 29 13:53:51.603707 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:51.603687    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59b894889c-cb865"
May 29 13:53:51.606187 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:51.606165    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
May 29 13:53:51.606283 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:51.606174    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-678zh\""
May 29 13:53:51.608493 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:51.608471    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59b894889c-cb865"]
May 29 13:53:51.686437 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:51.686400    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t5gg\" (UniqueName: \"kubernetes.io/projected/20ccc54c-7d8b-447f-be4c-4fdc54c966ba-kube-api-access-4t5gg\") pod \"maas-controller-59b894889c-cb865\" (UID: \"20ccc54c-7d8b-447f-be4c-4fdc54c966ba\") " pod="opendatahub/maas-controller-59b894889c-cb865"
May 29 13:53:51.748153 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:51.748119    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8db997c58-nhpk4"]
May 29 13:53:51.751199 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:51.751179    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8db997c58-nhpk4"
May 29 13:53:51.761584 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:51.761562    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8db997c58-nhpk4"]
May 29 13:53:51.787138 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:51.787109    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jlxf\" (UniqueName: \"kubernetes.io/projected/65b430b8-f603-42c1-bc73-89cb54abb5a1-kube-api-access-2jlxf\") pod \"maas-controller-8db997c58-nhpk4\" (UID: \"65b430b8-f603-42c1-bc73-89cb54abb5a1\") " pod="opendatahub/maas-controller-8db997c58-nhpk4"
May 29 13:53:51.787289 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:51.787152    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4t5gg\" (UniqueName: \"kubernetes.io/projected/20ccc54c-7d8b-447f-be4c-4fdc54c966ba-kube-api-access-4t5gg\") pod \"maas-controller-59b894889c-cb865\" (UID: \"20ccc54c-7d8b-447f-be4c-4fdc54c966ba\") " pod="opendatahub/maas-controller-59b894889c-cb865"
May 29 13:53:51.796407 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:51.796376    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t5gg\" (UniqueName: \"kubernetes.io/projected/20ccc54c-7d8b-447f-be4c-4fdc54c966ba-kube-api-access-4t5gg\") pod \"maas-controller-59b894889c-cb865\" (UID: \"20ccc54c-7d8b-447f-be4c-4fdc54c966ba\") " pod="opendatahub/maas-controller-59b894889c-cb865"
May 29 13:53:51.888075 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:51.887957    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2jlxf\" (UniqueName: \"kubernetes.io/projected/65b430b8-f603-42c1-bc73-89cb54abb5a1-kube-api-access-2jlxf\") pod \"maas-controller-8db997c58-nhpk4\" (UID: \"65b430b8-f603-42c1-bc73-89cb54abb5a1\") " pod="opendatahub/maas-controller-8db997c58-nhpk4"
May 29 13:53:51.900284 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:51.900253    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jlxf\" (UniqueName: \"kubernetes.io/projected/65b430b8-f603-42c1-bc73-89cb54abb5a1-kube-api-access-2jlxf\") pod \"maas-controller-8db997c58-nhpk4\" (UID: \"65b430b8-f603-42c1-bc73-89cb54abb5a1\") " pod="opendatahub/maas-controller-8db997c58-nhpk4"
May 29 13:53:51.915111 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:51.915087    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59b894889c-cb865"
May 29 13:53:52.039045 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:52.039022    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59b894889c-cb865"]
May 29 13:53:52.040955 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:53:52.040928    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ccc54c_7d8b_447f_be4c_4fdc54c966ba.slice/crio-20d14a054a96a02cc905d79363c1f66198b1f81d1c50c2f74709defa78e57412 WatchSource:0}: Error finding container 20d14a054a96a02cc905d79363c1f66198b1f81d1c50c2f74709defa78e57412: Status 404 returned error can't find the container with id 20d14a054a96a02cc905d79363c1f66198b1f81d1c50c2f74709defa78e57412
May 29 13:53:52.060582 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:52.060560    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8db997c58-nhpk4"
May 29 13:53:52.185680 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:52.185658    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8db997c58-nhpk4"]
May 29 13:53:52.187207 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:53:52.187178    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b430b8_f603_42c1_bc73_89cb54abb5a1.slice/crio-0adef73641eedebb9295bed1b2665bd763ad279afdf6381e296d72aeb049ee72 WatchSource:0}: Error finding container 0adef73641eedebb9295bed1b2665bd763ad279afdf6381e296d72aeb049ee72: Status 404 returned error can't find the container with id 0adef73641eedebb9295bed1b2665bd763ad279afdf6381e296d72aeb049ee72
May 29 13:53:52.313306 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:52.313275    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59b894889c-cb865" event={"ID":"20ccc54c-7d8b-447f-be4c-4fdc54c966ba","Type":"ContainerStarted","Data":"20d14a054a96a02cc905d79363c1f66198b1f81d1c50c2f74709defa78e57412"}
May 29 13:53:52.314196 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:52.314173    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8db997c58-nhpk4" event={"ID":"65b430b8-f603-42c1-bc73-89cb54abb5a1","Type":"ContainerStarted","Data":"0adef73641eedebb9295bed1b2665bd763ad279afdf6381e296d72aeb049ee72"}
May 29 13:53:55.326236 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:55.326198    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8db997c58-nhpk4" event={"ID":"65b430b8-f603-42c1-bc73-89cb54abb5a1","Type":"ContainerStarted","Data":"745fa56873362bc11ed60763589ca33d436c4b80e6eabf0ce31df0a3c32da58f"}
May 29 13:53:55.326709 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:55.326256    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8db997c58-nhpk4"
May 29 13:53:55.327480 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:55.327453    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59b894889c-cb865" event={"ID":"20ccc54c-7d8b-447f-be4c-4fdc54c966ba","Type":"ContainerStarted","Data":"f7a0270d590f69fd2d612e84ac30409bcf9559c72ade6f384252d667e7adf37a"}
May 29 13:53:55.327611 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:55.327543    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-59b894889c-cb865"
May 29 13:53:55.391673 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:55.391611    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="opendatahub/maas-controller-8db997c58-nhpk4" podStartSLOduration=1.884468118 podStartE2EDuration="4.391592476s" podCreationTimestamp="2026-05-29 13:53:51 +0000 UTC" firstStartedPulling="2026-05-29 13:53:52.188436141 +0000 UTC m=+706.655506200" lastFinishedPulling="2026-05-29 13:53:54.695560499 +0000 UTC m=+709.162630558" observedRunningTime="2026-05-29 13:53:55.391362286 +0000 UTC m=+709.858432370" watchObservedRunningTime="2026-05-29 13:53:55.391592476 +0000 UTC m=+709.858662558"
May 29 13:53:55.428381 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:53:55.428291    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="opendatahub/maas-controller-59b894889c-cb865" podStartSLOduration=1.784563301 podStartE2EDuration="4.428274439s" podCreationTimestamp="2026-05-29 13:53:51 +0000 UTC" firstStartedPulling="2026-05-29 13:53:52.042355501 +0000 UTC m=+706.509425560" lastFinishedPulling="2026-05-29 13:53:54.686066636 +0000 UTC m=+709.153136698" observedRunningTime="2026-05-29 13:53:55.427895479 +0000 UTC m=+709.894965560" watchObservedRunningTime="2026-05-29 13:53:55.428274439 +0000 UTC m=+709.895344519"
May 29 13:54:06.336168 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:06.336128    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8db997c58-nhpk4"
May 29 13:54:06.336611 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:06.336441    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-59b894889c-cb865"
May 29 13:54:06.391540 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:06.391456    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59b894889c-cb865"]
May 29 13:54:06.391681 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:06.391657    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="opendatahub/maas-controller-59b894889c-cb865" podUID="20ccc54c-7d8b-447f-be4c-4fdc54c966ba" containerName="manager" containerID="cri-o://f7a0270d590f69fd2d612e84ac30409bcf9559c72ade6f384252d667e7adf37a" gracePeriod=30
May 29 13:54:06.625746 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:06.625724    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59b894889c-cb865"
May 29 13:54:06.707470 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:06.707389    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4t5gg\" (UniqueName: \"kubernetes.io/projected/20ccc54c-7d8b-447f-be4c-4fdc54c966ba-kube-api-access-4t5gg\") pod \"20ccc54c-7d8b-447f-be4c-4fdc54c966ba\" (UID: \"20ccc54c-7d8b-447f-be4c-4fdc54c966ba\") "
May 29 13:54:06.709548 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:06.709508    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ccc54c-7d8b-447f-be4c-4fdc54c966ba-kube-api-access-4t5gg" (OuterVolumeSpecName: "kube-api-access-4t5gg") pod "20ccc54c-7d8b-447f-be4c-4fdc54c966ba" (UID: "20ccc54c-7d8b-447f-be4c-4fdc54c966ba"). InnerVolumeSpecName "kube-api-access-4t5gg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 13:54:06.808709 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:06.808661    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t5gg\" (UniqueName: \"kubernetes.io/projected/20ccc54c-7d8b-447f-be4c-4fdc54c966ba-kube-api-access-4t5gg\") on node \"ip-10-0-141-1.ec2.internal\" DevicePath \"\""
May 29 13:54:07.369480 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:07.369448    2571 generic.go:358] "Generic (PLEG): container finished" podID="20ccc54c-7d8b-447f-be4c-4fdc54c966ba" containerID="f7a0270d590f69fd2d612e84ac30409bcf9559c72ade6f384252d667e7adf37a" exitCode=0
May 29 13:54:07.369864 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:07.369506    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59b894889c-cb865"
May 29 13:54:07.369864 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:07.369526    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59b894889c-cb865" event={"ID":"20ccc54c-7d8b-447f-be4c-4fdc54c966ba","Type":"ContainerDied","Data":"f7a0270d590f69fd2d612e84ac30409bcf9559c72ade6f384252d667e7adf37a"}
May 29 13:54:07.369864 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:07.369570    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59b894889c-cb865" event={"ID":"20ccc54c-7d8b-447f-be4c-4fdc54c966ba","Type":"ContainerDied","Data":"20d14a054a96a02cc905d79363c1f66198b1f81d1c50c2f74709defa78e57412"}
May 29 13:54:07.369864 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:07.369587    2571 scope.go:117] "RemoveContainer" containerID="f7a0270d590f69fd2d612e84ac30409bcf9559c72ade6f384252d667e7adf37a"
May 29 13:54:07.377943 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:07.377927    2571 scope.go:117] "RemoveContainer" containerID="f7a0270d590f69fd2d612e84ac30409bcf9559c72ade6f384252d667e7adf37a"
May 29 13:54:07.378223 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:54:07.378203    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7a0270d590f69fd2d612e84ac30409bcf9559c72ade6f384252d667e7adf37a\": container with ID starting with f7a0270d590f69fd2d612e84ac30409bcf9559c72ade6f384252d667e7adf37a not found: ID does not exist" containerID="f7a0270d590f69fd2d612e84ac30409bcf9559c72ade6f384252d667e7adf37a"
May 29 13:54:07.378312 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:07.378230    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7a0270d590f69fd2d612e84ac30409bcf9559c72ade6f384252d667e7adf37a"} err="failed to get container status \"f7a0270d590f69fd2d612e84ac30409bcf9559c72ade6f384252d667e7adf37a\": rpc error: code = NotFound desc = could not find container \"f7a0270d590f69fd2d612e84ac30409bcf9559c72ade6f384252d667e7adf37a\": container with ID starting with f7a0270d590f69fd2d612e84ac30409bcf9559c72ade6f384252d667e7adf37a not found: ID does not exist"
May 29 13:54:07.390090 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:07.390068    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59b894889c-cb865"]
May 29 13:54:07.392286 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:07.392264    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-59b894889c-cb865"]
May 29 13:54:08.148403 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:08.148368    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20ccc54c-7d8b-447f-be4c-4fdc54c966ba" path="/var/lib/kubelet/pods/20ccc54c-7d8b-447f-be4c-4fdc54c966ba/volumes"
May 29 13:54:24.987960 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:24.987929    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-69c897cd9b-pwrhr"]
May 29 13:54:24.988474 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:24.988238    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20ccc54c-7d8b-447f-be4c-4fdc54c966ba" containerName="manager"
May 29 13:54:24.988474 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:24.988251    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="20ccc54c-7d8b-447f-be4c-4fdc54c966ba" containerName="manager"
May 29 13:54:24.988474 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:24.988314    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="20ccc54c-7d8b-447f-be4c-4fdc54c966ba" containerName="manager"
May 29 13:54:24.997112 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:24.997088    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-69c897cd9b-pwrhr"
May 29 13:54:25.000095 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:25.000042    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\""
May 29 13:54:25.000610 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:25.000584    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-n24vl\""
May 29 13:54:25.001025 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:25.000997    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-69c897cd9b-pwrhr"]
May 29 13:54:25.151604 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:25.151568    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac32703e-80be-45e3-864f-6633814559b8-maas-api-tls\") pod \"maas-api-69c897cd9b-pwrhr\" (UID: \"ac32703e-80be-45e3-864f-6633814559b8\") " pod="opendatahub/maas-api-69c897cd9b-pwrhr"
May 29 13:54:25.151777 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:25.151673    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcxlr\" (UniqueName: \"kubernetes.io/projected/ac32703e-80be-45e3-864f-6633814559b8-kube-api-access-mcxlr\") pod \"maas-api-69c897cd9b-pwrhr\" (UID: \"ac32703e-80be-45e3-864f-6633814559b8\") " pod="opendatahub/maas-api-69c897cd9b-pwrhr"
May 29 13:54:25.253079 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:25.252975    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mcxlr\" (UniqueName: \"kubernetes.io/projected/ac32703e-80be-45e3-864f-6633814559b8-kube-api-access-mcxlr\") pod \"maas-api-69c897cd9b-pwrhr\" (UID: \"ac32703e-80be-45e3-864f-6633814559b8\") " pod="opendatahub/maas-api-69c897cd9b-pwrhr"
May 29 13:54:25.253254 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:25.253081    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac32703e-80be-45e3-864f-6633814559b8-maas-api-tls\") pod \"maas-api-69c897cd9b-pwrhr\" (UID: \"ac32703e-80be-45e3-864f-6633814559b8\") " pod="opendatahub/maas-api-69c897cd9b-pwrhr"
May 29 13:54:25.255427 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:25.255400    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac32703e-80be-45e3-864f-6633814559b8-maas-api-tls\") pod \"maas-api-69c897cd9b-pwrhr\" (UID: \"ac32703e-80be-45e3-864f-6633814559b8\") " pod="opendatahub/maas-api-69c897cd9b-pwrhr"
May 29 13:54:25.260833 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:25.260807    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcxlr\" (UniqueName: \"kubernetes.io/projected/ac32703e-80be-45e3-864f-6633814559b8-kube-api-access-mcxlr\") pod \"maas-api-69c897cd9b-pwrhr\" (UID: \"ac32703e-80be-45e3-864f-6633814559b8\") " pod="opendatahub/maas-api-69c897cd9b-pwrhr"
May 29 13:54:25.307823 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:25.307789    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-69c897cd9b-pwrhr"
May 29 13:54:25.447973 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:25.447943    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-69c897cd9b-pwrhr"]
May 29 13:54:25.449941 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:54:25.449913    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac32703e_80be_45e3_864f_6633814559b8.slice/crio-cf1da17bcbaf3d16e0e89a5c61bf9cbf72d2266260b2beac855e8888bb9a1dfe WatchSource:0}: Error finding container cf1da17bcbaf3d16e0e89a5c61bf9cbf72d2266260b2beac855e8888bb9a1dfe: Status 404 returned error can't find the container with id cf1da17bcbaf3d16e0e89a5c61bf9cbf72d2266260b2beac855e8888bb9a1dfe
May 29 13:54:25.451247 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:25.451230    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 29 13:54:26.428750 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:26.428707    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-69c897cd9b-pwrhr" event={"ID":"ac32703e-80be-45e3-864f-6633814559b8","Type":"ContainerStarted","Data":"cf1da17bcbaf3d16e0e89a5c61bf9cbf72d2266260b2beac855e8888bb9a1dfe"}
May 29 13:54:27.432982 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:27.432903    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-69c897cd9b-pwrhr" event={"ID":"ac32703e-80be-45e3-864f-6633814559b8","Type":"ContainerStarted","Data":"957cfd288d9b32da32d2ef37716a711f2176837d146f3444fdd59d745eefbbcd"}
May 29 13:54:27.433367 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:27.433009    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-69c897cd9b-pwrhr"
May 29 13:54:27.450520 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:27.450471    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="opendatahub/maas-api-69c897cd9b-pwrhr" podStartSLOduration=1.753159609 podStartE2EDuration="3.450458102s" podCreationTimestamp="2026-05-29 13:54:24 +0000 UTC" firstStartedPulling="2026-05-29 13:54:25.451390609 +0000 UTC m=+739.918460669" lastFinishedPulling="2026-05-29 13:54:27.148689096 +0000 UTC m=+741.615759162" observedRunningTime="2026-05-29 13:54:27.449602957 +0000 UTC m=+741.916673039" watchObservedRunningTime="2026-05-29 13:54:27.450458102 +0000 UTC m=+741.917528179"
May 29 13:54:33.440950 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:54:33.440914    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-69c897cd9b-pwrhr"
May 29 13:55:12.842124 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:12.842090    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6446674f67-nkqzm"]
May 29 13:55:12.848160 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:12.848138    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6446674f67-nkqzm"
May 29 13:55:12.850465 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:12.850447    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"authorino-oidc-ca\""
May 29 13:55:12.854954 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:12.854929    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6446674f67-nkqzm"]
May 29 13:55:12.916292 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:12.916254    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oidc-ca\" (UniqueName: \"kubernetes.io/configmap/e31f3533-f3da-4bc8-8ab5-0d932f6bfa08-oidc-ca\") pod \"authorino-6446674f67-nkqzm\" (UID: \"e31f3533-f3da-4bc8-8ab5-0d932f6bfa08\") " pod="kuadrant-system/authorino-6446674f67-nkqzm"
May 29 13:55:12.916470 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:12.916309    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e31f3533-f3da-4bc8-8ab5-0d932f6bfa08-tls-cert\") pod \"authorino-6446674f67-nkqzm\" (UID: \"e31f3533-f3da-4bc8-8ab5-0d932f6bfa08\") " pod="kuadrant-system/authorino-6446674f67-nkqzm"
May 29 13:55:12.916470 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:12.916406    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twv2q\" (UniqueName: \"kubernetes.io/projected/e31f3533-f3da-4bc8-8ab5-0d932f6bfa08-kube-api-access-twv2q\") pod \"authorino-6446674f67-nkqzm\" (UID: \"e31f3533-f3da-4bc8-8ab5-0d932f6bfa08\") " pod="kuadrant-system/authorino-6446674f67-nkqzm"
May 29 13:55:13.016848 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:13.016815    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-twv2q\" (UniqueName: \"kubernetes.io/projected/e31f3533-f3da-4bc8-8ab5-0d932f6bfa08-kube-api-access-twv2q\") pod \"authorino-6446674f67-nkqzm\" (UID: \"e31f3533-f3da-4bc8-8ab5-0d932f6bfa08\") " pod="kuadrant-system/authorino-6446674f67-nkqzm"
May 29 13:55:13.016964 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:13.016864    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oidc-ca\" (UniqueName: \"kubernetes.io/configmap/e31f3533-f3da-4bc8-8ab5-0d932f6bfa08-oidc-ca\") pod \"authorino-6446674f67-nkqzm\" (UID: \"e31f3533-f3da-4bc8-8ab5-0d932f6bfa08\") " pod="kuadrant-system/authorino-6446674f67-nkqzm"
May 29 13:55:13.016964 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:13.016904    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e31f3533-f3da-4bc8-8ab5-0d932f6bfa08-tls-cert\") pod \"authorino-6446674f67-nkqzm\" (UID: \"e31f3533-f3da-4bc8-8ab5-0d932f6bfa08\") " pod="kuadrant-system/authorino-6446674f67-nkqzm"
May 29 13:55:13.017666 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:13.017647    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oidc-ca\" (UniqueName: \"kubernetes.io/configmap/e31f3533-f3da-4bc8-8ab5-0d932f6bfa08-oidc-ca\") pod \"authorino-6446674f67-nkqzm\" (UID: \"e31f3533-f3da-4bc8-8ab5-0d932f6bfa08\") " pod="kuadrant-system/authorino-6446674f67-nkqzm"
May 29 13:55:13.019331 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:13.019311    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e31f3533-f3da-4bc8-8ab5-0d932f6bfa08-tls-cert\") pod \"authorino-6446674f67-nkqzm\" (UID: \"e31f3533-f3da-4bc8-8ab5-0d932f6bfa08\") " pod="kuadrant-system/authorino-6446674f67-nkqzm"
May 29 13:55:13.026609 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:13.026579    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twv2q\" (UniqueName: \"kubernetes.io/projected/e31f3533-f3da-4bc8-8ab5-0d932f6bfa08-kube-api-access-twv2q\") pod \"authorino-6446674f67-nkqzm\" (UID: \"e31f3533-f3da-4bc8-8ab5-0d932f6bfa08\") " pod="kuadrant-system/authorino-6446674f67-nkqzm"
May 29 13:55:13.157455 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:13.157366    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6446674f67-nkqzm"
May 29 13:55:13.283197 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:13.283169    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6446674f67-nkqzm"]
May 29 13:55:13.285799 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:55:13.285761    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode31f3533_f3da_4bc8_8ab5_0d932f6bfa08.slice/crio-abeb76fb1052d4532e851c04c7939b138fe4f100debc249e398d8524cd27abad WatchSource:0}: Error finding container abeb76fb1052d4532e851c04c7939b138fe4f100debc249e398d8524cd27abad: Status 404 returned error can't find the container with id abeb76fb1052d4532e851c04c7939b138fe4f100debc249e398d8524cd27abad
May 29 13:55:13.570733 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:13.570650    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6446674f67-nkqzm" event={"ID":"e31f3533-f3da-4bc8-8ab5-0d932f6bfa08","Type":"ContainerStarted","Data":"abeb76fb1052d4532e851c04c7939b138fe4f100debc249e398d8524cd27abad"}
May 29 13:55:14.574289 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:14.574250    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6446674f67-nkqzm" event={"ID":"e31f3533-f3da-4bc8-8ab5-0d932f6bfa08","Type":"ContainerStarted","Data":"2ce8ee4f7fbf0674dc657af964dee1305d5124f32b5f2438219584fc22af7b09"}
May 29 13:55:14.590875 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:14.590827    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-6446674f67-nkqzm" podStartSLOduration=2.21556032 podStartE2EDuration="2.590811867s" podCreationTimestamp="2026-05-29 13:55:12 +0000 UTC" firstStartedPulling="2026-05-29 13:55:13.287043329 +0000 UTC m=+787.754113389" lastFinishedPulling="2026-05-29 13:55:13.662294873 +0000 UTC m=+788.129364936" observedRunningTime="2026-05-29 13:55:14.590237526 +0000 UTC m=+789.057307608" watchObservedRunningTime="2026-05-29 13:55:14.590811867 +0000 UTC m=+789.057881948"
May 29 13:55:14.620339 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:14.620301    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5957664df-xcqcb"]
May 29 13:55:14.620612 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:14.620577    2571 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/authorino-5957664df-xcqcb" podUID="6e606b8c-8816-4f71-8bac-c0fd558da6e7" containerName="authorino" containerID="cri-o://f2a6a4f4b02e12805c2ea0483e65434097fbda21f92157b9066de85915752b17" gracePeriod=30
May 29 13:55:14.864544 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:14.864513    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5957664df-xcqcb"
May 29 13:55:14.931505 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:14.931458    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4rg4\" (UniqueName: \"kubernetes.io/projected/6e606b8c-8816-4f71-8bac-c0fd558da6e7-kube-api-access-f4rg4\") pod \"6e606b8c-8816-4f71-8bac-c0fd558da6e7\" (UID: \"6e606b8c-8816-4f71-8bac-c0fd558da6e7\") "
May 29 13:55:14.931679 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:14.931549    2571 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e606b8c-8816-4f71-8bac-c0fd558da6e7-tls-cert\") pod \"6e606b8c-8816-4f71-8bac-c0fd558da6e7\" (UID: \"6e606b8c-8816-4f71-8bac-c0fd558da6e7\") "
May 29 13:55:14.933643 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:14.933613    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e606b8c-8816-4f71-8bac-c0fd558da6e7-kube-api-access-f4rg4" (OuterVolumeSpecName: "kube-api-access-f4rg4") pod "6e606b8c-8816-4f71-8bac-c0fd558da6e7" (UID: "6e606b8c-8816-4f71-8bac-c0fd558da6e7"). InnerVolumeSpecName "kube-api-access-f4rg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 29 13:55:14.942075 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:14.942029    2571 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e606b8c-8816-4f71-8bac-c0fd558da6e7-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "6e606b8c-8816-4f71-8bac-c0fd558da6e7" (UID: "6e606b8c-8816-4f71-8bac-c0fd558da6e7"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 29 13:55:15.032146 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:15.032109    2571 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e606b8c-8816-4f71-8bac-c0fd558da6e7-tls-cert\") on node \"ip-10-0-141-1.ec2.internal\" DevicePath \"\""
May 29 13:55:15.032146 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:15.032146    2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4rg4\" (UniqueName: \"kubernetes.io/projected/6e606b8c-8816-4f71-8bac-c0fd558da6e7-kube-api-access-f4rg4\") on node \"ip-10-0-141-1.ec2.internal\" DevicePath \"\""
May 29 13:55:15.578692 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:15.578660    2571 generic.go:358] "Generic (PLEG): container finished" podID="6e606b8c-8816-4f71-8bac-c0fd558da6e7" containerID="f2a6a4f4b02e12805c2ea0483e65434097fbda21f92157b9066de85915752b17" exitCode=0
May 29 13:55:15.579148 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:15.578715    2571 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5957664df-xcqcb"
May 29 13:55:15.579148 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:15.578745    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5957664df-xcqcb" event={"ID":"6e606b8c-8816-4f71-8bac-c0fd558da6e7","Type":"ContainerDied","Data":"f2a6a4f4b02e12805c2ea0483e65434097fbda21f92157b9066de85915752b17"}
May 29 13:55:15.579148 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:15.578786    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5957664df-xcqcb" event={"ID":"6e606b8c-8816-4f71-8bac-c0fd558da6e7","Type":"ContainerDied","Data":"d8f0d7df5626d847ad68ce5df09704c1f9d16c3b6b5d5424f990500ae13320e5"}
May 29 13:55:15.579148 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:15.578806    2571 scope.go:117] "RemoveContainer" containerID="f2a6a4f4b02e12805c2ea0483e65434097fbda21f92157b9066de85915752b17"
May 29 13:55:15.586739 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:15.586720    2571 scope.go:117] "RemoveContainer" containerID="f2a6a4f4b02e12805c2ea0483e65434097fbda21f92157b9066de85915752b17"
May 29 13:55:15.586982 ip-10-0-141-1 kubenswrapper[2571]: E0529 13:55:15.586962    2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2a6a4f4b02e12805c2ea0483e65434097fbda21f92157b9066de85915752b17\": container with ID starting with f2a6a4f4b02e12805c2ea0483e65434097fbda21f92157b9066de85915752b17 not found: ID does not exist" containerID="f2a6a4f4b02e12805c2ea0483e65434097fbda21f92157b9066de85915752b17"
May 29 13:55:15.587047 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:15.586994    2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2a6a4f4b02e12805c2ea0483e65434097fbda21f92157b9066de85915752b17"} err="failed to get container status \"f2a6a4f4b02e12805c2ea0483e65434097fbda21f92157b9066de85915752b17\": rpc error: code = NotFound desc = could not find container \"f2a6a4f4b02e12805c2ea0483e65434097fbda21f92157b9066de85915752b17\": container with ID starting with f2a6a4f4b02e12805c2ea0483e65434097fbda21f92157b9066de85915752b17 not found: ID does not exist"
May 29 13:55:15.598334 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:15.598311    2571 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5957664df-xcqcb"]
May 29 13:55:15.602487 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:15.602467    2571 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5957664df-xcqcb"]
May 29 13:55:16.148653 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:16.148623    2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e606b8c-8816-4f71-8bac-c0fd558da6e7" path="/var/lib/kubelet/pods/6e606b8c-8816-4f71-8bac-c0fd558da6e7/volumes"
May 29 13:55:38.981390 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:38.981300    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f"]
May 29 13:55:38.981970 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:38.981727    2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e606b8c-8816-4f71-8bac-c0fd558da6e7" containerName="authorino"
May 29 13:55:38.981970 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:38.981744    2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e606b8c-8816-4f71-8bac-c0fd558da6e7" containerName="authorino"
May 29 13:55:38.981970 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:38.981816    2571 memory_manager.go:356] "RemoveStaleState removing state" podUID="6e606b8c-8816-4f71-8bac-c0fd558da6e7" containerName="authorino"
May 29 13:55:38.984981 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:38.984961    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f"
May 29 13:55:38.987471 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:38.987451    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
May 29 13:55:38.987578 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:38.987518    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vfxdw\""
May 29 13:55:38.987578 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:38.987451    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
May 29 13:55:38.987578 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:38.987551    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
May 29 13:55:38.994033 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:38.994012    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f"]
May 29 13:55:39.126272 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:39.126234    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f\" (UID: \"7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1\") " pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f"
May 29 13:55:39.126495 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:39.126287    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6z8c\" (UniqueName: \"kubernetes.io/projected/7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1-kube-api-access-f6z8c\") pod \"e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f\" (UID: \"7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1\") " pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f"
May 29 13:55:39.126495 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:39.126377    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1-home\") pod \"e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f\" (UID: \"7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1\") " pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f"
May 29 13:55:39.126495 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:39.126426    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1-dshm\") pod \"e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f\" (UID: \"7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1\") " pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f"
May 29 13:55:39.126495 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:39.126451    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f\" (UID: \"7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1\") " pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f"
May 29 13:55:39.126495 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:39.126484    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1-kserve-provision-location\") pod \"e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f\" (UID: \"7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1\") " pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f"
May 29 13:55:39.227321 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:39.227280    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1-home\") pod \"e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f\" (UID: \"7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1\") " pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f"
May 29 13:55:39.227321 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:39.227324    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1-dshm\") pod \"e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f\" (UID: \"7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1\") " pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f"
May 29 13:55:39.227554 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:39.227342    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f\" (UID: \"7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1\") " pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f"
May 29 13:55:39.227554 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:39.227358    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1-kserve-provision-location\") pod \"e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f\" (UID: \"7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1\") " pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f"
May 29 13:55:39.227554 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:39.227411    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f\" (UID: \"7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1\") " pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f"
May 29 13:55:39.227554 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:39.227436    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f6z8c\" (UniqueName: \"kubernetes.io/projected/7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1-kube-api-access-f6z8c\") pod \"e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f\" (UID: \"7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1\") " pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f"
May 29 13:55:39.227754 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:39.227687    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1-home\") pod \"e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f\" (UID: \"7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1\") " pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f"
May 29 13:55:39.227798 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:39.227772    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1-model-cache\") pod \"e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f\" (UID: \"7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1\") " pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f"
May 29 13:55:39.227834 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:39.227817    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1-kserve-provision-location\") pod \"e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f\" (UID: \"7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1\") " pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f"
May 29 13:55:39.229681 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:39.229661    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1-dshm\") pod \"e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f\" (UID: \"7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1\") " pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f"
May 29 13:55:39.229852 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:39.229835    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f\" (UID: \"7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1\") " pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f"
May 29 13:55:39.234772 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:39.234723    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6z8c\" (UniqueName: \"kubernetes.io/projected/7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1-kube-api-access-f6z8c\") pod \"e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f\" (UID: \"7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1\") " pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f"
May 29 13:55:39.295973 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:39.295936    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f"
May 29 13:55:39.421215 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:39.421191    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f"]
May 29 13:55:39.423341 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:55:39.423313    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa5084c_0c94_4148_b6a7_bf9d2aec9cc1.slice/crio-0a78f5a20672fab6ca0b015ac3ed0c6b696eb3cfa947a8df47b3c90544c2bac4 WatchSource:0}: Error finding container 0a78f5a20672fab6ca0b015ac3ed0c6b696eb3cfa947a8df47b3c90544c2bac4: Status 404 returned error can't find the container with id 0a78f5a20672fab6ca0b015ac3ed0c6b696eb3cfa947a8df47b3c90544c2bac4
May 29 13:55:39.662479 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:39.662442    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f" event={"ID":"7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1","Type":"ContainerStarted","Data":"0a78f5a20672fab6ca0b015ac3ed0c6b696eb3cfa947a8df47b3c90544c2bac4"}
May 29 13:55:42.674322 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:42.674284    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f" event={"ID":"7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1","Type":"ContainerStarted","Data":"a0ad1d8ec9713df8fb4248471f5ad84cebd205598980fe276707128ed216d642"}
May 29 13:55:43.364958 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:43.364926    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn"]
May 29 13:55:43.369914 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:43.369890    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn"
May 29 13:55:43.372265 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:43.372244    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
May 29 13:55:43.380213 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:43.380193    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn"]
May 29 13:55:43.461465 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:43.461429    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ffa5a7d-b1b4-4963-a969-e60c24270fe2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn\" (UID: \"3ffa5a7d-b1b4-4963-a969-e60c24270fe2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn"
May 29 13:55:43.461624 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:43.461481    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvq2c\" (UniqueName: \"kubernetes.io/projected/3ffa5a7d-b1b4-4963-a969-e60c24270fe2-kube-api-access-dvq2c\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn\" (UID: \"3ffa5a7d-b1b4-4963-a969-e60c24270fe2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn"
May 29 13:55:43.461624 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:43.461568    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ffa5a7d-b1b4-4963-a969-e60c24270fe2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn\" (UID: \"3ffa5a7d-b1b4-4963-a969-e60c24270fe2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn"
May 29 13:55:43.461624 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:43.461604    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ffa5a7d-b1b4-4963-a969-e60c24270fe2-kserve-provision-location\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn\" (UID: \"3ffa5a7d-b1b4-4963-a969-e60c24270fe2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn"
May 29 13:55:43.461731 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:43.461625    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ffa5a7d-b1b4-4963-a969-e60c24270fe2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn\" (UID: \"3ffa5a7d-b1b4-4963-a969-e60c24270fe2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn"
May 29 13:55:43.461731 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:43.461707    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ffa5a7d-b1b4-4963-a969-e60c24270fe2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn\" (UID: \"3ffa5a7d-b1b4-4963-a969-e60c24270fe2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn"
May 29 13:55:43.563140 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:43.563102    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ffa5a7d-b1b4-4963-a969-e60c24270fe2-kserve-provision-location\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn\" (UID: \"3ffa5a7d-b1b4-4963-a969-e60c24270fe2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn"
May 29 13:55:43.563284 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:43.563145    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ffa5a7d-b1b4-4963-a969-e60c24270fe2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn\" (UID: \"3ffa5a7d-b1b4-4963-a969-e60c24270fe2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn"
May 29 13:55:43.563356 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:43.563275    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ffa5a7d-b1b4-4963-a969-e60c24270fe2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn\" (UID: \"3ffa5a7d-b1b4-4963-a969-e60c24270fe2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn"
May 29 13:55:43.563356 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:43.563340    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ffa5a7d-b1b4-4963-a969-e60c24270fe2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn\" (UID: \"3ffa5a7d-b1b4-4963-a969-e60c24270fe2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn"
May 29 13:55:43.563471 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:43.563393    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dvq2c\" (UniqueName: \"kubernetes.io/projected/3ffa5a7d-b1b4-4963-a969-e60c24270fe2-kube-api-access-dvq2c\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn\" (UID: \"3ffa5a7d-b1b4-4963-a969-e60c24270fe2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn"
May 29 13:55:43.563471 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:43.563456    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ffa5a7d-b1b4-4963-a969-e60c24270fe2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn\" (UID: \"3ffa5a7d-b1b4-4963-a969-e60c24270fe2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn"
May 29 13:55:43.563471 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:43.563468    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ffa5a7d-b1b4-4963-a969-e60c24270fe2-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn\" (UID: \"3ffa5a7d-b1b4-4963-a969-e60c24270fe2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn"
May 29 13:55:43.563625 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:43.563518    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ffa5a7d-b1b4-4963-a969-e60c24270fe2-kserve-provision-location\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn\" (UID: \"3ffa5a7d-b1b4-4963-a969-e60c24270fe2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn"
May 29 13:55:43.563756 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:43.563726    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ffa5a7d-b1b4-4963-a969-e60c24270fe2-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn\" (UID: \"3ffa5a7d-b1b4-4963-a969-e60c24270fe2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn"
May 29 13:55:43.565503 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:43.565479    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ffa5a7d-b1b4-4963-a969-e60c24270fe2-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn\" (UID: \"3ffa5a7d-b1b4-4963-a969-e60c24270fe2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn"
May 29 13:55:43.565851 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:43.565833    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ffa5a7d-b1b4-4963-a969-e60c24270fe2-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn\" (UID: \"3ffa5a7d-b1b4-4963-a969-e60c24270fe2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn"
May 29 13:55:43.571977 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:43.571953    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvq2c\" (UniqueName: \"kubernetes.io/projected/3ffa5a7d-b1b4-4963-a969-e60c24270fe2-kube-api-access-dvq2c\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn\" (UID: \"3ffa5a7d-b1b4-4963-a969-e60c24270fe2\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn"
May 29 13:55:43.679037 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:43.678946    2571 generic.go:358] "Generic (PLEG): container finished" podID="7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1" containerID="a0ad1d8ec9713df8fb4248471f5ad84cebd205598980fe276707128ed216d642" exitCode=0
May 29 13:55:43.679037 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:43.679008    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f" event={"ID":"7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1","Type":"ContainerDied","Data":"a0ad1d8ec9713df8fb4248471f5ad84cebd205598980fe276707128ed216d642"}
May 29 13:55:43.679550 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:43.679389    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn"
May 29 13:55:43.804238 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:43.804213    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn"]
May 29 13:55:43.806640 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:55:43.806612    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ffa5a7d_b1b4_4963_a969_e60c24270fe2.slice/crio-c9e2f7697a23a4be4b345ab0d062aa197f0f06cb0e837ee94dd56533520b674d WatchSource:0}: Error finding container c9e2f7697a23a4be4b345ab0d062aa197f0f06cb0e837ee94dd56533520b674d: Status 404 returned error can't find the container with id c9e2f7697a23a4be4b345ab0d062aa197f0f06cb0e837ee94dd56533520b674d
May 29 13:55:44.683870 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:44.683823    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn" event={"ID":"3ffa5a7d-b1b4-4963-a969-e60c24270fe2","Type":"ContainerStarted","Data":"54f28bebfe1442b87f926e69c89f202dadff2b10d816b2e060d0bec7c8722e03"}
May 29 13:55:44.683870 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:44.683865    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn" event={"ID":"3ffa5a7d-b1b4-4963-a969-e60c24270fe2","Type":"ContainerStarted","Data":"c9e2f7697a23a4be4b345ab0d062aa197f0f06cb0e837ee94dd56533520b674d"}
May 29 13:55:45.688522 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:45.688489    2571 generic.go:358] "Generic (PLEG): container finished" podID="3ffa5a7d-b1b4-4963-a969-e60c24270fe2" containerID="54f28bebfe1442b87f926e69c89f202dadff2b10d816b2e060d0bec7c8722e03" exitCode=0
May 29 13:55:45.688980 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:45.688568    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn" event={"ID":"3ffa5a7d-b1b4-4963-a969-e60c24270fe2","Type":"ContainerDied","Data":"54f28bebfe1442b87f926e69c89f202dadff2b10d816b2e060d0bec7c8722e03"}
May 29 13:55:45.690315 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:45.690291    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f" event={"ID":"7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1","Type":"ContainerStarted","Data":"e034d41a3263f53c820415230c484f864ba45ee659bf440d2852f4181928d14b"}
May 29 13:55:45.726633 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:45.726574    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f" podStartSLOduration=2.264468421 podStartE2EDuration="7.726560547s" podCreationTimestamp="2026-05-29 13:55:38 +0000 UTC" firstStartedPulling="2026-05-29 13:55:39.42493808 +0000 UTC m=+813.892008139" lastFinishedPulling="2026-05-29 13:55:44.887030206 +0000 UTC m=+819.354100265" observedRunningTime="2026-05-29 13:55:45.726530269 +0000 UTC m=+820.193600352" watchObservedRunningTime="2026-05-29 13:55:45.726560547 +0000 UTC m=+820.193630628"
May 29 13:55:46.694471 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:46.694432    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn" event={"ID":"3ffa5a7d-b1b4-4963-a969-e60c24270fe2","Type":"ContainerStarted","Data":"da24a4cff86f6f5e828052ad8a4cd2c38451b375c52f1055255998dcfaad0eaf"}
May 29 13:55:46.718064 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:46.718004    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn" podStartSLOduration=3.462514401 podStartE2EDuration="3.717989324s" podCreationTimestamp="2026-05-29 13:55:43 +0000 UTC" firstStartedPulling="2026-05-29 13:55:45.689142752 +0000 UTC m=+820.156212811" lastFinishedPulling="2026-05-29 13:55:45.94461766 +0000 UTC m=+820.411687734" observedRunningTime="2026-05-29 13:55:46.716764081 +0000 UTC m=+821.183834163" watchObservedRunningTime="2026-05-29 13:55:46.717989324 +0000 UTC m=+821.185059405"
May 29 13:55:49.296076 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:49.296022    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f"
May 29 13:55:49.296076 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:49.296080    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f"
May 29 13:55:49.308209 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:49.308187    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f"
May 29 13:55:49.715518 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:49.715489    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f"
May 29 13:55:53.680156 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:53.680122    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn"
May 29 13:55:53.680156 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:53.680161    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn"
May 29 13:55:53.692350 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:53.692330    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn"
May 29 13:55:53.727839 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:53.727813    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn"
May 29 13:55:58.883732 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:58.883693    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs"]
May 29 13:55:58.890735 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:58.890710    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs"
May 29 13:55:58.893798 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:58.893779    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
May 29 13:55:58.915296 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:58.915267    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs"]
May 29 13:55:58.993566 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:58.993523    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f9f4090-5511-4177-8b35-cea137b15f78-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs\" (UID: \"2f9f4090-5511-4177-8b35-cea137b15f78\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs"
May 29 13:55:58.993749 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:58.993580    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f9f4090-5511-4177-8b35-cea137b15f78-home\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs\" (UID: \"2f9f4090-5511-4177-8b35-cea137b15f78\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs"
May 29 13:55:58.993749 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:58.993612    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgqrg\" (UniqueName: \"kubernetes.io/projected/2f9f4090-5511-4177-8b35-cea137b15f78-kube-api-access-fgqrg\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs\" (UID: \"2f9f4090-5511-4177-8b35-cea137b15f78\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs"
May 29 13:55:58.993749 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:58.993675    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f9f4090-5511-4177-8b35-cea137b15f78-model-cache\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs\" (UID: \"2f9f4090-5511-4177-8b35-cea137b15f78\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs"
May 29 13:55:58.993749 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:58.993697    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f9f4090-5511-4177-8b35-cea137b15f78-dshm\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs\" (UID: \"2f9f4090-5511-4177-8b35-cea137b15f78\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs"
May 29 13:55:58.993749 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:58.993721    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f9f4090-5511-4177-8b35-cea137b15f78-kserve-provision-location\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs\" (UID: \"2f9f4090-5511-4177-8b35-cea137b15f78\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs"
May 29 13:55:59.094614 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:59.094579    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f9f4090-5511-4177-8b35-cea137b15f78-model-cache\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs\" (UID: \"2f9f4090-5511-4177-8b35-cea137b15f78\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs"
May 29 13:55:59.094614 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:59.094613    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f9f4090-5511-4177-8b35-cea137b15f78-dshm\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs\" (UID: \"2f9f4090-5511-4177-8b35-cea137b15f78\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs"
May 29 13:55:59.094830 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:59.094637    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f9f4090-5511-4177-8b35-cea137b15f78-kserve-provision-location\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs\" (UID: \"2f9f4090-5511-4177-8b35-cea137b15f78\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs"
May 29 13:55:59.094830 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:59.094682    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f9f4090-5511-4177-8b35-cea137b15f78-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs\" (UID: \"2f9f4090-5511-4177-8b35-cea137b15f78\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs"
May 29 13:55:59.094830 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:59.094706    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f9f4090-5511-4177-8b35-cea137b15f78-home\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs\" (UID: \"2f9f4090-5511-4177-8b35-cea137b15f78\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs"
May 29 13:55:59.094830 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:59.094729    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fgqrg\" (UniqueName: \"kubernetes.io/projected/2f9f4090-5511-4177-8b35-cea137b15f78-kube-api-access-fgqrg\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs\" (UID: \"2f9f4090-5511-4177-8b35-cea137b15f78\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs"
May 29 13:55:59.095064 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:59.095023    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f9f4090-5511-4177-8b35-cea137b15f78-model-cache\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs\" (UID: \"2f9f4090-5511-4177-8b35-cea137b15f78\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs"
May 29 13:55:59.095134 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:59.095099    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f9f4090-5511-4177-8b35-cea137b15f78-kserve-provision-location\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs\" (UID: \"2f9f4090-5511-4177-8b35-cea137b15f78\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs"
May 29 13:55:59.095193 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:59.095152    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f9f4090-5511-4177-8b35-cea137b15f78-home\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs\" (UID: \"2f9f4090-5511-4177-8b35-cea137b15f78\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs"
May 29 13:55:59.096866 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:59.096844    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f9f4090-5511-4177-8b35-cea137b15f78-dshm\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs\" (UID: \"2f9f4090-5511-4177-8b35-cea137b15f78\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs"
May 29 13:55:59.097120 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:59.097102    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f9f4090-5511-4177-8b35-cea137b15f78-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs\" (UID: \"2f9f4090-5511-4177-8b35-cea137b15f78\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs"
May 29 13:55:59.103101 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:59.103082    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgqrg\" (UniqueName: \"kubernetes.io/projected/2f9f4090-5511-4177-8b35-cea137b15f78-kube-api-access-fgqrg\") pod \"facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs\" (UID: \"2f9f4090-5511-4177-8b35-cea137b15f78\") " pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs"
May 29 13:55:59.200903 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:59.200815    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs"
May 29 13:55:59.325192 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:59.325159    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs"]
May 29 13:55:59.328318 ip-10-0-141-1 kubenswrapper[2571]: W0529 13:55:59.328286    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9f4090_5511_4177_8b35_cea137b15f78.slice/crio-1ccef67d67223bd4f0b2b10b0f754ea569c265146dff41d57b5fe41687e539ff WatchSource:0}: Error finding container 1ccef67d67223bd4f0b2b10b0f754ea569c265146dff41d57b5fe41687e539ff: Status 404 returned error can't find the container with id 1ccef67d67223bd4f0b2b10b0f754ea569c265146dff41d57b5fe41687e539ff
May 29 13:55:59.736936 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:59.736899    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs" event={"ID":"2f9f4090-5511-4177-8b35-cea137b15f78","Type":"ContainerStarted","Data":"d236f21cdfd321e260058fe74403ad77f2e6e74c4594311bd40d03fd5a0bdf11"}
May 29 13:55:59.736936 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:55:59.736942    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs" event={"ID":"2f9f4090-5511-4177-8b35-cea137b15f78","Type":"ContainerStarted","Data":"1ccef67d67223bd4f0b2b10b0f754ea569c265146dff41d57b5fe41687e539ff"}
May 29 13:56:01.744606 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:56:01.744576    2571 generic.go:358] "Generic (PLEG): container finished" podID="2f9f4090-5511-4177-8b35-cea137b15f78" containerID="d236f21cdfd321e260058fe74403ad77f2e6e74c4594311bd40d03fd5a0bdf11" exitCode=0
May 29 13:56:01.744977 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:56:01.744648    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs" event={"ID":"2f9f4090-5511-4177-8b35-cea137b15f78","Type":"ContainerDied","Data":"d236f21cdfd321e260058fe74403ad77f2e6e74c4594311bd40d03fd5a0bdf11"}
May 29 13:56:02.749814 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:56:02.749784    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs" event={"ID":"2f9f4090-5511-4177-8b35-cea137b15f78","Type":"ContainerStarted","Data":"1222e759f7102bbcf21b478d4cad83f1099aae1d59379a3ce44a1bece47d9574"}
May 29 13:56:02.769239 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:56:02.769191    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs" podStartSLOduration=4.466819134 podStartE2EDuration="4.769175115s" podCreationTimestamp="2026-05-29 13:55:58 +0000 UTC" firstStartedPulling="2026-05-29 13:56:01.745240593 +0000 UTC m=+836.212310651" lastFinishedPulling="2026-05-29 13:56:02.047596568 +0000 UTC m=+836.514666632" observedRunningTime="2026-05-29 13:56:02.768348045 +0000 UTC m=+837.235418128" watchObservedRunningTime="2026-05-29 13:56:02.769175115 +0000 UTC m=+837.236245196"
May 29 13:56:09.200963 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:56:09.200928    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs"
May 29 13:56:09.200963 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:56:09.200966    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs"
May 29 13:56:09.213708 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:56:09.213684    2571 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs"
May 29 13:56:09.782077 ip-10-0-141-1 kubenswrapper[2571]: I0529 13:56:09.782033    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs"
May 29 14:02:13.466111 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:13.466081    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6446674f67-nkqzm_e31f3533-f3da-4bc8-8ab5-0d932f6bfa08/authorino/0.log"
May 29 14:02:18.482112 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:18.482081    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-69c897cd9b-pwrhr_ac32703e-80be-45e3-864f-6633814559b8/maas-api/0.log"
May 29 14:02:18.698160 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:18.698132    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8db997c58-nhpk4_65b430b8-f603-42c1-bc73-89cb54abb5a1/manager/0.log"
May 29 14:02:18.812108 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:18.812002    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-6577596cd5-dfvjf_ec7a7005-5d22-4311-87b9-2faf1e135e15/server/0.log"
May 29 14:02:18.928839 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:18.928788    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-76588b95-g42dw_59222dfa-c8a8-4d60-91eb-467a6c42f78a/manager/0.log"
May 29 14:02:19.187365 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:19.187341    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cf69667cd-pglk6_326d47a7-e9cf-4834-9b3e-11812596c49e/copy-manifests/0.log"
May 29 14:02:19.193482 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:19.193458    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cf69667cd-pglk6_326d47a7-e9cf-4834-9b3e-11812596c49e/manager/0.log"
May 29 14:02:20.711680 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:20.711644    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6446674f67-nkqzm_e31f3533-f3da-4bc8-8ab5-0d932f6bfa08/authorino/0.log"
May 29 14:02:21.052802 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:21.052724    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6cb54b5c86-zxjxr_d9ede80d-22ba-4e81-81c1-15b8b5631ab7/kuadrant-console-plugin/0.log"
May 29 14:02:22.087215 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:22.087185    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-855f784d5b-5fqkq_deae1b19-73c8-40fc-a0cc-e7663d86c726/kube-auth-proxy/0.log"
May 29 14:02:23.121002 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:23.120973    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f_7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1/storage-initializer/0.log"
May 29 14:02:23.128739 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:23.128719    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-84f46c7fb4-qxt4f_7fa5084c-0c94-4148-b6a7-bf9d2aec9cc1/main/0.log"
May 29 14:02:23.247150 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:23.247122    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn_3ffa5a7d-b1b4-4963-a969-e60c24270fe2/storage-initializer/0.log"
May 29 14:02:23.255191 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:23.255171    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-bdd7f9gdqqn_3ffa5a7d-b1b4-4963-a969-e60c24270fe2/main/0.log"
May 29 14:02:23.369755 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:23.369728    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs_2f9f4090-5511-4177-8b35-cea137b15f78/main/0.log"
May 29 14:02:23.377403 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:23.377337    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-59dcdf78f5-hbcxs_2f9f4090-5511-4177-8b35-cea137b15f78/storage-initializer/0.log"
May 29 14:02:27.749813 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:27.749776    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mcwqr/must-gather-64n88"]
May 29 14:02:27.753183 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:27.753158    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mcwqr/must-gather-64n88"
May 29 14:02:27.762204 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:27.762173    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mcwqr\"/\"kube-root-ca.crt\""
May 29 14:02:27.762204 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:27.762182    2571 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mcwqr\"/\"openshift-service-ca.crt\""
May 29 14:02:27.769003 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:27.768981    2571 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mcwqr\"/\"default-dockercfg-846cz\""
May 29 14:02:27.771221 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:27.771204    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mcwqr/must-gather-64n88"]
May 29 14:02:27.805324 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:27.805301    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75j28\" (UniqueName: \"kubernetes.io/projected/55ed117a-80a4-4f3c-837a-ec5ad0c4566c-kube-api-access-75j28\") pod \"must-gather-64n88\" (UID: \"55ed117a-80a4-4f3c-837a-ec5ad0c4566c\") " pod="openshift-must-gather-mcwqr/must-gather-64n88"
May 29 14:02:27.805424 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:27.805344    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/55ed117a-80a4-4f3c-837a-ec5ad0c4566c-must-gather-output\") pod \"must-gather-64n88\" (UID: \"55ed117a-80a4-4f3c-837a-ec5ad0c4566c\") " pod="openshift-must-gather-mcwqr/must-gather-64n88"
May 29 14:02:27.906293 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:27.906244    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-75j28\" (UniqueName: \"kubernetes.io/projected/55ed117a-80a4-4f3c-837a-ec5ad0c4566c-kube-api-access-75j28\") pod \"must-gather-64n88\" (UID: \"55ed117a-80a4-4f3c-837a-ec5ad0c4566c\") " pod="openshift-must-gather-mcwqr/must-gather-64n88"
May 29 14:02:27.906472 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:27.906400    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/55ed117a-80a4-4f3c-837a-ec5ad0c4566c-must-gather-output\") pod \"must-gather-64n88\" (UID: \"55ed117a-80a4-4f3c-837a-ec5ad0c4566c\") " pod="openshift-must-gather-mcwqr/must-gather-64n88"
May 29 14:02:27.906747 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:27.906728    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/55ed117a-80a4-4f3c-837a-ec5ad0c4566c-must-gather-output\") pod \"must-gather-64n88\" (UID: \"55ed117a-80a4-4f3c-837a-ec5ad0c4566c\") " pod="openshift-must-gather-mcwqr/must-gather-64n88"
May 29 14:02:27.922606 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:27.922573    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75j28\" (UniqueName: \"kubernetes.io/projected/55ed117a-80a4-4f3c-837a-ec5ad0c4566c-kube-api-access-75j28\") pod \"must-gather-64n88\" (UID: \"55ed117a-80a4-4f3c-837a-ec5ad0c4566c\") " pod="openshift-must-gather-mcwqr/must-gather-64n88"
May 29 14:02:28.062615 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:28.062525    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mcwqr/must-gather-64n88"
May 29 14:02:28.190662 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:28.190637    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mcwqr/must-gather-64n88"]
May 29 14:02:28.192850 ip-10-0-141-1 kubenswrapper[2571]: W0529 14:02:28.192826    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55ed117a_80a4_4f3c_837a_ec5ad0c4566c.slice/crio-8c2b7ae73971322d5bea67fd068a73a92a4d44e6d9e4b65fa8656d4878f01166 WatchSource:0}: Error finding container 8c2b7ae73971322d5bea67fd068a73a92a4d44e6d9e4b65fa8656d4878f01166: Status 404 returned error can't find the container with id 8c2b7ae73971322d5bea67fd068a73a92a4d44e6d9e4b65fa8656d4878f01166
May 29 14:02:28.194457 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:28.194442    2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 29 14:02:28.943098 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:28.943064    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mcwqr/must-gather-64n88" event={"ID":"55ed117a-80a4-4f3c-837a-ec5ad0c4566c","Type":"ContainerStarted","Data":"8c2b7ae73971322d5bea67fd068a73a92a4d44e6d9e4b65fa8656d4878f01166"}
May 29 14:02:29.948624 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:29.948583    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mcwqr/must-gather-64n88" event={"ID":"55ed117a-80a4-4f3c-837a-ec5ad0c4566c","Type":"ContainerStarted","Data":"4313158573a0caae0c1e1d36493b5eb69ac085a3cea1cfad757a201b12fdb8bf"}
May 29 14:02:29.950069 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:29.950011    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mcwqr/must-gather-64n88" event={"ID":"55ed117a-80a4-4f3c-837a-ec5ad0c4566c","Type":"ContainerStarted","Data":"55e18d009a427a59180f9e17301f046de9e04dea15d16537522df7459d97f5c8"}
May 29 14:02:29.968133 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:29.968083    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-mcwqr/must-gather-64n88" podStartSLOduration=2.183257679 podStartE2EDuration="2.968065824s" podCreationTimestamp="2026-05-29 14:02:27 +0000 UTC" firstStartedPulling="2026-05-29 14:02:28.194597466 +0000 UTC m=+1222.661667526" lastFinishedPulling="2026-05-29 14:02:28.979405612 +0000 UTC m=+1223.446475671" observedRunningTime="2026-05-29 14:02:29.966239162 +0000 UTC m=+1224.433309245" watchObservedRunningTime="2026-05-29 14:02:29.968065824 +0000 UTC m=+1224.435135938"
May 29 14:02:30.807391 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:30.807360    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-w4pw2_0d8451cb-e6fd-48aa-a319-bca020321b7a/global-pull-secret-syncer/0.log"
May 29 14:02:30.918277 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:30.918253    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bz56r_2d022803-adf3-4686-8136-a57ab660afad/konnectivity-agent/0.log"
May 29 14:02:31.010884 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:31.010836    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-1.ec2.internal_adf64b7f1c8724ca5ea45aa00abb85e2/haproxy/0.log"
May 29 14:02:34.804866 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:34.804829    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6446674f67-nkqzm_e31f3533-f3da-4bc8-8ab5-0d932f6bfa08/authorino/0.log"
May 29 14:02:34.904290 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:34.904259    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-6cb54b5c86-zxjxr_d9ede80d-22ba-4e81-81c1-15b8b5631ab7/kuadrant-console-plugin/0.log"
May 29 14:02:36.663272 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:36.663242    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nvrh8_49ce8cda-457b-40e1-9f39-1820887e3970/node-exporter/0.log"
May 29 14:02:36.684535 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:36.684510    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nvrh8_49ce8cda-457b-40e1-9f39-1820887e3970/kube-rbac-proxy/0.log"
May 29 14:02:36.727807 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:36.727716    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nvrh8_49ce8cda-457b-40e1-9f39-1820887e3970/init-textfile/0.log"
May 29 14:02:40.143276 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:40.143238    2571 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mcwqr/perf-node-gather-daemonset-pkrqb"]
May 29 14:02:40.150718 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:40.150687    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mcwqr/perf-node-gather-daemonset-pkrqb"
May 29 14:02:40.159325 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:40.159295    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mcwqr/perf-node-gather-daemonset-pkrqb"]
May 29 14:02:40.227583 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:40.227547    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc-sys\") pod \"perf-node-gather-daemonset-pkrqb\" (UID: \"2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc\") " pod="openshift-must-gather-mcwqr/perf-node-gather-daemonset-pkrqb"
May 29 14:02:40.227583 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:40.227587    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc-podres\") pod \"perf-node-gather-daemonset-pkrqb\" (UID: \"2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc\") " pod="openshift-must-gather-mcwqr/perf-node-gather-daemonset-pkrqb"
May 29 14:02:40.227837 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:40.227772    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc-lib-modules\") pod \"perf-node-gather-daemonset-pkrqb\" (UID: \"2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc\") " pod="openshift-must-gather-mcwqr/perf-node-gather-daemonset-pkrqb"
May 29 14:02:40.227837 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:40.227817    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fhg6\" (UniqueName: \"kubernetes.io/projected/2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc-kube-api-access-2fhg6\") pod \"perf-node-gather-daemonset-pkrqb\" (UID: \"2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc\") " pod="openshift-must-gather-mcwqr/perf-node-gather-daemonset-pkrqb"
May 29 14:02:40.227940 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:40.227867    2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc-proc\") pod \"perf-node-gather-daemonset-pkrqb\" (UID: \"2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc\") " pod="openshift-must-gather-mcwqr/perf-node-gather-daemonset-pkrqb"
May 29 14:02:40.328879 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:40.328841    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc-sys\") pod \"perf-node-gather-daemonset-pkrqb\" (UID: \"2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc\") " pod="openshift-must-gather-mcwqr/perf-node-gather-daemonset-pkrqb"
May 29 14:02:40.329284 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:40.329267    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc-podres\") pod \"perf-node-gather-daemonset-pkrqb\" (UID: \"2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc\") " pod="openshift-must-gather-mcwqr/perf-node-gather-daemonset-pkrqb"
May 29 14:02:40.329638 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:40.329199    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc-sys\") pod \"perf-node-gather-daemonset-pkrqb\" (UID: \"2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc\") " pod="openshift-must-gather-mcwqr/perf-node-gather-daemonset-pkrqb"
May 29 14:02:40.329753 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:40.329528    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc-podres\") pod \"perf-node-gather-daemonset-pkrqb\" (UID: \"2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc\") " pod="openshift-must-gather-mcwqr/perf-node-gather-daemonset-pkrqb"
May 29 14:02:40.329961 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:40.329947    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc-lib-modules\") pod \"perf-node-gather-daemonset-pkrqb\" (UID: \"2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc\") " pod="openshift-must-gather-mcwqr/perf-node-gather-daemonset-pkrqb"
May 29 14:02:40.330596 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:40.330577    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2fhg6\" (UniqueName: \"kubernetes.io/projected/2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc-kube-api-access-2fhg6\") pod \"perf-node-gather-daemonset-pkrqb\" (UID: \"2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc\") " pod="openshift-must-gather-mcwqr/perf-node-gather-daemonset-pkrqb"
May 29 14:02:40.331337 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:40.330226    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc-lib-modules\") pod \"perf-node-gather-daemonset-pkrqb\" (UID: \"2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc\") " pod="openshift-must-gather-mcwqr/perf-node-gather-daemonset-pkrqb"
May 29 14:02:40.331490 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:40.331475    2571 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc-proc\") pod \"perf-node-gather-daemonset-pkrqb\" (UID: \"2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc\") " pod="openshift-must-gather-mcwqr/perf-node-gather-daemonset-pkrqb"
May 29 14:02:40.331727 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:40.331623    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc-proc\") pod \"perf-node-gather-daemonset-pkrqb\" (UID: \"2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc\") " pod="openshift-must-gather-mcwqr/perf-node-gather-daemonset-pkrqb"
May 29 14:02:40.341435 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:40.341384    2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fhg6\" (UniqueName: \"kubernetes.io/projected/2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc-kube-api-access-2fhg6\") pod \"perf-node-gather-daemonset-pkrqb\" (UID: \"2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc\") " pod="openshift-must-gather-mcwqr/perf-node-gather-daemonset-pkrqb"
May 29 14:02:40.468129 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:40.468033    2571 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mcwqr/perf-node-gather-daemonset-pkrqb"
May 29 14:02:40.644937 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:40.644912    2571 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mcwqr/perf-node-gather-daemonset-pkrqb"]
May 29 14:02:40.647966 ip-10-0-141-1 kubenswrapper[2571]: W0529 14:02:40.647934    2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2ec04a15_f0d3_42cb_8e92_699d1fbdb5dc.slice/crio-a3618c40514c9253b98beb9610ed8b2719afd12bba7deeecdfe7a94348bcaecb WatchSource:0}: Error finding container a3618c40514c9253b98beb9610ed8b2719afd12bba7deeecdfe7a94348bcaecb: Status 404 returned error can't find the container with id a3618c40514c9253b98beb9610ed8b2719afd12bba7deeecdfe7a94348bcaecb
May 29 14:02:41.009470 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:41.009389    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mcwqr/perf-node-gather-daemonset-pkrqb" event={"ID":"2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc","Type":"ContainerStarted","Data":"cb709ac5f2e56520d29b4fb8ce7271a31564aa1cafc72cf185b9f67433dacb4b"}
May 29 14:02:41.009470 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:41.009428    2571 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mcwqr/perf-node-gather-daemonset-pkrqb" event={"ID":"2ec04a15-f0d3-42cb-8e92-699d1fbdb5dc","Type":"ContainerStarted","Data":"a3618c40514c9253b98beb9610ed8b2719afd12bba7deeecdfe7a94348bcaecb"}
May 29 14:02:41.009688 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:41.009526    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mcwqr/perf-node-gather-daemonset-pkrqb"
May 29 14:02:41.026745 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:41.026682    2571 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-mcwqr/perf-node-gather-daemonset-pkrqb" podStartSLOduration=1.026665082 podStartE2EDuration="1.026665082s" podCreationTimestamp="2026-05-29 14:02:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 14:02:41.02626748 +0000 UTC m=+1235.493337573" watchObservedRunningTime="2026-05-29 14:02:41.026665082 +0000 UTC m=+1235.493735165"
May 29 14:02:41.590600 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:41.590568    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7csjl_05bfd411-03f2-462d-bc97-568b19e3626d/dns/0.log"
May 29 14:02:41.608747 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:41.608716    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7csjl_05bfd411-03f2-462d-bc97-568b19e3626d/kube-rbac-proxy/0.log"
May 29 14:02:41.724993 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:41.724965    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xqvkm_5ded8d51-ee6a-492b-a1d4-5afefbf29dec/dns-node-resolver/0.log"
May 29 14:02:42.210894 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:42.210852    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9wzn2_af79b7f2-4958-48fc-ad8d-ef0c731dc642/node-ca/0.log"
May 29 14:02:43.203491 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:43.203457    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-855f784d5b-5fqkq_deae1b19-73c8-40fc-a0cc-e7663d86c726/kube-auth-proxy/0.log"
May 29 14:02:43.853833 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:43.853810    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dn82q_29951f01-4535-401e-b698-8096a4fbc290/serve-healthcheck-canary/0.log"
May 29 14:02:44.544700 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:44.544672    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xw9s8_88909f40-15e8-4baa-9365-c504b94066c4/kube-rbac-proxy/0.log"
May 29 14:02:44.566009 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:44.565982    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xw9s8_88909f40-15e8-4baa-9365-c504b94066c4/exporter/0.log"
May 29 14:02:44.587802 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:44.587775    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xw9s8_88909f40-15e8-4baa-9365-c504b94066c4/extractor/0.log"
May 29 14:02:46.653741 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:46.653708    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-69c897cd9b-pwrhr_ac32703e-80be-45e3-864f-6633814559b8/maas-api/0.log"
May 29 14:02:46.705993 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:46.705952    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-8db997c58-nhpk4_65b430b8-f603-42c1-bc73-89cb54abb5a1/manager/0.log"
May 29 14:02:46.722823 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:46.722801    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-6577596cd5-dfvjf_ec7a7005-5d22-4311-87b9-2faf1e135e15/server/0.log"
May 29 14:02:46.768658 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:46.768618    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-76588b95-g42dw_59222dfa-c8a8-4d60-91eb-467a6c42f78a/manager/0.log"
May 29 14:02:46.880477 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:46.880450    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cf69667cd-pglk6_326d47a7-e9cf-4834-9b3e-11812596c49e/manager/0.log"
May 29 14:02:46.901833 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:46.901801    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cf69667cd-pglk6_326d47a7-e9cf-4834-9b3e-11812596c49e/copy-manifests/0.log"
May 29 14:02:47.025889 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:47.025808    2571 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mcwqr/perf-node-gather-daemonset-pkrqb"
May 29 14:02:48.469750 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:48.469711    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-699cb996b8-lc86g_cb007466-d172-40b4-b1e8-087723079292/manager/0.log"
May 29 14:02:55.056164 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:55.056132    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-46jjq_758a6d2f-ef64-4995-9a3b-d66b09cc98f1/kube-multus/0.log"
May 29 14:02:55.104325 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:55.104298    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4vtqj_1b2d1d27-e735-4a58-a93f-5d0bf642aa78/kube-multus-additional-cni-plugins/0.log"
May 29 14:02:55.124008 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:55.123982    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4vtqj_1b2d1d27-e735-4a58-a93f-5d0bf642aa78/egress-router-binary-copy/0.log"
May 29 14:02:55.147864 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:55.147838    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4vtqj_1b2d1d27-e735-4a58-a93f-5d0bf642aa78/cni-plugins/0.log"
May 29 14:02:55.168410 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:55.168387    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4vtqj_1b2d1d27-e735-4a58-a93f-5d0bf642aa78/bond-cni-plugin/0.log"
May 29 14:02:55.187972 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:55.187948    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4vtqj_1b2d1d27-e735-4a58-a93f-5d0bf642aa78/routeoverride-cni/0.log"
May 29 14:02:55.210320 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:55.210297    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4vtqj_1b2d1d27-e735-4a58-a93f-5d0bf642aa78/whereabouts-cni-bincopy/0.log"
May 29 14:02:55.231827 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:55.231804    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4vtqj_1b2d1d27-e735-4a58-a93f-5d0bf642aa78/whereabouts-cni/0.log"
May 29 14:02:55.662020 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:55.661938    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-n48xk_2a114752-df85-41bc-b828-17308b5311de/network-metrics-daemon/0.log"
May 29 14:02:55.680644 ip-10-0-141-1 kubenswrapper[2571]: I0529 14:02:55.680618    2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-n48xk_2a114752-df85-41bc-b828-17308b5311de/kube-rbac-proxy/0.log"