Jul 06 19:05:24.694889 ip-10-0-129-80 systemd[1]: Starting Kubernetes Kubelet... Jul 06 19:05:25.101361 ip-10-0-129-80 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:05:25.101361 ip-10-0-129-80 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 19:05:25.101361 ip-10-0-129-80 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:05:25.101361 ip-10-0-129-80 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 19:05:25.101361 ip-10-0-129-80 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 19:05:25.102957 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.102865 2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 19:05:25.109460 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109433 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:05:25.109460 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109457 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:05:25.109460 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109461 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:05:25.109460 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109465 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:05:25.109460 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109468 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:05:25.109688 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109471 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:05:25.109688 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109474 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:05:25.109688 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109477 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:05:25.109688 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109480 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:05:25.109688 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109483 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:05:25.109688 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109486 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:05:25.109688 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109489 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:05:25.109688 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109492 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:05:25.109688 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109495 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:05:25.109688 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109498 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:05:25.109688 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109501 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:05:25.109688 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109504 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:05:25.109688 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109507 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:05:25.109688 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109509 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:05:25.109688 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109512 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:05:25.109688 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109515 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:05:25.109688 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109518 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:05:25.109688 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109521 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:05:25.109688 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109524 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:05:25.110180 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109528 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:05:25.110180 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109533 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:05:25.110180 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109537 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:05:25.110180 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109540 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:05:25.110180 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109543 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:05:25.110180 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109547 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:05:25.110180 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109549 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:05:25.110180 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109552 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:05:25.110180 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109562 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:05:25.110180 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109565 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:05:25.110180 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109567 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:05:25.110180 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109570 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:05:25.110180 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109573 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:05:25.110180 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109576 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:05:25.110180 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109579 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:05:25.110180 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109582 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:05:25.110180 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109585 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:05:25.110180 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109587 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:05:25.110180 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109590 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:05:25.110180 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109593 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:05:25.110862 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109595 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:05:25.110862 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109598 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:05:25.110862 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109601 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:05:25.110862 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109604 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:05:25.110862 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109606 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:05:25.110862 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109609 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:05:25.110862 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109612 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:05:25.110862 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109616 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:05:25.110862 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109621 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:05:25.110862 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109625 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:05:25.110862 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109628 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:05:25.110862 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109631 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:05:25.110862 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109633 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:05:25.110862 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109636 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:05:25.110862 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109639 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:05:25.110862 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109657 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:05:25.110862 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109660 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:05:25.110862 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109665 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:05:25.110862 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109668 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:05:25.111355 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109671 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:05:25.111355 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109673 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:05:25.111355 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109676 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:05:25.111355 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109679 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:05:25.111355 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109682 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:05:25.111355 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109685 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:05:25.111355 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109688 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:05:25.111355 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109692 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:05:25.111355 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109695 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:05:25.111355 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109698 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:05:25.111355 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109701 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:05:25.111355 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109704 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:05:25.111355 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109707 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:05:25.111355 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109710 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:05:25.111355 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109713 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:05:25.111355 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109717 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:05:25.111355 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109720 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:05:25.111355 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109722 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:05:25.111355 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109725 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:05:25.111355 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109729 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:05:25.111355 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109733 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:05:25.111902 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109735 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:05:25.111902 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109738 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:05:25.111902 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109741 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:05:25.111902 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109744 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:05:25.111902 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109753 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:05:25.111902 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109757 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:05:25.111902 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109761 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:05:25.111902 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109764 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:05:25.111902 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109767 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:05:25.111902 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109770 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:05:25.111902 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109773 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:05:25.111902 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109777 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:05:25.111902 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109780 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:05:25.111902 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109783 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:05:25.111902 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.109786 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:05:25.111902 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111453 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:05:25.111902 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111461 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:05:25.111902 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111465 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:05:25.112360 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111468 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:05:25.112360 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111471 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:05:25.112360 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111474 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:05:25.112360 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111477 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:05:25.112360 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111480 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:05:25.112360 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111483 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:05:25.112360 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111486 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:05:25.112360 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111488 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:05:25.112360 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111491 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:05:25.112360 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111494 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:05:25.112360 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111496 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:05:25.112360 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111499 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:05:25.112360 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111503 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:05:25.112360 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111505 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:05:25.112360 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111509 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:05:25.112360 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111512 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:05:25.112360 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111515 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:05:25.112360 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111518 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:05:25.112360 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111521 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:05:25.112360 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111524 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:05:25.112360 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111527 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:05:25.112912 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111530 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:05:25.112912 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111532 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:05:25.112912 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111535 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:05:25.112912 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111538 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:05:25.112912 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111541 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:05:25.112912 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111544 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:05:25.112912 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111547 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:05:25.112912 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111549 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:05:25.112912 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111552 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:05:25.112912 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111555 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:05:25.112912 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111558 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:05:25.112912 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111560 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:05:25.112912 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111563 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:05:25.112912 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111566 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:05:25.112912 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111568 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:05:25.112912 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111571 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:05:25.112912 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111574 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:05:25.112912 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111577 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:05:25.112912 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111580 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:05:25.112912 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111582 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:05:25.113465 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111586 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:05:25.113465 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111589 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:05:25.113465 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111593 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:05:25.113465 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111597 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:05:25.113465 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111600 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:05:25.113465 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111603 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:05:25.113465 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111607 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:05:25.113465 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111610 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:05:25.113465 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111612 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:05:25.113465 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111615 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:05:25.113465 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111618 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:05:25.113465 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111636 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:05:25.113465 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111639 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:05:25.113465 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111642 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:05:25.113465 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111646 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:05:25.113465 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111649 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:05:25.113465 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111652 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:05:25.113465 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111655 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:05:25.113465 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111659 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:05:25.113974 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111663 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:05:25.113974 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111666 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:05:25.113974 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111670 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:05:25.113974 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111673 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:05:25.113974 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111676 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:05:25.113974 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111679 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:05:25.113974 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111681 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:05:25.113974 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111684 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:05:25.113974 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111687 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:05:25.113974 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111690 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:05:25.113974 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111693 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:05:25.113974 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111697 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:05:25.113974 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111702 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:05:25.113974 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111705 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:05:25.113974 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111708 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:05:25.113974 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111710 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:05:25.113974 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111713 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:05:25.113974 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111716 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:05:25.113974 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111719 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:05:25.114457 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111722 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:05:25.114457 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111726 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:05:25.114457 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111729 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:05:25.114457 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111732 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:05:25.114457 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111735 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:05:25.114457 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111737 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:05:25.114457 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111740 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:05:25.114457 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111742 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:05:25.114457 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111746 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:05:25.114457 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111749 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:05:25.114457 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111752 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:05:25.114457 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111755 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:05:25.114457 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111757 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:05:25.114457 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111760 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:05:25.114457 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111763 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:05:25.114457 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111766 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:05:25.114457 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.111769 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:05:25.114457 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112197 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:05:25.114457 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112202 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:05:25.114457 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112205 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:05:25.114983 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112208 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:05:25.114983 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112211 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:05:25.114983 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112214 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:05:25.114983 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112217 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:05:25.114983 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112220 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:05:25.114983 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112223 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:05:25.114983 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112226 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:05:25.114983 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112229 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:05:25.114983 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112234 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:05:25.114983 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112237 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:05:25.114983 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112240 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:05:25.114983 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112243 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:05:25.114983 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112245 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:05:25.114983 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112249 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:05:25.114983 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112251 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:05:25.114983 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112254 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:05:25.114983 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112273 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:05:25.114983 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112278 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:05:25.114983 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112282 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:05:25.115477 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112286 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:05:25.115477 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112290 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:05:25.115477 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112294 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:05:25.115477 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112297 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:05:25.115477 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112300 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:05:25.115477 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112303 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:05:25.115477 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112306 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:05:25.115477 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112309 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:05:25.115477 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112312 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:05:25.115477 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112314 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:05:25.115477 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112317 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:05:25.115477 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112320 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:05:25.115477 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112323 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:05:25.115477 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112325 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:05:25.115477 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112328 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:05:25.115477 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112331 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:05:25.115477 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112334 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:05:25.115477 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112337 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:05:25.115477 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112340 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:05:25.115982 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112343 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:05:25.115982 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112345 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:05:25.115982 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112348 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:05:25.115982 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112351 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:05:25.115982 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112354 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:05:25.115982 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112357 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:05:25.115982 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112359 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:05:25.115982 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112363 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:05:25.115982 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112367 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:05:25.115982 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112370 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:05:25.115982 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112373 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:05:25.115982 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112376 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:05:25.115982 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112379 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:05:25.115982 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112381 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:05:25.115982 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112384 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:05:25.115982 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112387 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:05:25.115982 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112391 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:05:25.115982 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112394 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:05:25.115982 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112396 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:05:25.116471 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112399 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:05:25.116471 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112402 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:05:25.116471 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112404 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:05:25.116471 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112407 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:05:25.116471 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112410 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:05:25.116471 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112413 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:05:25.116471 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112417 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:05:25.116471 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112421 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:05:25.116471 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112424 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:05:25.116471 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112427 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:05:25.116471 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112430 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:05:25.116471 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112436 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:05:25.116471 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112439 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:05:25.116471 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112442 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:05:25.116471 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112446 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:05:25.116471 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112449 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:05:25.116471 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112452 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:05:25.116471 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112455 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:05:25.116471 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112458 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:05:25.116471 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112461 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:05:25.116999 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112463 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:05:25.116999 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112466 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:05:25.116999 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112469 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:05:25.116999 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112472 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:05:25.116999 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112474 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:05:25.116999 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112477 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:05:25.116999 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112480 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:05:25.116999 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112482 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:05:25.116999 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112485 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:05:25.116999 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112488 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:05:25.116999 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112490 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:05:25.116999 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112493 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:05:25.116999 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112495 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:05:25.116999 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112498 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:05:25.116999 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112501 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:05:25.116999 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112503 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:05:25.116999 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112506 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:05:25.116999 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112508 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:05:25.116999 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.112511 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:05:25.116999 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112582 2560 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 19:05:25.116999 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112590 2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 19:05:25.117551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112596 2560 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 19:05:25.117551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112603 2560 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 19:05:25.117551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112613 2560 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 19:05:25.117551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112618 2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 19:05:25.117551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112625 2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 19:05:25.117551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112632 2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 19:05:25.117551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112637 2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 19:05:25.117551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112640 2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 19:05:25.117551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112644 2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 19:05:25.117551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112663 2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 19:05:25.117551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112667 2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 19:05:25.117551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112671 2560 flags.go:64] FLAG: --cgroup-root="" Jul 06 19:05:25.117551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112675 2560 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 19:05:25.117551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112678 2560 flags.go:64] FLAG: --client-ca-file="" Jul 06 19:05:25.117551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112683 2560 flags.go:64] FLAG: --cloud-provider="external" Jul 06 19:05:25.117551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112686 2560 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 19:05:25.117551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112691 2560 flags.go:64] FLAG: --cluster-domain="" Jul 06 19:05:25.117551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112695 2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 19:05:25.117551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112698 2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 19:05:25.117551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112701 2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 19:05:25.117551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112705 2560 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 19:05:25.117551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112709 2560 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 19:05:25.117551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112712 2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 19:05:25.118141 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112716 2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 19:05:25.118141 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112719 2560 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 19:05:25.118141 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112722 2560 flags.go:64] FLAG: --contention-profiling="false" Jul 06 19:05:25.118141 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112726 2560 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 19:05:25.118141 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112729 2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 19:05:25.118141 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112733 2560 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 19:05:25.118141 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112736 2560 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 19:05:25.118141 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112740 2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 19:05:25.118141 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112743 2560 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 19:05:25.118141 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112746 2560 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 19:05:25.118141 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112750 2560 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 19:05:25.118141 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112754 2560 flags.go:64] FLAG: --enable-server="true" Jul 06 19:05:25.118141 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112758 2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 19:05:25.118141 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112762 2560 flags.go:64] FLAG: --event-burst="100" Jul 06 19:05:25.118141 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112766 2560 flags.go:64] FLAG: --event-qps="50" Jul 06 19:05:25.118141 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112769 2560 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 19:05:25.118141 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112773 2560 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 19:05:25.118141 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112776 2560 flags.go:64] FLAG: --eviction-hard="" Jul 06 19:05:25.118141 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112780 2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 19:05:25.118141 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112783 2560 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 19:05:25.118141 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112786 2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 19:05:25.118141 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112789 2560 flags.go:64] FLAG: --eviction-soft="" Jul 06 19:05:25.118141 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112793 2560 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 19:05:25.118141 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112796 2560 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 19:05:25.118141 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112800 2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 19:05:25.118861 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112803 2560 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 19:05:25.118861 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112806 2560 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 19:05:25.118861 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112809 2560 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 19:05:25.118861 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112812 2560 flags.go:64] FLAG: --feature-gates="" Jul 06 19:05:25.118861 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112816 2560 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 19:05:25.118861 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112820 2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 19:05:25.118861 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112823 2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 19:05:25.118861 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112826 2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 19:05:25.118861 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112829 2560 flags.go:64] FLAG: --healthz-port="10248" Jul 06 19:05:25.118861 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112833 2560 flags.go:64] FLAG: --help="false" Jul 06 19:05:25.118861 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112836 2560 flags.go:64] FLAG: --hostname-override="ip-10-0-129-80.ec2.internal" Jul 06 19:05:25.118861 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112840 2560 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 19:05:25.118861 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112843 2560 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 19:05:25.118861 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112846 2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 19:05:25.118861 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112849 2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 19:05:25.118861 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112853 2560 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 19:05:25.118861 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112856 2560 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 19:05:25.118861 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112859 2560 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 19:05:25.118861 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112864 2560 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 19:05:25.118861 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112867 2560 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 19:05:25.118861 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112870 2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 19:05:25.118861 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112873 2560 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 19:05:25.118861 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112876 2560 flags.go:64] FLAG: --kube-reserved="" Jul 06 19:05:25.118861 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112879 2560 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 19:05:25.119487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112882 2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 19:05:25.119487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112885 2560 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 19:05:25.119487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112888 2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 19:05:25.119487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112891 2560 flags.go:64] FLAG: --lock-file="" Jul 06 19:05:25.119487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112894 2560 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 19:05:25.119487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112897 2560 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 19:05:25.119487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112900 2560 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 19:05:25.119487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112906 2560 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 19:05:25.119487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112910 2560 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 19:05:25.119487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112913 2560 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 19:05:25.119487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112916 2560 flags.go:64] FLAG: --logging-format="text" Jul 06 19:05:25.119487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112919 2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 19:05:25.119487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112923 2560 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 19:05:25.119487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112926 2560 flags.go:64] FLAG: --manifest-url="" Jul 06 19:05:25.119487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112929 2560 flags.go:64] FLAG: --manifest-url-header="" Jul 06 19:05:25.119487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112934 2560 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 19:05:25.119487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112939 2560 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 19:05:25.119487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112943 2560 flags.go:64] FLAG: --max-pods="110" Jul 06 19:05:25.119487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112946 2560 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 19:05:25.119487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112950 2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 19:05:25.119487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112953 2560 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 19:05:25.119487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112956 2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 19:05:25.119487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112959 2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 19:05:25.119487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112962 2560 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 19:05:25.119487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112965 2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 19:05:25.120624 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112973 2560 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 19:05:25.120624 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112977 2560 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 19:05:25.120624 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112980 2560 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 19:05:25.120624 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112984 2560 flags.go:64] FLAG: --pod-cidr="" Jul 06 19:05:25.120624 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112986 2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 19:05:25.120624 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112992 2560 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 19:05:25.120624 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112995 2560 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 19:05:25.120624 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.112998 2560 flags.go:64] FLAG: --pods-per-core="0" Jul 06 19:05:25.120624 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113001 2560 flags.go:64] FLAG: --port="10250" Jul 06 19:05:25.120624 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113005 2560 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 19:05:25.120624 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113008 2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0da2f1d3ede24abd9" Jul 06 19:05:25.120624 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113011 2560 flags.go:64] FLAG: --qos-reserved="" Jul 06 19:05:25.120624 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113014 2560 flags.go:64] FLAG: --read-only-port="10255" Jul 06 19:05:25.120624 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113017 2560 flags.go:64] FLAG: --register-node="true" Jul 06 19:05:25.120624 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113020 2560 flags.go:64] FLAG: --register-with-taints="" Jul 06 19:05:25.120624 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113024 2560 flags.go:64] FLAG: --registry-burst="10" Jul 06 19:05:25.120624 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113028 2560 flags.go:64] FLAG: --registry-qps="5" Jul 06 19:05:25.120624 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113031 2560 flags.go:64] FLAG: --reserved-cpus="" Jul 06 19:05:25.120624 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113034 2560 flags.go:64] FLAG: --reserved-memory="" Jul 06 19:05:25.120624 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113038 2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 19:05:25.120624 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113041 2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 19:05:25.120624 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113044 2560 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 19:05:25.120624 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113047 2560 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 19:05:25.120624 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113050 2560 flags.go:64] FLAG: --runonce="false" Jul 06 19:05:25.120624 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113055 2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 19:05:25.121810 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113059 2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 19:05:25.121810 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113062 2560 flags.go:64] FLAG: --seccomp-default="false" Jul 06 19:05:25.121810 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113065 2560 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 19:05:25.121810 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113068 2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 19:05:25.121810 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113071 2560 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 19:05:25.121810 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113074 2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 19:05:25.121810 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113078 2560 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 19:05:25.121810 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113081 2560 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 19:05:25.121810 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113084 2560 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 19:05:25.121810 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113087 2560 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 19:05:25.121810 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113090 2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 19:05:25.121810 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113094 2560 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 19:05:25.121810 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113097 2560 flags.go:64] FLAG: --system-cgroups="" Jul 06 19:05:25.121810 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113100 2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 19:05:25.121810 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113105 2560 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 19:05:25.121810 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113108 2560 flags.go:64] FLAG: --tls-cert-file="" Jul 06 19:05:25.121810 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113112 2560 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 19:05:25.121810 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113117 2560 flags.go:64] FLAG: --tls-min-version="" Jul 06 19:05:25.121810 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113120 2560 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 19:05:25.121810 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113123 2560 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 19:05:25.121810 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113126 2560 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 19:05:25.121810 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113129 2560 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 19:05:25.121810 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113132 2560 flags.go:64] FLAG: --v="2" Jul 06 19:05:25.121810 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113136 2560 flags.go:64] FLAG: --version="false" Jul 06 19:05:25.121810 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113141 2560 flags.go:64] FLAG: --vmodule="" Jul 06 19:05:25.122548 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113145 2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 19:05:25.122548 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113148 2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 19:05:25.122548 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113275 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:05:25.122548 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113279 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:05:25.122548 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113283 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:05:25.122548 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113287 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:05:25.122548 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113292 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:05:25.122548 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113295 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:05:25.122548 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113297 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:05:25.122548 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113301 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:05:25.122548 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113306 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:05:25.122548 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113309 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:05:25.122548 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113313 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:05:25.122548 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113315 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:05:25.122548 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113319 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:05:25.122548 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113322 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:05:25.122548 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113325 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:05:25.122548 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113328 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:05:25.122548 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113331 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:05:25.122548 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113333 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:05:25.123492 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113336 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:05:25.123492 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113339 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:05:25.123492 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113342 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:05:25.123492 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113345 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:05:25.123492 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113347 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:05:25.123492 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113350 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:05:25.123492 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113353 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:05:25.123492 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113356 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:05:25.123492 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113358 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:05:25.123492 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113361 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:05:25.123492 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113364 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:05:25.123492 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113366 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:05:25.123492 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113371 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:05:25.123492 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113375 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:05:25.123492 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113378 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:05:25.123492 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113381 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:05:25.123492 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113384 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:05:25.123492 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113386 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:05:25.123492 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113391 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:05:25.123492 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113394 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:05:25.124459 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113396 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:05:25.124459 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113399 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:05:25.124459 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113402 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:05:25.124459 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113406 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:05:25.124459 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113410 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:05:25.124459 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113413 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:05:25.124459 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113417 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:05:25.124459 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113420 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:05:25.124459 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113424 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:05:25.124459 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113427 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:05:25.124459 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113430 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:05:25.124459 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113433 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:05:25.124459 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113436 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:05:25.124459 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113439 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:05:25.124459 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113441 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:05:25.124459 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113444 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:05:25.124459 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113447 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:05:25.124459 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113450 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:05:25.124459 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113452 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:05:25.125044 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113455 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:05:25.125044 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113458 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:05:25.125044 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113460 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:05:25.125044 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113463 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:05:25.125044 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113466 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:05:25.125044 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113468 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:05:25.125044 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113472 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:05:25.125044 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113475 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:05:25.125044 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113478 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:05:25.125044 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113481 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:05:25.125044 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113483 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:05:25.125044 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113490 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:05:25.125044 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113493 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:05:25.125044 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113496 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:05:25.125044 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113498 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:05:25.125044 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113501 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:05:25.125044 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113504 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:05:25.125044 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113506 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:05:25.125044 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113509 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:05:25.125868 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113512 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:05:25.125868 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113515 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:05:25.125868 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113518 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:05:25.125868 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113520 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:05:25.125868 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113523 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:05:25.125868 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113526 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:05:25.125868 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113529 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:05:25.125868 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113531 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:05:25.125868 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113534 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:05:25.125868 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113537 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:05:25.125868 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113540 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:05:25.125868 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113543 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:05:25.125868 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113545 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:05:25.125868 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113548 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:05:25.125868 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113550 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:05:25.125868 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113553 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:05:25.125868 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113556 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:05:25.125868 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113559 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:05:25.125868 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113562 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:05:25.125868 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113565 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:05:25.126833 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113568 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:05:25.126833 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113571 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:05:25.126833 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.113573 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:05:25.126833 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.113579 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:05:25.126833 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.121961 2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 19:05:25.126833 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.121988 2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 19:05:25.126833 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.122786 2560 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 19:05:25.126833 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.122804 2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 19:05:25.126833 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.122935 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:05:25.126833 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.122944 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:05:25.126833 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.122950 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:05:25.126833 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.122955 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:05:25.126833 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.122960 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:05:25.126833 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.122965 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:05:25.126833 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.122971 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:05:25.127277 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.122976 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:05:25.127277 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.122982 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:05:25.127277 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.122987 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:05:25.127277 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.122991 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:05:25.127277 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.122996 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:05:25.127277 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123001 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:05:25.127277 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123005 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:05:25.127277 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123010 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:05:25.127277 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123015 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:05:25.127277 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123019 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:05:25.127277 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123024 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:05:25.127277 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123028 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:05:25.127277 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123033 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:05:25.127277 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123037 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:05:25.127277 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123042 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:05:25.127277 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123046 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:05:25.127277 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123052 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:05:25.127277 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123057 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:05:25.127277 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123062 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:05:25.127277 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123066 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:05:25.127938 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123071 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:05:25.127938 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123076 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:05:25.127938 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123082 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:05:25.127938 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123086 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:05:25.127938 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123091 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:05:25.127938 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123096 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:05:25.127938 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123103 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:05:25.127938 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123110 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:05:25.127938 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123115 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:05:25.127938 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123120 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:05:25.127938 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123127 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:05:25.127938 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123132 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:05:25.127938 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123137 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:05:25.127938 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123141 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:05:25.127938 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123146 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:05:25.127938 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123151 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:05:25.127938 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123155 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:05:25.127938 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123160 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:05:25.127938 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123164 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:05:25.128487 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123168 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:05:25.128487 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123174 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:05:25.128487 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123178 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:05:25.128487 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123183 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:05:25.128487 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123188 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:05:25.128487 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123193 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:05:25.128487 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123197 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:05:25.128487 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123202 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:05:25.128487 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123207 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:05:25.128487 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123211 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:05:25.128487 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123216 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:05:25.128487 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123220 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:05:25.128487 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123225 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:05:25.128487 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123230 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:05:25.128487 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123235 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:05:25.128487 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123239 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:05:25.128487 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123244 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:05:25.128487 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123249 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:05:25.128487 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123254 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:05:25.128487 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123258 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:05:25.129086 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123262 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:05:25.129086 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123267 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:05:25.129086 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123271 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:05:25.129086 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123276 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:05:25.129086 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123283 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:05:25.129086 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123288 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:05:25.129086 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123293 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:05:25.129086 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123299 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:05:25.129086 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123304 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:05:25.129086 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123309 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:05:25.129086 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123314 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:05:25.129086 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123318 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:05:25.129086 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123322 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:05:25.129086 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123327 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:05:25.129086 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123332 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:05:25.129086 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123336 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:05:25.129086 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123340 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:05:25.129086 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123345 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:05:25.129086 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123349 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:05:25.129584 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123353 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:05:25.129584 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123358 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:05:25.129584 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123363 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:05:25.129584 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123367 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:05:25.129584 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123372 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:05:25.129584 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123376 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:05:25.129584 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123381 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:05:25.129584 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123385 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:05:25.129584 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123390 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:05:25.129584 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123395 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:05:25.129584 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123400 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:05:25.129584 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123405 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:05:25.129584 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123409 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:05:25.129584 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123414 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:05:25.129584 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.123423 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:05:25.130080 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123617 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 19:05:25.130080 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123626 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 19:05:25.130080 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123631 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 19:05:25.130080 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123654 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 19:05:25.130080 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123660 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 19:05:25.130080 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123678 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 19:05:25.130080 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123683 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 19:05:25.130080 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123688 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 19:05:25.130080 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123693 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 19:05:25.130080 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123697 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 19:05:25.130080 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123702 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 19:05:25.130080 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123707 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 19:05:25.130080 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123712 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 19:05:25.130080 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123717 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 19:05:25.130080 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123721 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 19:05:25.130080 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123727 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 19:05:25.130080 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123734 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 19:05:25.130080 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123739 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 19:05:25.130080 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123744 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 19:05:25.130625 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123749 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 19:05:25.130625 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123754 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 19:05:25.130625 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123758 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 19:05:25.130625 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123763 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 19:05:25.130625 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123767 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 19:05:25.130625 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123772 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 19:05:25.130625 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123777 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 19:05:25.130625 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123781 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 06 19:05:25.130625 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123785 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 19:05:25.130625 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123791 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 19:05:25.130625 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123796 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 19:05:25.130625 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123801 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 19:05:25.130625 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123805 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 19:05:25.130625 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123810 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 19:05:25.130625 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123814 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 19:05:25.130625 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123821 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 19:05:25.130625 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123827 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 19:05:25.130625 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123833 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 19:05:25.130625 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123837 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 19:05:25.130625 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123843 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 19:05:25.131194 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123847 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 19:05:25.131194 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123852 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 19:05:25.131194 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123856 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 19:05:25.131194 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123861 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 19:05:25.131194 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123865 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 19:05:25.131194 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123870 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 19:05:25.131194 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123874 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 19:05:25.131194 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123879 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 19:05:25.131194 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123884 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 19:05:25.131194 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123888 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 19:05:25.131194 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123893 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 19:05:25.131194 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123897 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 19:05:25.131194 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123901 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 19:05:25.131194 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123906 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 19:05:25.131194 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123911 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 19:05:25.131194 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123916 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 19:05:25.131194 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123920 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 19:05:25.131194 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123925 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 19:05:25.131194 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123929 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 19:05:25.131194 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123934 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 19:05:25.132116 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123938 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 19:05:25.132116 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123943 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 19:05:25.132116 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123949 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 19:05:25.132116 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123954 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 19:05:25.132116 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123959 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 19:05:25.132116 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123963 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 19:05:25.132116 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123968 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 19:05:25.132116 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123973 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 19:05:25.132116 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123978 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 19:05:25.132116 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123985 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 19:05:25.132116 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123990 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 19:05:25.132116 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.123996 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 19:05:25.132116 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124002 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 19:05:25.132116 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124007 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 19:05:25.132116 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124012 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 19:05:25.132116 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124017 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 19:05:25.132116 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124021 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 19:05:25.132116 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124025 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 19:05:25.132116 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124030 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 19:05:25.132116 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124035 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 19:05:25.132985 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124039 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 19:05:25.132985 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124044 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 19:05:25.132985 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124049 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 19:05:25.132985 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124053 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 19:05:25.132985 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124057 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 19:05:25.132985 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124062 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 19:05:25.132985 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124067 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 19:05:25.132985 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124071 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 19:05:25.132985 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124075 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 19:05:25.132985 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124080 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 19:05:25.132985 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124084 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 19:05:25.132985 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124089 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 19:05:25.132985 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124094 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 19:05:25.132985 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124099 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 19:05:25.132985 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124104 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 19:05:25.132985 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124109 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 19:05:25.132985 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124114 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 19:05:25.132985 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124119 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 19:05:25.132985 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124123 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 19:05:25.132985 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:25.124128 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 19:05:25.133830 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.124136 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 19:05:25.133830 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.124346 2560 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 19:05:25.133830 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.127318 2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 19:05:25.133830 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.128829 2560 server.go:1019] "Starting client certificate rotation" Jul 06 19:05:25.133830 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.128939 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:05:25.133830 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.128987 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 19:05:25.133830 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.131869 2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:05:25.138888 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.138861 2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 19:05:25.151671 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.151654 2560 log.go:25] "Validated CRI v1 runtime API" Jul 06 19:05:25.153828 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.153812 2560 log.go:25] "Validated CRI v1 image API" Jul 06 19:05:25.155248 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.155228 2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 19:05:25.159943 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.159924 2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:05:25.161799 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.161772 2560 fs.go:135] Filesystem UUIDs: map[2f33d712-9cb5-4b3f-a482-d02e712666f8:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f336a267-6417-49e9-a592-27739b2a5576:/dev/nvme0n1p4] Jul 06 19:05:25.161888 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.161800 2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 19:05:25.167287 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.167167 2560 manager.go:216] Machine: {Timestamp:2026-07-06 19:05:25.165920803 +0000 UTC m=+0.357288491 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100367 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ecf9824728680a9cf0d64cb405343 SystemUUID:ec2ecf98-2472-8680-a9cf-0d64cb405343 BootID:3745efec-b719-4c9a-9a52-55505d71852f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4e:eb:11:0e:b7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4e:eb:11:0e:b7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:72:fd:04:e1:6b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 19:05:25.167287 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.167281 2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 19:05:25.167398 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.167367 2560 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 19:05:25.170419 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.170388 2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 19:05:25.170579 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.170420 2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-80.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 19:05:25.171192 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.171181 2560 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 19:05:25.171232 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.171195 2560 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 19:05:25.171232 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.171203 2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:05:25.171869 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.171858 2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 19:05:25.173231 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.173220 2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 19:05:25.176390 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.176378 2560 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:05:25.176505 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.176495 2560 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 19:05:25.176536 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.176529 2560 kubelet.go:486] "Attempting to sync node with API server" Jul 06 19:05:25.176565 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.176540 2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 19:05:25.176565 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.176552 2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 19:05:25.176565 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.176562 2560 kubelet.go:398] "Adding apiserver pod source" Jul 06 19:05:25.176693 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.176570 2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 19:05:25.177025 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.177011 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:05:25.177108 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.177033 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 19:05:25.179186 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.179170 2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 19:05:25.181326 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.181312 2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 19:05:25.181382 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.181345 2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 19:05:25.182620 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.182609 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 19:05:25.182673 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.182626 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 19:05:25.182673 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.182641 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 19:05:25.182673 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.182662 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 19:05:25.182673 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.182673 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 19:05:25.182793 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.182678 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 19:05:25.182793 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.182688 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 19:05:25.182793 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.182694 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 19:05:25.182793 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.182701 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 19:05:25.182793 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.182707 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 19:05:25.182793 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.182721 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 19:05:25.182793 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.182726 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 19:05:25.183686 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.183675 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 19:05:25.183728 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.183686 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 19:05:25.187213 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.187196 2560 server.go:1268] "Started kubelet" Jul 06 19:05:25.187350 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.187296 2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 19:05:25.187350 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.187333 2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 19:05:25.187469 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.187374 2560 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 19:05:25.189175 ip-10-0-129-80 systemd[1]: Started Kubernetes Kubelet. Jul 06 19:05:25.189323 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.189255 2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 19:05:25.192006 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.191985 2560 server.go:310] "Adding debug handlers to kubelet server" Jul 06 19:05:25.194041 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:25.194016 2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 19:05:25.195177 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.195159 2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-80.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 19:05:25.195891 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:25.195868 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 19:05:25.195942 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:25.195873 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-80.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 19:05:25.198331 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.198315 2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 19:05:25.198403 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.198331 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 19:05:25.198403 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:25.198383 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found" Jul 06 19:05:25.199397 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.199376 2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 19:05:25.199397 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.199397 2560 factory.go:55] Registering systemd factory Jul 06 19:05:25.199616 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.199408 2560 factory.go:223] Registration of the systemd container factory successfully Jul 06 19:05:25.199616 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.199497 2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 19:05:25.199616 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.199526 2560 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 19:05:25.199616 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.199537 2560 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 19:05:25.199833 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.199676 2560 reconstruct.go:97] "Volume reconstruction finished" Jul 06 19:05:25.199833 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.199689 2560 reconciler.go:29] "Reconciler: start to sync state" Jul 06 19:05:25.199925 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.199865 2560 factory.go:153] Registering CRI-O factory Jul 06 19:05:25.200075 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.200060 2560 factory.go:223] Registration of the crio container factory successfully Jul 06 19:05:25.200142 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.200097 2560 factory.go:103] Registering Raw factory Jul 06 19:05:25.200142 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.200114 2560 manager.go:1196] Started watching for new ooms in manager Jul 06 19:05:25.200707 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.200691 2560 manager.go:349] Starting recovery of all containers Jul 06 19:05:25.201292 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.201272 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-869k9" Jul 06 19:05:25.204265 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.204238 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-869k9" Jul 06 19:05:25.208195 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:25.208049 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 19:05:25.210230 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.210212 2560 manager.go:354] Recovery completed Jul 06 19:05:25.212035 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:25.212006 2560 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 06 19:05:25.215753 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.215738 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:05:25.218781 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:25.218756 2560 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-80.ec2.internal\" not found" node="ip-10-0-129-80.ec2.internal" Jul 06 19:05:25.220118 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.220095 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:05:25.220222 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.220136 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:05:25.220222 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.220154 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:05:25.221091 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.221076 2560 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 19:05:25.221169 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.221093 2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 19:05:25.221169 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.221106 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:05:25.221169 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.221116 2560 state_mem.go:36] "Initialized new in-memory state store" Jul 06 19:05:25.222199 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.222180 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:05:25.222270 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.222211 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:05:25.222270 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.222222 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:05:25.222692 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.222679 2560 policy_none.go:49] "None policy: Start" Jul 06 19:05:25.222759 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.222700 2560 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 19:05:25.222759 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.222716 2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 19:05:25.224159 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.224145 2560 policy_none.go:47] "Start" Jul 06 19:05:25.265300 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.265282 2560 manager.go:337] "Starting Device Plugin manager" Jul 06 19:05:25.265408 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:25.265328 2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 19:05:25.265408 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.265339 2560 server.go:87] "Starting device plugin registration server" Jul 06 19:05:25.265688 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.265672 2560 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 19:05:25.265760 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.265693 2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 19:05:25.265816 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.265771 2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 19:05:25.265876 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.265862 2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 19:05:25.265935 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.265877 2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 19:05:25.266484 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:25.266460 2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 19:05:25.266583 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:25.266521 2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-80.ec2.internal\" not found" Jul 06 19:05:25.368473 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.368440 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:05:25.369690 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.369591 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:05:25.369690 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.369619 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:05:25.369690 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.369630 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:05:25.369690 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.369657 2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-80.ec2.internal" Jul 06 19:05:25.379202 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.379184 2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-80.ec2.internal" Jul 06 19:05:25.379284 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:25.379207 2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-80.ec2.internal\": node \"ip-10-0-129-80.ec2.internal\" not found" Jul 06 19:05:25.383042 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.383013 2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 19:05:25.384384 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.384356 2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 19:05:25.384480 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.384396 2560 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 19:05:25.384480 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.384420 2560 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 19:05:25.384480 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:25.384454 2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 19:05:25.387224 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.387201 2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:05:25.394867 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:25.394844 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found" Jul 06 19:05:25.491246 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.491135 2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal"] Jul 06 19:05:25.491246 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.491239 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:05:25.493137 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.493121 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:05:25.493196 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.493155 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:05:25.493196 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.493165 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:05:25.494632 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.494618 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:05:25.494783 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.494767 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" Jul 06 19:05:25.494822 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.494801 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:05:25.494901 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:25.494886 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found" Jul 06 19:05:25.495380 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.495358 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:05:25.495453 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.495359 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:05:25.495453 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.495419 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:05:25.495453 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.495430 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:05:25.495545 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.495396 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:05:25.495545 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.495475 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:05:25.496993 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.496976 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal" Jul 06 19:05:25.497091 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.497034 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 19:05:25.497764 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.497750 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientMemory" Jul 06 19:05:25.497831 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.497776 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 19:05:25.497831 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.497791 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeHasSufficientPID" Jul 06 19:05:25.525047 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:25.525017 2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-80.ec2.internal\" not found" node="ip-10-0-129-80.ec2.internal" Jul 06 19:05:25.529971 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:25.529952 2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-80.ec2.internal\" not found" node="ip-10-0-129-80.ec2.internal" Jul 06 19:05:25.595088 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:25.595060 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found" Jul 06 19:05:25.600618 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.600594 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c8611695537a81ab018c1f7cb211fa60-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal\" (UID: \"c8611695537a81ab018c1f7cb211fa60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" Jul 06 19:05:25.600687 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.600628 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8611695537a81ab018c1f7cb211fa60-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal\" (UID: \"c8611695537a81ab018c1f7cb211fa60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" Jul 06 19:05:25.600687 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.600649 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/831a3fedbcc35899a7ac03738589a136-config\") pod \"kube-apiserver-proxy-ip-10-0-129-80.ec2.internal\" (UID: \"831a3fedbcc35899a7ac03738589a136\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal" Jul 06 19:05:25.696251 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:25.696204 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found" Jul 06 19:05:25.700872 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.700849 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8611695537a81ab018c1f7cb211fa60-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal\" (UID: \"c8611695537a81ab018c1f7cb211fa60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" Jul 06 19:05:25.700923 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.700883 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/831a3fedbcc35899a7ac03738589a136-config\") pod \"kube-apiserver-proxy-ip-10-0-129-80.ec2.internal\" (UID: \"831a3fedbcc35899a7ac03738589a136\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal" Jul 06 19:05:25.700923 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.700906 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c8611695537a81ab018c1f7cb211fa60-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal\" (UID: \"c8611695537a81ab018c1f7cb211fa60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" Jul 06 19:05:25.700991 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.700962 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/831a3fedbcc35899a7ac03738589a136-config\") pod \"kube-apiserver-proxy-ip-10-0-129-80.ec2.internal\" (UID: \"831a3fedbcc35899a7ac03738589a136\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal" Jul 06 19:05:25.700991 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.700962 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8611695537a81ab018c1f7cb211fa60-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal\" (UID: \"c8611695537a81ab018c1f7cb211fa60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" Jul 06 19:05:25.701055 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.701020 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c8611695537a81ab018c1f7cb211fa60-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal\" (UID: \"c8611695537a81ab018c1f7cb211fa60\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" Jul 06 19:05:25.799825 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:25.799731 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found" Jul 06 19:05:25.829288 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.829260 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" Jul 06 19:05:25.833835 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:25.833548 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal" Jul 06 19:05:25.900446 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:25.900409 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found" Jul 06 19:05:26.005506 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:26.005462 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found" Jul 06 19:05:26.106710 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:26.106637 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found" Jul 06 19:05:26.129506 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:26.129476 2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 19:05:26.129705 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:26.129681 2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 19:05:26.206985 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:26.206950 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found" Jul 06 19:05:26.206985 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:26.206977 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 19:00:25 +0000 UTC" deadline="2027-12-25 10:16:21.825019832 +0000 UTC" Jul 06 19:05:26.207139 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:26.207000 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12879h10m55.618023524s" Jul 06 19:05:26.208485 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:26.208467 2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:05:26.307699 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:26.307667 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found" Jul 06 19:05:26.408283 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:26.408220 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found" Jul 06 19:05:26.508810 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:26.508773 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-80.ec2.internal\" not found" Jul 06 19:05:26.526751 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:26.526708 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8611695537a81ab018c1f7cb211fa60.slice/crio-69d1468750cb35f51311ea577be14c0135d21b2941010034effda1e60bf68f46 WatchSource:0}: Error finding container 69d1468750cb35f51311ea577be14c0135d21b2941010034effda1e60bf68f46: Status 404 returned error can't find the container with id 69d1468750cb35f51311ea577be14c0135d21b2941010034effda1e60bf68f46 Jul 06 19:05:26.527426 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:26.527403 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod831a3fedbcc35899a7ac03738589a136.slice/crio-da9c57489f325d2241725629a6159ff9d8646a0172b60a8a93af2a5fd9514cc0 WatchSource:0}: Error finding container da9c57489f325d2241725629a6159ff9d8646a0172b60a8a93af2a5fd9514cc0: Status 404 returned error can't find the container with id da9c57489f325d2241725629a6159ff9d8646a0172b60a8a93af2a5fd9514cc0 Jul 06 19:05:26.534359 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:26.534344 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:05:26.536246 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:26.536230 2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:05:26.583821 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:26.583789 2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:05:26.598514 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:26.598489 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" Jul 06 19:05:26.609457 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:26.609431 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:05:26.610447 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:26.610430 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal" Jul 06 19:05:26.619404 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:26.619389 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 19:05:27.177584 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.177543 2560 apiserver.go:52] "Watching apiserver" Jul 06 19:05:27.187403 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.187371 2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 19:05:27.187889 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.187858 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-gghgg","openshift-multus/multus-additional-cni-plugins-vj7fc","openshift-network-diagnostics/network-check-target-ggvtt","openshift-network-operator/iptables-alerter-jbl9d","openshift-ovn-kubernetes/ovnkube-node-ng9fp","kube-system/global-pull-secret-syncer-dl7lj","kube-system/konnectivity-agent-z9v9z","kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal","openshift-cluster-node-tuning-operator/tuned-26m4d","openshift-dns/node-resolver-m5pz7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal","openshift-multus/multus-q9mrm","openshift-multus/network-metrics-daemon-mz7xr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh"] Jul 06 19:05:27.189568 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.189536 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.191148 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.190917 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vj7fc" Jul 06 19:05:27.192784 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.192501 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 19:05:27.192784 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.192635 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 19:05:27.192963 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.192943 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:27.193044 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:27.193015 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mz7xr" podUID="1f0f8d44-df3e-4fb6-afab-15fb76427980" Jul 06 19:05:27.193487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.193235 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-79fmx\"" Jul 06 19:05:27.193487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.193266 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 19:05:27.193487 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.193328 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 19:05:27.193779 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.193679 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 19:05:27.194746 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.194545 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 19:05:27.194746 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.194670 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vsqh8\"" Jul 06 19:05:27.194963 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.194948 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jbl9d" Jul 06 19:05:27.196149 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.196015 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z9v9z" Jul 06 19:05:27.197465 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.197443 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:27.197578 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:27.197507 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dl7lj" podUID="de165982-d4dc-4449-a40d-1fc3bf2754cc" Jul 06 19:05:27.197757 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.197633 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 19:05:27.197757 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.197688 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 19:05:27.197995 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.197977 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qb556\"" Jul 06 19:05:27.198156 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.197980 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:05:27.199082 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.198621 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 19:05:27.199082 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.198884 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 19:05:27.199231 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.199118 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 19:05:27.199336 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.199296 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gghgg" Jul 06 19:05:27.199804 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.199343 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5qmrt\"" Jul 06 19:05:27.202504 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.202321 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 19:05:27.202614 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.202553 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.202614 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.202601 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 19:05:27.202768 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.202673 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 19:05:27.203179 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.203150 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9htb5\"" Jul 06 19:05:27.205592 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.204245 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m5pz7" Jul 06 19:05:27.205937 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.205847 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.206104 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.206080 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:05:27.206332 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.206315 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 19:05:27.206559 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.206542 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kn4bb\"" Jul 06 19:05:27.207211 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.207176 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 19:05:27.207608 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.207571 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rg6tk\"" Jul 06 19:05:27.207947 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.207924 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 19:05:27.209406 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.209384 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xlqm9\"" Jul 06 19:05:27.209566 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.209393 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 19:05:27.209811 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.209754 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 19:05:27.209912 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.209660 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 19:05:27.210047 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.209703 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 19:05:27.210322 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.209603 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 19:05:27.210491 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.210474 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:05:27.210579 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:27.210547 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvtt" podUID="fc451a85-49aa-4966-b376-70bf49518b2d" Jul 06 19:05:27.211802 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.211741 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-host-var-lib-kubelet\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.211802 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.211777 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-multus-daemon-config\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.211802 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.211802 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/58a69bd8-7312-4728-9ac1-efc5384484bd-agent-certs\") pod \"konnectivity-agent-z9v9z\" (UID: \"58a69bd8-7312-4728-9ac1-efc5384484bd\") " pod="kube-system/konnectivity-agent-z9v9z" Jul 06 19:05:27.212036 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.211831 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b6e4a6e5-96c2-4496-af5a-e162d61d9c20-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vj7fc\" (UID: \"b6e4a6e5-96c2-4496-af5a-e162d61d9c20\") " pod="openshift-multus/multus-additional-cni-plugins-vj7fc" Jul 06 19:05:27.212036 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.211854 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjwws\" (UniqueName: \"kubernetes.io/projected/fe072be2-5f94-440e-b0e4-83e1e4899fc8-kube-api-access-xjwws\") pod \"node-ca-gghgg\" (UID: \"fe072be2-5f94-440e-b0e4-83e1e4899fc8\") " pod="openshift-image-registry/node-ca-gghgg" Jul 06 19:05:27.212036 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.211880 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-etc-sysconfig\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.212036 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.211904 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6scz\" (UniqueName: \"kubernetes.io/projected/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-kube-api-access-x6scz\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.212036 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.211931 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/58a69bd8-7312-4728-9ac1-efc5384484bd-konnectivity-ca\") pod \"konnectivity-agent-z9v9z\" (UID: \"58a69bd8-7312-4728-9ac1-efc5384484bd\") " pod="kube-system/konnectivity-agent-z9v9z" Jul 06 19:05:27.212036 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.211956 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft8b5\" (UniqueName: \"kubernetes.io/projected/b6e4a6e5-96c2-4496-af5a-e162d61d9c20-kube-api-access-ft8b5\") pod \"multus-additional-cni-plugins-vj7fc\" (UID: \"b6e4a6e5-96c2-4496-af5a-e162d61d9c20\") " pod="openshift-multus/multus-additional-cni-plugins-vj7fc" Jul 06 19:05:27.212036 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.211981 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0f8d44-df3e-4fb6-afab-15fb76427980-metrics-certs\") pod \"network-metrics-daemon-mz7xr\" (UID: \"1f0f8d44-df3e-4fb6-afab-15fb76427980\") " pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:27.212036 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212006 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b6e4a6e5-96c2-4496-af5a-e162d61d9c20-os-release\") pod \"multus-additional-cni-plugins-vj7fc\" (UID: \"b6e4a6e5-96c2-4496-af5a-e162d61d9c20\") " pod="openshift-multus/multus-additional-cni-plugins-vj7fc" Jul 06 19:05:27.212036 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212028 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-etc-systemd\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.212518 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212052 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-system-cni-dir\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.212518 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212078 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6e4a6e5-96c2-4496-af5a-e162d61d9c20-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vj7fc\" (UID: \"b6e4a6e5-96c2-4496-af5a-e162d61d9c20\") " pod="openshift-multus/multus-additional-cni-plugins-vj7fc" Jul 06 19:05:27.212518 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212106 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6cd7edc4-1efb-4b5e-9bf9-2e309a0a5ead-host-slash\") pod \"iptables-alerter-jbl9d\" (UID: \"6cd7edc4-1efb-4b5e-9bf9-2e309a0a5ead\") " pod="openshift-network-operator/iptables-alerter-jbl9d" Jul 06 19:05:27.212518 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212132 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/de165982-d4dc-4449-a40d-1fc3bf2754cc-kubelet-config\") pod \"global-pull-secret-syncer-dl7lj\" (UID: \"de165982-d4dc-4449-a40d-1fc3bf2754cc\") " pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:27.212518 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212155 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fe072be2-5f94-440e-b0e4-83e1e4899fc8-serviceca\") pod \"node-ca-gghgg\" (UID: \"fe072be2-5f94-440e-b0e4-83e1e4899fc8\") " pod="openshift-image-registry/node-ca-gghgg" Jul 06 19:05:27.212518 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212177 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-etc-sysctl-conf\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.212518 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212201 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-run\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.212518 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212224 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c61b821d-85b4-4950-af28-27ecb921cc34-tmp\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.212518 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212246 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-hostroot\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.212518 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212269 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6e4a6e5-96c2-4496-af5a-e162d61d9c20-system-cni-dir\") pod \"multus-additional-cni-plugins-vj7fc\" (UID: \"b6e4a6e5-96c2-4496-af5a-e162d61d9c20\") " pod="openshift-multus/multus-additional-cni-plugins-vj7fc" Jul 06 19:05:27.212518 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212294 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6e4a6e5-96c2-4496-af5a-e162d61d9c20-cnibin\") pod \"multus-additional-cni-plugins-vj7fc\" (UID: \"b6e4a6e5-96c2-4496-af5a-e162d61d9c20\") " pod="openshift-multus/multus-additional-cni-plugins-vj7fc" Jul 06 19:05:27.212518 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212322 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b6e4a6e5-96c2-4496-af5a-e162d61d9c20-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vj7fc\" (UID: \"b6e4a6e5-96c2-4496-af5a-e162d61d9c20\") " pod="openshift-multus/multus-additional-cni-plugins-vj7fc" Jul 06 19:05:27.212518 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212346 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/de165982-d4dc-4449-a40d-1fc3bf2754cc-dbus\") pod \"global-pull-secret-syncer-dl7lj\" (UID: \"de165982-d4dc-4449-a40d-1fc3bf2754cc\") " pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:27.212518 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212391 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-sys\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.212518 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212416 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-var-lib-kubelet\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.212518 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212442 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-host-run-netns\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.212518 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212484 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-host-run-multus-certs\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.213405 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212510 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6cd7edc4-1efb-4b5e-9bf9-2e309a0a5ead-iptables-alerter-script\") pod \"iptables-alerter-jbl9d\" (UID: \"6cd7edc4-1efb-4b5e-9bf9-2e309a0a5ead\") " pod="openshift-network-operator/iptables-alerter-jbl9d" Jul 06 19:05:27.213405 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212535 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe072be2-5f94-440e-b0e4-83e1e4899fc8-host\") pod \"node-ca-gghgg\" (UID: \"fe072be2-5f94-440e-b0e4-83e1e4899fc8\") " pod="openshift-image-registry/node-ca-gghgg" Jul 06 19:05:27.213405 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212558 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c61b821d-85b4-4950-af28-27ecb921cc34-etc-tuned\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.213405 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212584 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-multus-cni-dir\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.213405 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212610 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-cni-binary-copy\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.213405 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212625 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 19:05:27.213405 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212639 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-multus-socket-dir-parent\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.213405 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212682 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-host-run-k8s-cni-cncf-io\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.213405 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212723 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-host-var-lib-cni-bin\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.213405 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212772 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggq6j\" (UniqueName: \"kubernetes.io/projected/6cd7edc4-1efb-4b5e-9bf9-2e309a0a5ead-kube-api-access-ggq6j\") pod \"iptables-alerter-jbl9d\" (UID: \"6cd7edc4-1efb-4b5e-9bf9-2e309a0a5ead\") " pod="openshift-network-operator/iptables-alerter-jbl9d" Jul 06 19:05:27.213405 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212853 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de165982-d4dc-4449-a40d-1fc3bf2754cc-original-pull-secret\") pod \"global-pull-secret-syncer-dl7lj\" (UID: \"de165982-d4dc-4449-a40d-1fc3bf2754cc\") " pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:27.213405 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212898 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-etc-modprobe-d\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.213405 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212937 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-etc-kubernetes\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.213405 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212963 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-etc-sysctl-d\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.213405 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.212987 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-lib-modules\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.213405 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.213013 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p6sd\" (UniqueName: \"kubernetes.io/projected/c61b821d-85b4-4950-af28-27ecb921cc34-kube-api-access-6p6sd\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.213405 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.213048 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-cnibin\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.214314 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.213080 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhlk8\" (UniqueName: \"kubernetes.io/projected/1f0f8d44-df3e-4fb6-afab-15fb76427980-kube-api-access-fhlk8\") pod \"network-metrics-daemon-mz7xr\" (UID: \"1f0f8d44-df3e-4fb6-afab-15fb76427980\") " pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:27.214314 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.213104 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-host\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.214314 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.213139 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-os-release\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.214314 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.213210 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-multus-conf-dir\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.214314 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.213234 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-etc-kubernetes\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.214314 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.213257 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6e4a6e5-96c2-4496-af5a-e162d61d9c20-cni-binary-copy\") pod \"multus-additional-cni-plugins-vj7fc\" (UID: \"b6e4a6e5-96c2-4496-af5a-e162d61d9c20\") " pod="openshift-multus/multus-additional-cni-plugins-vj7fc" Jul 06 19:05:27.214314 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.213293 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" Jul 06 19:05:27.214314 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.213313 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-host-var-lib-cni-multus\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.214793 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.214771 2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 19:05:27.215702 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.215680 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 19:05:27.216159 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.216136 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 19:05:27.216245 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.216137 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 19:05:27.216422 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.216396 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w7z95\"" Jul 06 19:05:27.234161 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.234136 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-g2jxd" Jul 06 19:05:27.242985 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.242961 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-g2jxd" Jul 06 19:05:27.301188 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.301162 2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 19:05:27.313980 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.313948 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/58a69bd8-7312-4728-9ac1-efc5384484bd-konnectivity-ca\") pod \"konnectivity-agent-z9v9z\" (UID: \"58a69bd8-7312-4728-9ac1-efc5384484bd\") " pod="kube-system/konnectivity-agent-z9v9z" Jul 06 19:05:27.314105 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.313987 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6e4a6e5-96c2-4496-af5a-e162d61d9c20-cni-binary-copy\") pod \"multus-additional-cni-plugins-vj7fc\" (UID: \"b6e4a6e5-96c2-4496-af5a-e162d61d9c20\") " pod="openshift-multus/multus-additional-cni-plugins-vj7fc" Jul 06 19:05:27.314105 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314017 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-host-var-lib-cni-multus\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.314105 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314041 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/58a69bd8-7312-4728-9ac1-efc5384484bd-agent-certs\") pod \"konnectivity-agent-z9v9z\" (UID: \"58a69bd8-7312-4728-9ac1-efc5384484bd\") " pod="kube-system/konnectivity-agent-z9v9z" Jul 06 19:05:27.314105 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314073 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/59dc1490-ee33-4cc6-86ff-d6a3172dcc05-sys-fs\") pod \"aws-ebs-csi-driver-node-lxnbh\" (UID: \"59dc1490-ee33-4cc6-86ff-d6a3172dcc05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" Jul 06 19:05:27.314105 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314098 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-host-kubelet\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.314397 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314128 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-run-openvswitch\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.314397 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314155 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-etc-sysconfig\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.314397 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314184 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfwxp\" (UniqueName: \"kubernetes.io/projected/fc451a85-49aa-4966-b376-70bf49518b2d-kube-api-access-pfwxp\") pod \"network-check-target-ggvtt\" (UID: \"fc451a85-49aa-4966-b376-70bf49518b2d\") " pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:05:27.314397 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314216 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0f8d44-df3e-4fb6-afab-15fb76427980-metrics-certs\") pod \"network-metrics-daemon-mz7xr\" (UID: \"1f0f8d44-df3e-4fb6-afab-15fb76427980\") " pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:27.314397 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314242 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b6e4a6e5-96c2-4496-af5a-e162d61d9c20-os-release\") pod \"multus-additional-cni-plugins-vj7fc\" (UID: \"b6e4a6e5-96c2-4496-af5a-e162d61d9c20\") " pod="openshift-multus/multus-additional-cni-plugins-vj7fc" Jul 06 19:05:27.314397 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314268 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-etc-systemd\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.314397 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314291 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-system-cni-dir\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.314397 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314314 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-host-run-netns\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.314397 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314339 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6cd7edc4-1efb-4b5e-9bf9-2e309a0a5ead-host-slash\") pod \"iptables-alerter-jbl9d\" (UID: \"6cd7edc4-1efb-4b5e-9bf9-2e309a0a5ead\") " pod="openshift-network-operator/iptables-alerter-jbl9d" Jul 06 19:05:27.314397 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314366 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/de165982-d4dc-4449-a40d-1fc3bf2754cc-kubelet-config\") pod \"global-pull-secret-syncer-dl7lj\" (UID: \"de165982-d4dc-4449-a40d-1fc3bf2754cc\") " pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:27.314397 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314393 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-run\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.315043 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314416 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c61b821d-85b4-4950-af28-27ecb921cc34-tmp\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.315043 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314445 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-hostroot\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.315043 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314471 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84c43c19-1748-4d6b-8c86-b9171445c9b8-tmp-dir\") pod \"node-resolver-m5pz7\" (UID: \"84c43c19-1748-4d6b-8c86-b9171445c9b8\") " pod="openshift-dns/node-resolver-m5pz7" Jul 06 19:05:27.315043 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314495 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nsmf\" (UniqueName: \"kubernetes.io/projected/84c43c19-1748-4d6b-8c86-b9171445c9b8-kube-api-access-9nsmf\") pod \"node-resolver-m5pz7\" (UID: \"84c43c19-1748-4d6b-8c86-b9171445c9b8\") " pod="openshift-dns/node-resolver-m5pz7" Jul 06 19:05:27.315043 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314522 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c223b4bf-f78a-41a5-9568-d60697508aa5-ovnkube-script-lib\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.315043 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314549 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6e4a6e5-96c2-4496-af5a-e162d61d9c20-cnibin\") pod \"multus-additional-cni-plugins-vj7fc\" (UID: \"b6e4a6e5-96c2-4496-af5a-e162d61d9c20\") " pod="openshift-multus/multus-additional-cni-plugins-vj7fc" Jul 06 19:05:27.315043 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314573 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-sys\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.315043 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314599 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-var-lib-kubelet\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.315043 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314625 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-host-run-multus-certs\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.315043 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314675 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/59dc1490-ee33-4cc6-86ff-d6a3172dcc05-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lxnbh\" (UID: \"59dc1490-ee33-4cc6-86ff-d6a3172dcc05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" Jul 06 19:05:27.315043 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314691 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/58a69bd8-7312-4728-9ac1-efc5384484bd-konnectivity-ca\") pod \"konnectivity-agent-z9v9z\" (UID: \"58a69bd8-7312-4728-9ac1-efc5384484bd\") " pod="kube-system/konnectivity-agent-z9v9z" Jul 06 19:05:27.315043 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314714 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-var-lib-openvswitch\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.315043 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314743 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe072be2-5f94-440e-b0e4-83e1e4899fc8-host\") pod \"node-ca-gghgg\" (UID: \"fe072be2-5f94-440e-b0e4-83e1e4899fc8\") " pod="openshift-image-registry/node-ca-gghgg" Jul 06 19:05:27.315043 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314769 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c61b821d-85b4-4950-af28-27ecb921cc34-etc-tuned\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.315043 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314796 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-multus-socket-dir-parent\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.315043 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314822 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-multus-daemon-config\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.315043 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314849 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggq6j\" (UniqueName: \"kubernetes.io/projected/6cd7edc4-1efb-4b5e-9bf9-2e309a0a5ead-kube-api-access-ggq6j\") pod \"iptables-alerter-jbl9d\" (UID: \"6cd7edc4-1efb-4b5e-9bf9-2e309a0a5ead\") " pod="openshift-network-operator/iptables-alerter-jbl9d" Jul 06 19:05:27.315940 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314878 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de165982-d4dc-4449-a40d-1fc3bf2754cc-original-pull-secret\") pod \"global-pull-secret-syncer-dl7lj\" (UID: \"de165982-d4dc-4449-a40d-1fc3bf2754cc\") " pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:27.315940 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314905 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-etc-modprobe-d\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.315940 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314915 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-run\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.315940 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314970 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-etc-kubernetes\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.315940 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.314989 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-lib-modules\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.315940 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.315005 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p6sd\" (UniqueName: \"kubernetes.io/projected/c61b821d-85b4-4950-af28-27ecb921cc34-kube-api-access-6p6sd\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.315940 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.315050 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-cnibin\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.315940 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.315193 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-system-cni-dir\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.315940 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.315207 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-etc-sysconfig\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.315940 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.315308 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b6e4a6e5-96c2-4496-af5a-e162d61d9c20-os-release\") pod \"multus-additional-cni-plugins-vj7fc\" (UID: \"b6e4a6e5-96c2-4496-af5a-e162d61d9c20\") " pod="openshift-multus/multus-additional-cni-plugins-vj7fc" Jul 06 19:05:27.315940 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:27.315397 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:05:27.315940 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.315403 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fe072be2-5f94-440e-b0e4-83e1e4899fc8-host\") pod \"node-ca-gghgg\" (UID: \"fe072be2-5f94-440e-b0e4-83e1e4899fc8\") " pod="openshift-image-registry/node-ca-gghgg" Jul 06 19:05:27.315940 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.315437 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-run-ovn\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.315940 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.315469 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6e4a6e5-96c2-4496-af5a-e162d61d9c20-cni-binary-copy\") pod \"multus-additional-cni-plugins-vj7fc\" (UID: \"b6e4a6e5-96c2-4496-af5a-e162d61d9c20\") " pod="openshift-multus/multus-additional-cni-plugins-vj7fc" Jul 06 19:05:27.315940 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:27.315478 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f0f8d44-df3e-4fb6-afab-15fb76427980-metrics-certs podName:1f0f8d44-df3e-4fb6-afab-15fb76427980 nodeName:}" failed. No retries permitted until 2026-07-06 19:05:27.815439599 +0000 UTC m=+3.006807269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f0f8d44-df3e-4fb6-afab-15fb76427980-metrics-certs") pod "network-metrics-daemon-mz7xr" (UID: "1f0f8d44-df3e-4fb6-afab-15fb76427980") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:05:27.315940 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.315491 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-cnibin\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.315940 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.315509 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-host\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.316905 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.315534 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/de165982-d4dc-4449-a40d-1fc3bf2754cc-kubelet-config\") pod \"global-pull-secret-syncer-dl7lj\" (UID: \"de165982-d4dc-4449-a40d-1fc3bf2754cc\") " pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:27.316905 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.315538 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c223b4bf-f78a-41a5-9568-d60697508aa5-ovnkube-config\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.316905 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.315546 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-multus-socket-dir-parent\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.316905 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.315525 2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 19:05:27.316905 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.315618 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-etc-systemd\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.316905 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.315682 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-host-var-lib-cni-multus\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.316905 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.315723 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6cd7edc4-1efb-4b5e-9bf9-2e309a0a5ead-host-slash\") pod \"iptables-alerter-jbl9d\" (UID: \"6cd7edc4-1efb-4b5e-9bf9-2e309a0a5ead\") " pod="openshift-network-operator/iptables-alerter-jbl9d" Jul 06 19:05:27.316905 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.315777 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-host\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.316905 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.315787 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/59dc1490-ee33-4cc6-86ff-d6a3172dcc05-device-dir\") pod \"aws-ebs-csi-driver-node-lxnbh\" (UID: \"59dc1490-ee33-4cc6-86ff-d6a3172dcc05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" Jul 06 19:05:27.316905 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.315821 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-host-run-ovn-kubernetes\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.316905 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.315829 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-hostroot\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.316905 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.315852 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b6e4a6e5-96c2-4496-af5a-e162d61d9c20-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vj7fc\" (UID: \"b6e4a6e5-96c2-4496-af5a-e162d61d9c20\") " pod="openshift-multus/multus-additional-cni-plugins-vj7fc" Jul 06 19:05:27.316905 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.315884 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjwws\" (UniqueName: \"kubernetes.io/projected/fe072be2-5f94-440e-b0e4-83e1e4899fc8-kube-api-access-xjwws\") pod \"node-ca-gghgg\" (UID: \"fe072be2-5f94-440e-b0e4-83e1e4899fc8\") " pod="openshift-image-registry/node-ca-gghgg" Jul 06 19:05:27.316905 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.315912 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6scz\" (UniqueName: \"kubernetes.io/projected/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-kube-api-access-x6scz\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.316905 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.315938 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/59dc1490-ee33-4cc6-86ff-d6a3172dcc05-socket-dir\") pod \"aws-ebs-csi-driver-node-lxnbh\" (UID: \"59dc1490-ee33-4cc6-86ff-d6a3172dcc05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" Jul 06 19:05:27.316905 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.315965 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/59dc1490-ee33-4cc6-86ff-d6a3172dcc05-registration-dir\") pod \"aws-ebs-csi-driver-node-lxnbh\" (UID: \"59dc1490-ee33-4cc6-86ff-d6a3172dcc05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" Jul 06 19:05:27.316905 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.315993 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-run-systemd\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.317758 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.316018 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-node-log\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.317758 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.316044 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-log-socket\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.317758 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.316144 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft8b5\" (UniqueName: \"kubernetes.io/projected/b6e4a6e5-96c2-4496-af5a-e162d61d9c20-kube-api-access-ft8b5\") pod \"multus-additional-cni-plugins-vj7fc\" (UID: \"b6e4a6e5-96c2-4496-af5a-e162d61d9c20\") " pod="openshift-multus/multus-additional-cni-plugins-vj7fc" Jul 06 19:05:27.317758 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.316175 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znlb6\" (UniqueName: \"kubernetes.io/projected/59dc1490-ee33-4cc6-86ff-d6a3172dcc05-kube-api-access-znlb6\") pod \"aws-ebs-csi-driver-node-lxnbh\" (UID: \"59dc1490-ee33-4cc6-86ff-d6a3172dcc05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" Jul 06 19:05:27.317758 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.316205 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.317758 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.316237 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6e4a6e5-96c2-4496-af5a-e162d61d9c20-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vj7fc\" (UID: \"b6e4a6e5-96c2-4496-af5a-e162d61d9c20\") " pod="openshift-multus/multus-additional-cni-plugins-vj7fc" Jul 06 19:05:27.317758 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.316279 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fe072be2-5f94-440e-b0e4-83e1e4899fc8-serviceca\") pod \"node-ca-gghgg\" (UID: \"fe072be2-5f94-440e-b0e4-83e1e4899fc8\") " pod="openshift-image-registry/node-ca-gghgg" Jul 06 19:05:27.317758 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.316306 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-etc-sysctl-conf\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.317758 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.316331 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-etc-openvswitch\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.317758 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.316358 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64lh6\" (UniqueName: \"kubernetes.io/projected/c223b4bf-f78a-41a5-9568-d60697508aa5-kube-api-access-64lh6\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.317758 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.316368 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-multus-daemon-config\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.317758 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.316388 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6e4a6e5-96c2-4496-af5a-e162d61d9c20-system-cni-dir\") pod \"multus-additional-cni-plugins-vj7fc\" (UID: \"b6e4a6e5-96c2-4496-af5a-e162d61d9c20\") " pod="openshift-multus/multus-additional-cni-plugins-vj7fc" Jul 06 19:05:27.317758 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.316418 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b6e4a6e5-96c2-4496-af5a-e162d61d9c20-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vj7fc\" (UID: \"b6e4a6e5-96c2-4496-af5a-e162d61d9c20\") " pod="openshift-multus/multus-additional-cni-plugins-vj7fc" Jul 06 19:05:27.317758 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.316445 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/de165982-d4dc-4449-a40d-1fc3bf2754cc-dbus\") pod \"global-pull-secret-syncer-dl7lj\" (UID: \"de165982-d4dc-4449-a40d-1fc3bf2754cc\") " pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:27.317758 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.316476 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-host-run-netns\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.317758 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.316506 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/84c43c19-1748-4d6b-8c86-b9171445c9b8-hosts-file\") pod \"node-resolver-m5pz7\" (UID: \"84c43c19-1748-4d6b-8c86-b9171445c9b8\") " pod="openshift-dns/node-resolver-m5pz7" Jul 06 19:05:27.318551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.316534 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c223b4bf-f78a-41a5-9568-d60697508aa5-ovn-node-metrics-cert\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.318551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.316564 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6cd7edc4-1efb-4b5e-9bf9-2e309a0a5ead-iptables-alerter-script\") pod \"iptables-alerter-jbl9d\" (UID: \"6cd7edc4-1efb-4b5e-9bf9-2e309a0a5ead\") " pod="openshift-network-operator/iptables-alerter-jbl9d" Jul 06 19:05:27.318551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.316591 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-multus-cni-dir\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.318551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.316619 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-cni-binary-copy\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.318551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.316661 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-host-run-k8s-cni-cncf-io\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.318551 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:27.316689 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:05:27.318551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.316698 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-host-var-lib-cni-bin\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.318551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.316730 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-host-var-lib-kubelet\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.318551 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:27.316749 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de165982-d4dc-4449-a40d-1fc3bf2754cc-original-pull-secret podName:de165982-d4dc-4449-a40d-1fc3bf2754cc nodeName:}" failed. No retries permitted until 2026-07-06 19:05:27.816730196 +0000 UTC m=+3.008097856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de165982-d4dc-4449-a40d-1fc3bf2754cc-original-pull-secret") pod "global-pull-secret-syncer-dl7lj" (UID: "de165982-d4dc-4449-a40d-1fc3bf2754cc") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:05:27.318551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.316777 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-host-var-lib-kubelet\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.318551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.316857 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6e4a6e5-96c2-4496-af5a-e162d61d9c20-system-cni-dir\") pod \"multus-additional-cni-plugins-vj7fc\" (UID: \"b6e4a6e5-96c2-4496-af5a-e162d61d9c20\") " pod="openshift-multus/multus-additional-cni-plugins-vj7fc" Jul 06 19:05:27.318551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.317348 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6e4a6e5-96c2-4496-af5a-e162d61d9c20-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vj7fc\" (UID: \"b6e4a6e5-96c2-4496-af5a-e162d61d9c20\") " pod="openshift-multus/multus-additional-cni-plugins-vj7fc" Jul 06 19:05:27.318551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.317378 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b6e4a6e5-96c2-4496-af5a-e162d61d9c20-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vj7fc\" (UID: \"b6e4a6e5-96c2-4496-af5a-e162d61d9c20\") " pod="openshift-multus/multus-additional-cni-plugins-vj7fc" Jul 06 19:05:27.318551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.317455 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-etc-sysctl-conf\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.318551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.317504 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/de165982-d4dc-4449-a40d-1fc3bf2754cc-dbus\") pod \"global-pull-secret-syncer-dl7lj\" (UID: \"de165982-d4dc-4449-a40d-1fc3bf2754cc\") " pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:27.318551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.317547 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6e4a6e5-96c2-4496-af5a-e162d61d9c20-cnibin\") pod \"multus-additional-cni-plugins-vj7fc\" (UID: \"b6e4a6e5-96c2-4496-af5a-e162d61d9c20\") " pod="openshift-multus/multus-additional-cni-plugins-vj7fc" Jul 06 19:05:27.318551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.317568 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-lib-modules\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.319488 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.317593 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-sys\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.319488 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.317622 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-host-run-netns\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.319488 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.317638 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-var-lib-kubelet\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.319488 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.317690 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-host-run-multus-certs\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.319488 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.318100 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fe072be2-5f94-440e-b0e4-83e1e4899fc8-serviceca\") pod \"node-ca-gghgg\" (UID: \"fe072be2-5f94-440e-b0e4-83e1e4899fc8\") " pod="openshift-image-registry/node-ca-gghgg" Jul 06 19:05:27.319488 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.318266 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6cd7edc4-1efb-4b5e-9bf9-2e309a0a5ead-iptables-alerter-script\") pod \"iptables-alerter-jbl9d\" (UID: \"6cd7edc4-1efb-4b5e-9bf9-2e309a0a5ead\") " pod="openshift-network-operator/iptables-alerter-jbl9d" Jul 06 19:05:27.319488 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.318349 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-multus-cni-dir\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.319488 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.318612 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b6e4a6e5-96c2-4496-af5a-e162d61d9c20-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vj7fc\" (UID: \"b6e4a6e5-96c2-4496-af5a-e162d61d9c20\") " pod="openshift-multus/multus-additional-cni-plugins-vj7fc" Jul 06 19:05:27.319488 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.318879 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-cni-binary-copy\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.319488 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.318934 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-host-cni-bin\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.320036 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.319681 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-etc-modprobe-d\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.320036 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.318971 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c223b4bf-f78a-41a5-9568-d60697508aa5-env-overrides\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.320036 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.319787 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-host-run-k8s-cni-cncf-io\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.320036 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.319867 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-etc-kubernetes\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.320276 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.320253 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/58a69bd8-7312-4728-9ac1-efc5384484bd-agent-certs\") pod \"konnectivity-agent-z9v9z\" (UID: \"58a69bd8-7312-4728-9ac1-efc5384484bd\") " pod="kube-system/konnectivity-agent-z9v9z" Jul 06 19:05:27.325270 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.320424 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-etc-sysctl-d\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.325270 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.320500 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/59dc1490-ee33-4cc6-86ff-d6a3172dcc05-etc-selinux\") pod \"aws-ebs-csi-driver-node-lxnbh\" (UID: \"59dc1490-ee33-4cc6-86ff-d6a3172dcc05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" Jul 06 19:05:27.325270 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.320580 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-systemd-units\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.325270 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.320616 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-host-cni-netd\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.325270 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.320674 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhlk8\" (UniqueName: \"kubernetes.io/projected/1f0f8d44-df3e-4fb6-afab-15fb76427980-kube-api-access-fhlk8\") pod \"network-metrics-daemon-mz7xr\" (UID: \"1f0f8d44-df3e-4fb6-afab-15fb76427980\") " pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:27.325270 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.320715 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-os-release\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.325270 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.321009 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-multus-conf-dir\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.325270 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.321055 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-etc-kubernetes\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.325270 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.321087 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c61b821d-85b4-4950-af28-27ecb921cc34-etc-sysctl-d\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.325270 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.321093 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-host-slash\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.325270 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.321153 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c61b821d-85b4-4950-af28-27ecb921cc34-etc-tuned\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.325270 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.321163 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-os-release\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.325270 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.321207 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-host-var-lib-cni-bin\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.325270 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.321377 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-multus-conf-dir\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.325270 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.321486 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-etc-kubernetes\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.325270 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.323462 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c61b821d-85b4-4950-af28-27ecb921cc34-tmp\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.328273 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.328244 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p6sd\" (UniqueName: \"kubernetes.io/projected/c61b821d-85b4-4950-af28-27ecb921cc34-kube-api-access-6p6sd\") pod \"tuned-26m4d\" (UID: \"c61b821d-85b4-4950-af28-27ecb921cc34\") " pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.328524 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.328359 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft8b5\" (UniqueName: \"kubernetes.io/projected/b6e4a6e5-96c2-4496-af5a-e162d61d9c20-kube-api-access-ft8b5\") pod \"multus-additional-cni-plugins-vj7fc\" (UID: \"b6e4a6e5-96c2-4496-af5a-e162d61d9c20\") " pod="openshift-multus/multus-additional-cni-plugins-vj7fc" Jul 06 19:05:27.329468 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.329429 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjwws\" (UniqueName: \"kubernetes.io/projected/fe072be2-5f94-440e-b0e4-83e1e4899fc8-kube-api-access-xjwws\") pod \"node-ca-gghgg\" (UID: \"fe072be2-5f94-440e-b0e4-83e1e4899fc8\") " pod="openshift-image-registry/node-ca-gghgg" Jul 06 19:05:27.329942 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.329895 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6scz\" (UniqueName: \"kubernetes.io/projected/8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28-kube-api-access-x6scz\") pod \"multus-q9mrm\" (UID: \"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28\") " pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.330471 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.330384 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggq6j\" (UniqueName: \"kubernetes.io/projected/6cd7edc4-1efb-4b5e-9bf9-2e309a0a5ead-kube-api-access-ggq6j\") pod \"iptables-alerter-jbl9d\" (UID: \"6cd7edc4-1efb-4b5e-9bf9-2e309a0a5ead\") " pod="openshift-network-operator/iptables-alerter-jbl9d" Jul 06 19:05:27.330551 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.330484 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhlk8\" (UniqueName: \"kubernetes.io/projected/1f0f8d44-df3e-4fb6-afab-15fb76427980-kube-api-access-fhlk8\") pod \"network-metrics-daemon-mz7xr\" (UID: \"1f0f8d44-df3e-4fb6-afab-15fb76427980\") " pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:27.391818 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.391755 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" event={"ID":"c8611695537a81ab018c1f7cb211fa60","Type":"ContainerStarted","Data":"69d1468750cb35f51311ea577be14c0135d21b2941010034effda1e60bf68f46"} Jul 06 19:05:27.393280 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.393250 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal" event={"ID":"831a3fedbcc35899a7ac03738589a136","Type":"ContainerStarted","Data":"da9c57489f325d2241725629a6159ff9d8646a0172b60a8a93af2a5fd9514cc0"} Jul 06 19:05:27.421807 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.421754 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-host-run-netns\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.421807 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.421802 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84c43c19-1748-4d6b-8c86-b9171445c9b8-tmp-dir\") pod \"node-resolver-m5pz7\" (UID: \"84c43c19-1748-4d6b-8c86-b9171445c9b8\") " pod="openshift-dns/node-resolver-m5pz7" Jul 06 19:05:27.422050 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.421829 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nsmf\" (UniqueName: \"kubernetes.io/projected/84c43c19-1748-4d6b-8c86-b9171445c9b8-kube-api-access-9nsmf\") pod \"node-resolver-m5pz7\" (UID: \"84c43c19-1748-4d6b-8c86-b9171445c9b8\") " pod="openshift-dns/node-resolver-m5pz7" Jul 06 19:05:27.422050 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.421856 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c223b4bf-f78a-41a5-9568-d60697508aa5-ovnkube-script-lib\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.422050 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.421883 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/59dc1490-ee33-4cc6-86ff-d6a3172dcc05-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lxnbh\" (UID: \"59dc1490-ee33-4cc6-86ff-d6a3172dcc05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" Jul 06 19:05:27.422050 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.421896 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-host-run-netns\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.422285 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422249 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84c43c19-1748-4d6b-8c86-b9171445c9b8-tmp-dir\") pod \"node-resolver-m5pz7\" (UID: \"84c43c19-1748-4d6b-8c86-b9171445c9b8\") " pod="openshift-dns/node-resolver-m5pz7" Jul 06 19:05:27.422285 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.421908 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-var-lib-openvswitch\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.422464 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422339 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/59dc1490-ee33-4cc6-86ff-d6a3172dcc05-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lxnbh\" (UID: \"59dc1490-ee33-4cc6-86ff-d6a3172dcc05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" Jul 06 19:05:27.422464 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422352 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-run-ovn\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.422464 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422367 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-var-lib-openvswitch\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.422464 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422391 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c223b4bf-f78a-41a5-9568-d60697508aa5-ovnkube-config\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.422464 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422413 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-run-ovn\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.422464 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422424 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/59dc1490-ee33-4cc6-86ff-d6a3172dcc05-device-dir\") pod \"aws-ebs-csi-driver-node-lxnbh\" (UID: \"59dc1490-ee33-4cc6-86ff-d6a3172dcc05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" Jul 06 19:05:27.422464 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422467 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-host-run-ovn-kubernetes\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.422868 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422475 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/59dc1490-ee33-4cc6-86ff-d6a3172dcc05-device-dir\") pod \"aws-ebs-csi-driver-node-lxnbh\" (UID: \"59dc1490-ee33-4cc6-86ff-d6a3172dcc05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" Jul 06 19:05:27.422868 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422490 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/59dc1490-ee33-4cc6-86ff-d6a3172dcc05-socket-dir\") pod \"aws-ebs-csi-driver-node-lxnbh\" (UID: \"59dc1490-ee33-4cc6-86ff-d6a3172dcc05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" Jul 06 19:05:27.422868 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422516 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/59dc1490-ee33-4cc6-86ff-d6a3172dcc05-registration-dir\") pod \"aws-ebs-csi-driver-node-lxnbh\" (UID: \"59dc1490-ee33-4cc6-86ff-d6a3172dcc05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" Jul 06 19:05:27.422868 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422522 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-host-run-ovn-kubernetes\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.422868 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422543 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-run-systemd\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.422868 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422569 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-node-log\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.422868 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422578 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/59dc1490-ee33-4cc6-86ff-d6a3172dcc05-registration-dir\") pod \"aws-ebs-csi-driver-node-lxnbh\" (UID: \"59dc1490-ee33-4cc6-86ff-d6a3172dcc05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" Jul 06 19:05:27.422868 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422624 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-run-systemd\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.422868 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422648 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-log-socket\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.422868 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422634 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/59dc1490-ee33-4cc6-86ff-d6a3172dcc05-socket-dir\") pod \"aws-ebs-csi-driver-node-lxnbh\" (UID: \"59dc1490-ee33-4cc6-86ff-d6a3172dcc05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" Jul 06 19:05:27.422868 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422627 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-node-log\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.422868 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422687 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znlb6\" (UniqueName: \"kubernetes.io/projected/59dc1490-ee33-4cc6-86ff-d6a3172dcc05-kube-api-access-znlb6\") pod \"aws-ebs-csi-driver-node-lxnbh\" (UID: \"59dc1490-ee33-4cc6-86ff-d6a3172dcc05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" Jul 06 19:05:27.422868 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422713 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-log-socket\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.422868 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422719 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.422868 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422747 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c223b4bf-f78a-41a5-9568-d60697508aa5-ovnkube-script-lib\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.422868 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422761 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-etc-openvswitch\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.422868 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422762 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.423697 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422804 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-etc-openvswitch\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.423697 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422806 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64lh6\" (UniqueName: \"kubernetes.io/projected/c223b4bf-f78a-41a5-9568-d60697508aa5-kube-api-access-64lh6\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.423697 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422853 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/84c43c19-1748-4d6b-8c86-b9171445c9b8-hosts-file\") pod \"node-resolver-m5pz7\" (UID: \"84c43c19-1748-4d6b-8c86-b9171445c9b8\") " pod="openshift-dns/node-resolver-m5pz7" Jul 06 19:05:27.423697 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422880 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c223b4bf-f78a-41a5-9568-d60697508aa5-ovn-node-metrics-cert\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.423697 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422880 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c223b4bf-f78a-41a5-9568-d60697508aa5-ovnkube-config\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.423697 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422940 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-host-cni-bin\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.423697 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422988 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c223b4bf-f78a-41a5-9568-d60697508aa5-env-overrides\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.423697 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.422993 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/84c43c19-1748-4d6b-8c86-b9171445c9b8-hosts-file\") pod \"node-resolver-m5pz7\" (UID: \"84c43c19-1748-4d6b-8c86-b9171445c9b8\") " pod="openshift-dns/node-resolver-m5pz7" Jul 06 19:05:27.423697 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.423015 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/59dc1490-ee33-4cc6-86ff-d6a3172dcc05-etc-selinux\") pod \"aws-ebs-csi-driver-node-lxnbh\" (UID: \"59dc1490-ee33-4cc6-86ff-d6a3172dcc05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" Jul 06 19:05:27.423697 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.423044 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-systemd-units\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.423697 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.423052 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-host-cni-bin\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.423697 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.423077 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-host-cni-netd\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.423697 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.423093 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-systemd-units\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.423697 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.423109 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-host-slash\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.423697 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.423136 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-host-cni-netd\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.423697 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.423144 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/59dc1490-ee33-4cc6-86ff-d6a3172dcc05-etc-selinux\") pod \"aws-ebs-csi-driver-node-lxnbh\" (UID: \"59dc1490-ee33-4cc6-86ff-d6a3172dcc05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" Jul 06 19:05:27.423697 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.423149 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/59dc1490-ee33-4cc6-86ff-d6a3172dcc05-sys-fs\") pod \"aws-ebs-csi-driver-node-lxnbh\" (UID: \"59dc1490-ee33-4cc6-86ff-d6a3172dcc05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" Jul 06 19:05:27.424601 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.423175 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-host-slash\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.424601 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.423195 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-host-kubelet\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.424601 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.423200 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/59dc1490-ee33-4cc6-86ff-d6a3172dcc05-sys-fs\") pod \"aws-ebs-csi-driver-node-lxnbh\" (UID: \"59dc1490-ee33-4cc6-86ff-d6a3172dcc05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" Jul 06 19:05:27.424601 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.423223 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-run-openvswitch\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.424601 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.423248 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-host-kubelet\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.424601 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.423254 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfwxp\" (UniqueName: \"kubernetes.io/projected/fc451a85-49aa-4966-b376-70bf49518b2d-kube-api-access-pfwxp\") pod \"network-check-target-ggvtt\" (UID: \"fc451a85-49aa-4966-b376-70bf49518b2d\") " pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:05:27.424601 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.423292 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c223b4bf-f78a-41a5-9568-d60697508aa5-run-openvswitch\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.424601 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.423406 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c223b4bf-f78a-41a5-9568-d60697508aa5-env-overrides\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.425634 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.425603 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c223b4bf-f78a-41a5-9568-d60697508aa5-ovn-node-metrics-cert\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.431602 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:27.431514 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:05:27.431602 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:27.431545 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:05:27.431602 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:27.431561 2560 projected.go:196] Error preparing data for projected volume kube-api-access-pfwxp for pod openshift-network-diagnostics/network-check-target-ggvtt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:05:27.431885 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:27.431650 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc451a85-49aa-4966-b376-70bf49518b2d-kube-api-access-pfwxp podName:fc451a85-49aa-4966-b376-70bf49518b2d nodeName:}" failed. No retries permitted until 2026-07-06 19:05:27.931630224 +0000 UTC m=+3.122997914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pfwxp" (UniqueName: "kubernetes.io/projected/fc451a85-49aa-4966-b376-70bf49518b2d-kube-api-access-pfwxp") pod "network-check-target-ggvtt" (UID: "fc451a85-49aa-4966-b376-70bf49518b2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:05:27.433081 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.433053 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nsmf\" (UniqueName: \"kubernetes.io/projected/84c43c19-1748-4d6b-8c86-b9171445c9b8-kube-api-access-9nsmf\") pod \"node-resolver-m5pz7\" (UID: \"84c43c19-1748-4d6b-8c86-b9171445c9b8\") " pod="openshift-dns/node-resolver-m5pz7" Jul 06 19:05:27.434932 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.434909 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znlb6\" (UniqueName: \"kubernetes.io/projected/59dc1490-ee33-4cc6-86ff-d6a3172dcc05-kube-api-access-znlb6\") pod \"aws-ebs-csi-driver-node-lxnbh\" (UID: \"59dc1490-ee33-4cc6-86ff-d6a3172dcc05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" Jul 06 19:05:27.435370 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.435346 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64lh6\" (UniqueName: \"kubernetes.io/projected/c223b4bf-f78a-41a5-9568-d60697508aa5-kube-api-access-64lh6\") pod \"ovnkube-node-ng9fp\" (UID: \"c223b4bf-f78a-41a5-9568-d60697508aa5\") " pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.506033 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.505983 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q9mrm" Jul 06 19:05:27.516401 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.516354 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vj7fc" Jul 06 19:05:27.527057 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.527028 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jbl9d" Jul 06 19:05:27.534799 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.534777 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z9v9z" Jul 06 19:05:27.544111 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.544058 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gghgg" Jul 06 19:05:27.552365 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.552344 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-26m4d" Jul 06 19:05:27.561409 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.561388 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m5pz7" Jul 06 19:05:27.569589 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.569557 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:27.576312 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.576294 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" Jul 06 19:05:27.629530 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.629500 2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 19:05:27.825337 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.825244 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0f8d44-df3e-4fb6-afab-15fb76427980-metrics-certs\") pod \"network-metrics-daemon-mz7xr\" (UID: \"1f0f8d44-df3e-4fb6-afab-15fb76427980\") " pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:27.825337 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:27.825286 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de165982-d4dc-4449-a40d-1fc3bf2754cc-original-pull-secret\") pod \"global-pull-secret-syncer-dl7lj\" (UID: \"de165982-d4dc-4449-a40d-1fc3bf2754cc\") " pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:27.825514 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:27.825429 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:05:27.825514 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:27.825477 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:05:27.825514 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:27.825506 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f0f8d44-df3e-4fb6-afab-15fb76427980-metrics-certs podName:1f0f8d44-df3e-4fb6-afab-15fb76427980 nodeName:}" failed. No retries permitted until 2026-07-06 19:05:28.825488106 +0000 UTC m=+4.016855760 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f0f8d44-df3e-4fb6-afab-15fb76427980-metrics-certs") pod "network-metrics-daemon-mz7xr" (UID: "1f0f8d44-df3e-4fb6-afab-15fb76427980") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:05:27.825662 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:27.825524 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de165982-d4dc-4449-a40d-1fc3bf2754cc-original-pull-secret podName:de165982-d4dc-4449-a40d-1fc3bf2754cc nodeName:}" failed. No retries permitted until 2026-07-06 19:05:28.825516953 +0000 UTC m=+4.016884607 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de165982-d4dc-4449-a40d-1fc3bf2754cc-original-pull-secret") pod "global-pull-secret-syncer-dl7lj" (UID: "de165982-d4dc-4449-a40d-1fc3bf2754cc") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:05:28.026435 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:28.026393 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfwxp\" (UniqueName: \"kubernetes.io/projected/fc451a85-49aa-4966-b376-70bf49518b2d-kube-api-access-pfwxp\") pod \"network-check-target-ggvtt\" (UID: \"fc451a85-49aa-4966-b376-70bf49518b2d\") " pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:05:28.026637 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:28.026533 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:05:28.026637 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:28.026551 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:05:28.026637 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:28.026564 2560 projected.go:196] Error preparing data for projected volume kube-api-access-pfwxp for pod openshift-network-diagnostics/network-check-target-ggvtt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:05:28.026817 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:28.026643 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc451a85-49aa-4966-b376-70bf49518b2d-kube-api-access-pfwxp podName:fc451a85-49aa-4966-b376-70bf49518b2d nodeName:}" failed. No retries permitted until 2026-07-06 19:05:29.026623552 +0000 UTC m=+4.217991218 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pfwxp" (UniqueName: "kubernetes.io/projected/fc451a85-49aa-4966-b376-70bf49518b2d-kube-api-access-pfwxp") pod "network-check-target-ggvtt" (UID: "fc451a85-49aa-4966-b376-70bf49518b2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:05:28.246257 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:28.246173 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:00:27 +0000 UTC" deadline="2027-12-01 06:53:31.133878041 +0000 UTC" Jul 06 19:05:28.246257 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:28.246205 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12299h48m2.887676868s" Jul 06 19:05:28.385376 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:28.385339 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:28.385549 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:28.385465 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mz7xr" podUID="1f0f8d44-df3e-4fb6-afab-15fb76427980" Jul 06 19:05:28.549392 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:28.549360 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61b821d_85b4_4950_af28_27ecb921cc34.slice/crio-59e5b610737240a7d74ab7e51072db5af1f046ddd324878f05d6ae600fc67880 WatchSource:0}: Error finding container 59e5b610737240a7d74ab7e51072db5af1f046ddd324878f05d6ae600fc67880: Status 404 returned error can't find the container with id 59e5b610737240a7d74ab7e51072db5af1f046ddd324878f05d6ae600fc67880 Jul 06 19:05:28.550695 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:28.550669 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59dc1490_ee33_4cc6_86ff_d6a3172dcc05.slice/crio-836603532af781ada6710b5b66f1ed467f5c09400688104e1e28e93999fa28b5 WatchSource:0}: Error finding container 836603532af781ada6710b5b66f1ed467f5c09400688104e1e28e93999fa28b5: Status 404 returned error can't find the container with id 836603532af781ada6710b5b66f1ed467f5c09400688104e1e28e93999fa28b5 Jul 06 19:05:28.552240 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:28.552228 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc223b4bf_f78a_41a5_9568_d60697508aa5.slice/crio-f32a5f253e2d52a5ba56ced60d81c9d76e05cf5ef1866aa8f08b7a2d8fddd40a WatchSource:0}: Error finding container f32a5f253e2d52a5ba56ced60d81c9d76e05cf5ef1866aa8f08b7a2d8fddd40a: Status 404 returned error can't find the container with id f32a5f253e2d52a5ba56ced60d81c9d76e05cf5ef1866aa8f08b7a2d8fddd40a Jul 06 19:05:28.554362 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:28.554299 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd7edc4_1efb_4b5e_9bf9_2e309a0a5ead.slice/crio-b3ab17634a0b395a954c227a771730ac8babd8d19bca4844180d6991d622a472 WatchSource:0}: Error finding container b3ab17634a0b395a954c227a771730ac8babd8d19bca4844180d6991d622a472: Status 404 returned error can't find the container with id b3ab17634a0b395a954c227a771730ac8babd8d19bca4844180d6991d622a472 Jul 06 19:05:28.557174 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:28.557124 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c43c19_1748_4d6b_8c86_b9171445c9b8.slice/crio-5cd840ac7453fe4ec266afcba60a5bfe2a4bd0c8d8243313df33430cb127099e WatchSource:0}: Error finding container 5cd840ac7453fe4ec266afcba60a5bfe2a4bd0c8d8243313df33430cb127099e: Status 404 returned error can't find the container with id 5cd840ac7453fe4ec266afcba60a5bfe2a4bd0c8d8243313df33430cb127099e Jul 06 19:05:28.559637 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:28.559123 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e4a6e5_96c2_4496_af5a_e162d61d9c20.slice/crio-a42203b0c340ca80eb77f49a2eb7ba2593fc483b7cd84fb935dab3bd7e1f0aff WatchSource:0}: Error finding container a42203b0c340ca80eb77f49a2eb7ba2593fc483b7cd84fb935dab3bd7e1f0aff: Status 404 returned error can't find the container with id a42203b0c340ca80eb77f49a2eb7ba2593fc483b7cd84fb935dab3bd7e1f0aff Jul 06 19:05:28.564598 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:05:28.563374 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a69bd8_7312_4728_9ac1_efc5384484bd.slice/crio-cbfb6ec8905817d9c9731ed866752a2ee36bb12163fa7d03d7b2c589ca51e15d WatchSource:0}: Error finding container cbfb6ec8905817d9c9731ed866752a2ee36bb12163fa7d03d7b2c589ca51e15d: Status 404 returned error can't find the container with id cbfb6ec8905817d9c9731ed866752a2ee36bb12163fa7d03d7b2c589ca51e15d Jul 06 19:05:28.832976 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:28.832782 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0f8d44-df3e-4fb6-afab-15fb76427980-metrics-certs\") pod \"network-metrics-daemon-mz7xr\" (UID: \"1f0f8d44-df3e-4fb6-afab-15fb76427980\") " pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:28.833074 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:28.833013 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de165982-d4dc-4449-a40d-1fc3bf2754cc-original-pull-secret\") pod \"global-pull-secret-syncer-dl7lj\" (UID: \"de165982-d4dc-4449-a40d-1fc3bf2754cc\") " pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:28.833074 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:28.832921 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:05:28.833157 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:28.833104 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f0f8d44-df3e-4fb6-afab-15fb76427980-metrics-certs podName:1f0f8d44-df3e-4fb6-afab-15fb76427980 nodeName:}" failed. No retries permitted until 2026-07-06 19:05:30.833084998 +0000 UTC m=+6.024452659 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f0f8d44-df3e-4fb6-afab-15fb76427980-metrics-certs") pod "network-metrics-daemon-mz7xr" (UID: "1f0f8d44-df3e-4fb6-afab-15fb76427980") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:05:28.833206 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:28.833163 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:05:28.833245 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:28.833222 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de165982-d4dc-4449-a40d-1fc3bf2754cc-original-pull-secret podName:de165982-d4dc-4449-a40d-1fc3bf2754cc nodeName:}" failed. No retries permitted until 2026-07-06 19:05:30.833207531 +0000 UTC m=+6.024575184 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de165982-d4dc-4449-a40d-1fc3bf2754cc-original-pull-secret") pod "global-pull-secret-syncer-dl7lj" (UID: "de165982-d4dc-4449-a40d-1fc3bf2754cc") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:05:29.034157 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:29.034115 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfwxp\" (UniqueName: \"kubernetes.io/projected/fc451a85-49aa-4966-b376-70bf49518b2d-kube-api-access-pfwxp\") pod \"network-check-target-ggvtt\" (UID: \"fc451a85-49aa-4966-b376-70bf49518b2d\") " pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:05:29.034360 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:29.034336 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:05:29.034440 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:29.034371 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:05:29.034440 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:29.034387 2560 projected.go:196] Error preparing data for projected volume kube-api-access-pfwxp for pod openshift-network-diagnostics/network-check-target-ggvtt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:05:29.034554 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:29.034454 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc451a85-49aa-4966-b376-70bf49518b2d-kube-api-access-pfwxp podName:fc451a85-49aa-4966-b376-70bf49518b2d nodeName:}" failed. No retries permitted until 2026-07-06 19:05:31.034434084 +0000 UTC m=+6.225801760 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pfwxp" (UniqueName: "kubernetes.io/projected/fc451a85-49aa-4966-b376-70bf49518b2d-kube-api-access-pfwxp") pod "network-check-target-ggvtt" (UID: "fc451a85-49aa-4966-b376-70bf49518b2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:05:29.246706 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:29.246639 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 19:00:27 +0000 UTC" deadline="2028-01-23 00:47:47.082083619 +0000 UTC" Jul 06 19:05:29.246706 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:29.246704 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13565h42m17.835383834s" Jul 06 19:05:29.387252 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:29.387118 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:29.387420 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:29.387246 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dl7lj" podUID="de165982-d4dc-4449-a40d-1fc3bf2754cc" Jul 06 19:05:29.387796 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:29.387676 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:05:29.387796 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:29.387773 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvtt" podUID="fc451a85-49aa-4966-b376-70bf49518b2d" Jul 06 19:05:29.413530 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:29.413492 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z9v9z" event={"ID":"58a69bd8-7312-4728-9ac1-efc5384484bd","Type":"ContainerStarted","Data":"cbfb6ec8905817d9c9731ed866752a2ee36bb12163fa7d03d7b2c589ca51e15d"} Jul 06 19:05:29.424896 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:29.424837 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vj7fc" event={"ID":"b6e4a6e5-96c2-4496-af5a-e162d61d9c20","Type":"ContainerStarted","Data":"a42203b0c340ca80eb77f49a2eb7ba2593fc483b7cd84fb935dab3bd7e1f0aff"} Jul 06 19:05:29.439718 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:29.439635 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q9mrm" event={"ID":"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28","Type":"ContainerStarted","Data":"f99df0e7670c55da1c98bc983ba14557891c29daca41e88e645962f688bb70ae"} Jul 06 19:05:29.450992 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:29.450948 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" event={"ID":"59dc1490-ee33-4cc6-86ff-d6a3172dcc05","Type":"ContainerStarted","Data":"836603532af781ada6710b5b66f1ed467f5c09400688104e1e28e93999fa28b5"} Jul 06 19:05:29.461721 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:29.461685 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-26m4d" event={"ID":"c61b821d-85b4-4950-af28-27ecb921cc34","Type":"ContainerStarted","Data":"59e5b610737240a7d74ab7e51072db5af1f046ddd324878f05d6ae600fc67880"} Jul 06 19:05:29.476597 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:29.476562 2560 generic.go:356] "Generic (PLEG): container finished" podID="c8611695537a81ab018c1f7cb211fa60" containerID="0ffc6b798df7e2f4286e02b80b326b2e9d9a056ecd7fe05d4bb56a3190e97fe6" exitCode=0 Jul 06 19:05:29.476798 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:29.476708 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" event={"ID":"c8611695537a81ab018c1f7cb211fa60","Type":"ContainerDied","Data":"0ffc6b798df7e2f4286e02b80b326b2e9d9a056ecd7fe05d4bb56a3190e97fe6"} Jul 06 19:05:29.480805 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:29.479903 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gghgg" event={"ID":"fe072be2-5f94-440e-b0e4-83e1e4899fc8","Type":"ContainerStarted","Data":"b4c8acbddce7170e3f6c7869b6036d5adc049863918af0a6bce1c8499adfeded"} Jul 06 19:05:29.492661 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:29.492575 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m5pz7" event={"ID":"84c43c19-1748-4d6b-8c86-b9171445c9b8","Type":"ContainerStarted","Data":"5cd840ac7453fe4ec266afcba60a5bfe2a4bd0c8d8243313df33430cb127099e"} Jul 06 19:05:29.513737 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:29.513696 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jbl9d" event={"ID":"6cd7edc4-1efb-4b5e-9bf9-2e309a0a5ead","Type":"ContainerStarted","Data":"b3ab17634a0b395a954c227a771730ac8babd8d19bca4844180d6991d622a472"} Jul 06 19:05:29.517258 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:29.517217 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" event={"ID":"c223b4bf-f78a-41a5-9568-d60697508aa5","Type":"ContainerStarted","Data":"f32a5f253e2d52a5ba56ced60d81c9d76e05cf5ef1866aa8f08b7a2d8fddd40a"} Jul 06 19:05:29.528605 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:29.528570 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal" event={"ID":"831a3fedbcc35899a7ac03738589a136","Type":"ContainerStarted","Data":"bb2fedaaba8c340b6981c5a6ee1f3dec1e108666135734af42b533b44e49f5b8"} Jul 06 19:05:29.545202 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:29.545140 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-80.ec2.internal" podStartSLOduration=2.545120168 podStartE2EDuration="2.545120168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:05:29.543931764 +0000 UTC m=+4.735299441" watchObservedRunningTime="2026-07-06 19:05:29.545120168 +0000 UTC m=+4.736487846" Jul 06 19:05:30.385466 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:30.385426 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:30.386027 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:30.385587 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mz7xr" podUID="1f0f8d44-df3e-4fb6-afab-15fb76427980" Jul 06 19:05:30.553751 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:30.553711 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" event={"ID":"c8611695537a81ab018c1f7cb211fa60","Type":"ContainerStarted","Data":"eb14fc6ae41ca67c7aeecc222437fe9820fc94e92a4de49a65da05def6dd72ad"} Jul 06 19:05:30.849397 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:30.849290 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0f8d44-df3e-4fb6-afab-15fb76427980-metrics-certs\") pod \"network-metrics-daemon-mz7xr\" (UID: \"1f0f8d44-df3e-4fb6-afab-15fb76427980\") " pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:30.849397 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:30.849358 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de165982-d4dc-4449-a40d-1fc3bf2754cc-original-pull-secret\") pod \"global-pull-secret-syncer-dl7lj\" (UID: \"de165982-d4dc-4449-a40d-1fc3bf2754cc\") " pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:30.849907 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:30.849518 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:05:30.849907 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:30.849587 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de165982-d4dc-4449-a40d-1fc3bf2754cc-original-pull-secret podName:de165982-d4dc-4449-a40d-1fc3bf2754cc nodeName:}" failed. No retries permitted until 2026-07-06 19:05:34.849567667 +0000 UTC m=+10.040935339 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de165982-d4dc-4449-a40d-1fc3bf2754cc-original-pull-secret") pod "global-pull-secret-syncer-dl7lj" (UID: "de165982-d4dc-4449-a40d-1fc3bf2754cc") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:05:30.850063 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:30.850040 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:05:30.850117 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:30.850099 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f0f8d44-df3e-4fb6-afab-15fb76427980-metrics-certs podName:1f0f8d44-df3e-4fb6-afab-15fb76427980 nodeName:}" failed. No retries permitted until 2026-07-06 19:05:34.850081552 +0000 UTC m=+10.041449208 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f0f8d44-df3e-4fb6-afab-15fb76427980-metrics-certs") pod "network-metrics-daemon-mz7xr" (UID: "1f0f8d44-df3e-4fb6-afab-15fb76427980") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:05:31.052303 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:31.051553 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfwxp\" (UniqueName: \"kubernetes.io/projected/fc451a85-49aa-4966-b376-70bf49518b2d-kube-api-access-pfwxp\") pod \"network-check-target-ggvtt\" (UID: \"fc451a85-49aa-4966-b376-70bf49518b2d\") " pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:05:31.052303 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:31.051756 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:05:31.052303 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:31.051780 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:05:31.052303 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:31.051794 2560 projected.go:196] Error preparing data for projected volume kube-api-access-pfwxp for pod openshift-network-diagnostics/network-check-target-ggvtt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:05:31.052303 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:31.051858 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc451a85-49aa-4966-b376-70bf49518b2d-kube-api-access-pfwxp podName:fc451a85-49aa-4966-b376-70bf49518b2d nodeName:}" failed. No retries permitted until 2026-07-06 19:05:35.051839209 +0000 UTC m=+10.243206866 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pfwxp" (UniqueName: "kubernetes.io/projected/fc451a85-49aa-4966-b376-70bf49518b2d-kube-api-access-pfwxp") pod "network-check-target-ggvtt" (UID: "fc451a85-49aa-4966-b376-70bf49518b2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:05:31.392918 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:31.392887 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:05:31.393406 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:31.393010 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvtt" podUID="fc451a85-49aa-4966-b376-70bf49518b2d" Jul 06 19:05:31.393474 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:31.393403 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:31.393535 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:31.393500 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dl7lj" podUID="de165982-d4dc-4449-a40d-1fc3bf2754cc" Jul 06 19:05:32.385230 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:32.385190 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:32.385428 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:32.385358 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mz7xr" podUID="1f0f8d44-df3e-4fb6-afab-15fb76427980" Jul 06 19:05:33.385999 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:33.385732 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:05:33.385999 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:33.385888 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvtt" podUID="fc451a85-49aa-4966-b376-70bf49518b2d" Jul 06 19:05:33.386681 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:33.386301 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:33.386681 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:33.386401 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dl7lj" podUID="de165982-d4dc-4449-a40d-1fc3bf2754cc" Jul 06 19:05:34.384824 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:34.384766 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:34.385016 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:34.384896 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mz7xr" podUID="1f0f8d44-df3e-4fb6-afab-15fb76427980" Jul 06 19:05:34.884872 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:34.884787 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0f8d44-df3e-4fb6-afab-15fb76427980-metrics-certs\") pod \"network-metrics-daemon-mz7xr\" (UID: \"1f0f8d44-df3e-4fb6-afab-15fb76427980\") " pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:34.884872 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:34.884851 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de165982-d4dc-4449-a40d-1fc3bf2754cc-original-pull-secret\") pod \"global-pull-secret-syncer-dl7lj\" (UID: \"de165982-d4dc-4449-a40d-1fc3bf2754cc\") " pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:34.885355 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:34.885002 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:05:34.885355 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:34.885064 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de165982-d4dc-4449-a40d-1fc3bf2754cc-original-pull-secret podName:de165982-d4dc-4449-a40d-1fc3bf2754cc nodeName:}" failed. No retries permitted until 2026-07-06 19:05:42.885043717 +0000 UTC m=+18.076411374 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de165982-d4dc-4449-a40d-1fc3bf2754cc-original-pull-secret") pod "global-pull-secret-syncer-dl7lj" (UID: "de165982-d4dc-4449-a40d-1fc3bf2754cc") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:05:34.885355 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:34.885196 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:05:34.885355 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:34.885256 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f0f8d44-df3e-4fb6-afab-15fb76427980-metrics-certs podName:1f0f8d44-df3e-4fb6-afab-15fb76427980 nodeName:}" failed. No retries permitted until 2026-07-06 19:05:42.885238127 +0000 UTC m=+18.076605780 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f0f8d44-df3e-4fb6-afab-15fb76427980-metrics-certs") pod "network-metrics-daemon-mz7xr" (UID: "1f0f8d44-df3e-4fb6-afab-15fb76427980") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:05:35.087136 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:35.087101 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfwxp\" (UniqueName: \"kubernetes.io/projected/fc451a85-49aa-4966-b376-70bf49518b2d-kube-api-access-pfwxp\") pod \"network-check-target-ggvtt\" (UID: \"fc451a85-49aa-4966-b376-70bf49518b2d\") " pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:05:35.087337 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:35.087299 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:05:35.087337 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:35.087328 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:05:35.087458 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:35.087341 2560 projected.go:196] Error preparing data for projected volume kube-api-access-pfwxp for pod openshift-network-diagnostics/network-check-target-ggvtt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:05:35.087458 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:35.087408 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc451a85-49aa-4966-b376-70bf49518b2d-kube-api-access-pfwxp podName:fc451a85-49aa-4966-b376-70bf49518b2d nodeName:}" failed. No retries permitted until 2026-07-06 19:05:43.087389403 +0000 UTC m=+18.278757061 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pfwxp" (UniqueName: "kubernetes.io/projected/fc451a85-49aa-4966-b376-70bf49518b2d-kube-api-access-pfwxp") pod "network-check-target-ggvtt" (UID: "fc451a85-49aa-4966-b376-70bf49518b2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:05:35.385919 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:35.385892 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:35.386074 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:35.385996 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dl7lj" podUID="de165982-d4dc-4449-a40d-1fc3bf2754cc" Jul 06 19:05:35.386074 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:35.386046 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:05:35.386184 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:35.386131 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvtt" podUID="fc451a85-49aa-4966-b376-70bf49518b2d" Jul 06 19:05:36.385545 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:36.385518 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:36.385987 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:36.385654 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mz7xr" podUID="1f0f8d44-df3e-4fb6-afab-15fb76427980" Jul 06 19:05:37.385120 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:37.385073 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:05:37.385289 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:37.385214 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvtt" podUID="fc451a85-49aa-4966-b376-70bf49518b2d" Jul 06 19:05:37.385364 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:37.385076 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:37.385483 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:37.385462 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dl7lj" podUID="de165982-d4dc-4449-a40d-1fc3bf2754cc" Jul 06 19:05:38.385102 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:38.385066 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:38.385560 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:38.385197 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mz7xr" podUID="1f0f8d44-df3e-4fb6-afab-15fb76427980" Jul 06 19:05:39.384931 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:39.384857 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:05:39.385055 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:39.384991 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvtt" podUID="fc451a85-49aa-4966-b376-70bf49518b2d" Jul 06 19:05:39.385055 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:39.385033 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:39.385167 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:39.385115 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dl7lj" podUID="de165982-d4dc-4449-a40d-1fc3bf2754cc" Jul 06 19:05:40.384665 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:40.384628 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:40.384853 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:40.384783 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mz7xr" podUID="1f0f8d44-df3e-4fb6-afab-15fb76427980" Jul 06 19:05:41.384782 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:41.384750 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:05:41.385250 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:41.384885 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvtt" podUID="fc451a85-49aa-4966-b376-70bf49518b2d" Jul 06 19:05:41.385250 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:41.384972 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:41.385250 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:41.385086 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dl7lj" podUID="de165982-d4dc-4449-a40d-1fc3bf2754cc" Jul 06 19:05:42.385055 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:42.385015 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:42.385521 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:42.385149 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mz7xr" podUID="1f0f8d44-df3e-4fb6-afab-15fb76427980" Jul 06 19:05:42.948160 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:42.948123 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0f8d44-df3e-4fb6-afab-15fb76427980-metrics-certs\") pod \"network-metrics-daemon-mz7xr\" (UID: \"1f0f8d44-df3e-4fb6-afab-15fb76427980\") " pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:42.948363 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:42.948177 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de165982-d4dc-4449-a40d-1fc3bf2754cc-original-pull-secret\") pod \"global-pull-secret-syncer-dl7lj\" (UID: \"de165982-d4dc-4449-a40d-1fc3bf2754cc\") " pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:42.948363 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:42.948285 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:05:42.948363 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:42.948311 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:05:42.948363 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:42.948346 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de165982-d4dc-4449-a40d-1fc3bf2754cc-original-pull-secret podName:de165982-d4dc-4449-a40d-1fc3bf2754cc nodeName:}" failed. No retries permitted until 2026-07-06 19:05:58.948327553 +0000 UTC m=+34.139695204 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de165982-d4dc-4449-a40d-1fc3bf2754cc-original-pull-secret") pod "global-pull-secret-syncer-dl7lj" (UID: "de165982-d4dc-4449-a40d-1fc3bf2754cc") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:05:42.948540 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:42.948375 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f0f8d44-df3e-4fb6-afab-15fb76427980-metrics-certs podName:1f0f8d44-df3e-4fb6-afab-15fb76427980 nodeName:}" failed. No retries permitted until 2026-07-06 19:05:58.948355161 +0000 UTC m=+34.139722818 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f0f8d44-df3e-4fb6-afab-15fb76427980-metrics-certs") pod "network-metrics-daemon-mz7xr" (UID: "1f0f8d44-df3e-4fb6-afab-15fb76427980") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:05:43.149791 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:43.149747 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfwxp\" (UniqueName: \"kubernetes.io/projected/fc451a85-49aa-4966-b376-70bf49518b2d-kube-api-access-pfwxp\") pod \"network-check-target-ggvtt\" (UID: \"fc451a85-49aa-4966-b376-70bf49518b2d\") " pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:05:43.149992 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:43.149913 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:05:43.149992 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:43.149935 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:05:43.149992 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:43.149946 2560 projected.go:196] Error preparing data for projected volume kube-api-access-pfwxp for pod openshift-network-diagnostics/network-check-target-ggvtt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:05:43.150121 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:43.149995 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc451a85-49aa-4966-b376-70bf49518b2d-kube-api-access-pfwxp podName:fc451a85-49aa-4966-b376-70bf49518b2d nodeName:}" failed. No retries permitted until 2026-07-06 19:05:59.14998184 +0000 UTC m=+34.341349496 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pfwxp" (UniqueName: "kubernetes.io/projected/fc451a85-49aa-4966-b376-70bf49518b2d-kube-api-access-pfwxp") pod "network-check-target-ggvtt" (UID: "fc451a85-49aa-4966-b376-70bf49518b2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:05:43.388340 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:43.388311 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:43.388762 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:43.388427 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dl7lj" podUID="de165982-d4dc-4449-a40d-1fc3bf2754cc" Jul 06 19:05:43.388762 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:43.388486 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:05:43.388762 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:43.388615 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvtt" podUID="fc451a85-49aa-4966-b376-70bf49518b2d" Jul 06 19:05:44.385437 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:44.385398 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:44.385629 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:44.385537 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mz7xr" podUID="1f0f8d44-df3e-4fb6-afab-15fb76427980" Jul 06 19:05:45.385463 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:45.385428 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:05:45.385463 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:45.385461 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:45.386000 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:45.385534 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvtt" podUID="fc451a85-49aa-4966-b376-70bf49518b2d" Jul 06 19:05:45.386000 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:45.385684 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dl7lj" podUID="de165982-d4dc-4449-a40d-1fc3bf2754cc" Jul 06 19:05:46.385502 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:46.385471 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:46.385910 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:46.385597 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mz7xr" podUID="1f0f8d44-df3e-4fb6-afab-15fb76427980" Jul 06 19:05:47.385717 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:47.385382 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:47.385717 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:47.385503 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dl7lj" podUID="de165982-d4dc-4449-a40d-1fc3bf2754cc" Jul 06 19:05:47.385717 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:47.385605 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:05:47.385717 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:47.385680 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvtt" podUID="fc451a85-49aa-4966-b376-70bf49518b2d" Jul 06 19:05:47.590013 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:47.589847 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" event={"ID":"59dc1490-ee33-4cc6-86ff-d6a3172dcc05","Type":"ContainerStarted","Data":"80bf20bc62eed6217878aa179485f56b6486162c79889c9494018968cf9eda32"} Jul 06 19:05:47.591638 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:47.591559 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-26m4d" event={"ID":"c61b821d-85b4-4950-af28-27ecb921cc34","Type":"ContainerStarted","Data":"be2e25369c77cd03633a09a91ad870b3a4dc71f3e496cf153cde4ac35abc47a8"} Jul 06 19:05:47.593105 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:47.593084 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gghgg" event={"ID":"fe072be2-5f94-440e-b0e4-83e1e4899fc8","Type":"ContainerStarted","Data":"45daa38a1530127b56edd4c291a18c50c65440551cce7721fd2c1c240fff15a5"} Jul 06 19:05:47.594464 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:47.594440 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m5pz7" event={"ID":"84c43c19-1748-4d6b-8c86-b9171445c9b8","Type":"ContainerStarted","Data":"ed0a05fc109d7fd7c14d489d63955c01b00bf64c08ea30de2114ea10fc404178"} Jul 06 19:05:47.595954 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:47.595934 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" event={"ID":"c223b4bf-f78a-41a5-9568-d60697508aa5","Type":"ContainerStarted","Data":"c1bb736a490726dc208334bb1da07766946213b0d090fae972a97c146e7b2133"} Jul 06 19:05:47.597294 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:47.597271 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z9v9z" event={"ID":"58a69bd8-7312-4728-9ac1-efc5384484bd","Type":"ContainerStarted","Data":"8c9f2ab008188f89a178aa397ce967b3b78dd0b508618db2712eaeebfc9075a7"} Jul 06 19:05:47.598803 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:47.598781 2560 generic.go:356] "Generic (PLEG): container finished" podID="b6e4a6e5-96c2-4496-af5a-e162d61d9c20" containerID="a53f3789b065a33df55668f69ed777c9d70caf110a20fcd93daa02f6ecb2a59e" exitCode=0 Jul 06 19:05:47.598886 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:47.598862 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vj7fc" event={"ID":"b6e4a6e5-96c2-4496-af5a-e162d61d9c20","Type":"ContainerDied","Data":"a53f3789b065a33df55668f69ed777c9d70caf110a20fcd93daa02f6ecb2a59e"} Jul 06 19:05:47.601112 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:47.601089 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q9mrm" event={"ID":"8ed9d212-0c7f-47e6-8dd9-9bbb9545dd28","Type":"ContainerStarted","Data":"3544f7682af89f739179850eac426962b7735d188d9d513e4f0a9f24650dd9f3"} Jul 06 19:05:47.608759 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:47.608720 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-80.ec2.internal" podStartSLOduration=20.608708764 podStartE2EDuration="20.608708764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:05:30.570713168 +0000 UTC m=+5.762080863" watchObservedRunningTime="2026-07-06 19:05:47.608708764 +0000 UTC m=+22.800076437" Jul 06 19:05:47.609429 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:47.609394 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-26m4d" podStartSLOduration=2.858702338 podStartE2EDuration="21.609383489s" totalImagesPullingTime="18.750681151s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:05:47.60859671 +0000 UTC m=+22.799964399" watchObservedRunningTime="2026-07-06 19:05:47.609383489 +0000 UTC m=+22.800751165" Jul 06 19:05:47.629009 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:47.628518 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-q9mrm" podStartSLOduration=3.8405008609999998 podStartE2EDuration="22.62849738s" totalImagesPullingTime="18.787996519s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:05:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:05:47.627066302 +0000 UTC m=+22.818433979" watchObservedRunningTime="2026-07-06 19:05:47.62849738 +0000 UTC m=+22.819865054" Jul 06 19:05:47.644052 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:47.643996 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gghgg" podStartSLOduration=2.9048718129999997 podStartE2EDuration="21.64397503s" totalImagesPullingTime="18.739103217s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:05:47.643221434 +0000 UTC m=+22.834589107" watchObservedRunningTime="2026-07-06 19:05:47.64397503 +0000 UTC m=+22.835342711" Jul 06 19:05:47.658481 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:47.658423 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-m5pz7" podStartSLOduration=2.916907615 podStartE2EDuration="21.65840046s" totalImagesPullingTime="18.741492845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:05:47.657383946 +0000 UTC m=+22.848751620" watchObservedRunningTime="2026-07-06 19:05:47.65840046 +0000 UTC m=+22.849768134" Jul 06 19:05:47.697049 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:47.697006 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-z9v9z" podStartSLOduration=2.96031841 podStartE2EDuration="21.696991937s" totalImagesPullingTime="18.736673527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:05:47.696917926 +0000 UTC m=+22.888285598" watchObservedRunningTime="2026-07-06 19:05:47.696991937 +0000 UTC m=+22.888359610" Jul 06 19:05:48.384912 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:48.384894 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:48.385011 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:48.384992 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mz7xr" podUID="1f0f8d44-df3e-4fb6-afab-15fb76427980" Jul 06 19:05:48.605218 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:48.605141 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jbl9d" event={"ID":"6cd7edc4-1efb-4b5e-9bf9-2e309a0a5ead","Type":"ContainerStarted","Data":"df4110d0fa77a2ab2a1d8793269222d19b13e99056b40614c037190b6d3b07d0"} Jul 06 19:05:48.608180 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:48.608148 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" event={"ID":"c223b4bf-f78a-41a5-9568-d60697508aa5","Type":"ContainerStarted","Data":"deb7c95e97dc39ce53c20bbd99aaf16351f9eddd90f6ea5f96c8d71622924f66"} Jul 06 19:05:48.608313 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:48.608216 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" event={"ID":"c223b4bf-f78a-41a5-9568-d60697508aa5","Type":"ContainerStarted","Data":"f4eb40ad859d3db9063941293eaebc097daeb18342f71b9309cc3ede4db81341"} Jul 06 19:05:48.608313 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:48.608256 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" event={"ID":"c223b4bf-f78a-41a5-9568-d60697508aa5","Type":"ContainerStarted","Data":"438194d5e7df3053536b512ff5bdbca37ea9c72f6efb546acccf28f654b7c394"} Jul 06 19:05:48.608313 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:48.608286 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" event={"ID":"c223b4bf-f78a-41a5-9568-d60697508aa5","Type":"ContainerStarted","Data":"36598d92cdbbc82e52b2fb3b1e6ae920207b91ad222fbba0918e598bd57d772c"} Jul 06 19:05:48.608469 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:48.608325 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" event={"ID":"c223b4bf-f78a-41a5-9568-d60697508aa5","Type":"ContainerStarted","Data":"9a7854347bf0d01163f508ef57a9b7c6bc231980631c72627c44e6dc22beb939"} Jul 06 19:05:48.620151 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:48.620106 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jbl9d" podStartSLOduration=4.874261758 podStartE2EDuration="23.620093342s" totalImagesPullingTime="18.745831584s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:05:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:05:48.619865004 +0000 UTC m=+23.811232677" watchObservedRunningTime="2026-07-06 19:05:48.620093342 +0000 UTC m=+23.811461015" Jul 06 19:05:48.740882 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:48.740860 2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 19:05:49.278837 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:49.278715 2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T19:05:48.740879565Z","UUID":"75c55e5e-d2f5-4107-87df-aed40ecff046","Handler":null,"Name":"","Endpoint":""} Jul 06 19:05:49.281090 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:49.281064 2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 19:05:49.281222 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:49.281111 2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 19:05:49.384962 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:49.384918 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:49.385182 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:49.384963 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:05:49.385182 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:49.385075 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dl7lj" podUID="de165982-d4dc-4449-a40d-1fc3bf2754cc" Jul 06 19:05:49.385298 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:49.385180 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvtt" podUID="fc451a85-49aa-4966-b376-70bf49518b2d" Jul 06 19:05:49.612514 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:49.612429 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" event={"ID":"59dc1490-ee33-4cc6-86ff-d6a3172dcc05","Type":"ContainerStarted","Data":"a1b499863871291edf8fdc6f6ed442503d0400c077cc74a9d44c8b96f75e688c"} Jul 06 19:05:50.101269 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:50.101188 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-z9v9z" Jul 06 19:05:50.385543 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:50.385461 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:50.385705 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:50.385585 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mz7xr" podUID="1f0f8d44-df3e-4fb6-afab-15fb76427980" Jul 06 19:05:50.617181 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:50.616976 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" event={"ID":"59dc1490-ee33-4cc6-86ff-d6a3172dcc05","Type":"ContainerStarted","Data":"063dce8d53f9c6f9d804436fb659d7ddd30dfa8e6fa1dba272e7b056432925c3"} Jul 06 19:05:50.634457 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:50.634396 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lxnbh" podStartSLOduration=3.835715798 podStartE2EDuration="24.634378185s" totalImagesPullingTime="20.798662387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:05:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:05:50.634077451 +0000 UTC m=+25.825445117" watchObservedRunningTime="2026-07-06 19:05:50.634378185 +0000 UTC m=+25.825745858" Jul 06 19:05:51.384952 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:51.384922 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:51.385138 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:51.384922 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:05:51.385138 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:51.385027 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dl7lj" podUID="de165982-d4dc-4449-a40d-1fc3bf2754cc" Jul 06 19:05:51.385138 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:51.385099 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvtt" podUID="fc451a85-49aa-4966-b376-70bf49518b2d" Jul 06 19:05:51.626649 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:51.626608 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" event={"ID":"c223b4bf-f78a-41a5-9568-d60697508aa5","Type":"ContainerStarted","Data":"68b252fcee0a4059e6cb8c4a3dbe87ab8980a32c56ff7fcea56926714bb9f81b"} Jul 06 19:05:51.722645 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:51.722611 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-z9v9z" Jul 06 19:05:51.723277 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:51.723223 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-z9v9z" Jul 06 19:05:52.384917 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:52.384887 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:52.385097 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:52.385003 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mz7xr" podUID="1f0f8d44-df3e-4fb6-afab-15fb76427980" Jul 06 19:05:52.630258 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:52.630235 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-z9v9z" Jul 06 19:05:53.384869 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:53.384795 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:05:53.385005 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:53.384921 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvtt" podUID="fc451a85-49aa-4966-b376-70bf49518b2d" Jul 06 19:05:53.385005 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:53.384970 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:53.385078 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:53.385055 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dl7lj" podUID="de165982-d4dc-4449-a40d-1fc3bf2754cc" Jul 06 19:05:53.635948 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:53.635700 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" event={"ID":"c223b4bf-f78a-41a5-9568-d60697508aa5","Type":"ContainerStarted","Data":"dcbbbf419d04f9040b74197826b9820df6ac86c17bcbf9392840e1fc64e4c43c"} Jul 06 19:05:53.665750 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:53.665705 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" podStartSLOduration=8.840037571 podStartE2EDuration="27.665692097s" totalImagesPullingTime="18.825654526s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:05:53.665442158 +0000 UTC m=+28.856809831" watchObservedRunningTime="2026-07-06 19:05:53.665692097 +0000 UTC m=+28.857059770" Jul 06 19:05:53.848862 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:53.848839 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:54.385164 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:54.385130 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:54.385357 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:54.385233 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mz7xr" podUID="1f0f8d44-df3e-4fb6-afab-15fb76427980" Jul 06 19:05:54.639096 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:54.639004 2560 generic.go:356] "Generic (PLEG): container finished" podID="b6e4a6e5-96c2-4496-af5a-e162d61d9c20" containerID="209d220bb6bcc5bb1c9fb06c6093407010ad2a147e7b9946e337619d4a2cba9f" exitCode=0 Jul 06 19:05:54.639703 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:54.639095 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vj7fc" event={"ID":"b6e4a6e5-96c2-4496-af5a-e162d61d9c20","Type":"ContainerDied","Data":"209d220bb6bcc5bb1c9fb06c6093407010ad2a147e7b9946e337619d4a2cba9f"} Jul 06 19:05:54.639703 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:54.639696 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:54.639990 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:54.639766 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:54.655273 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:54.655250 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:54.655403 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:54.655350 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:05:55.386692 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:55.386651 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:05:55.386859 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:55.386770 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:55.386859 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:55.386789 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvtt" podUID="fc451a85-49aa-4966-b376-70bf49518b2d" Jul 06 19:05:55.386859 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:55.386849 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dl7lj" podUID="de165982-d4dc-4449-a40d-1fc3bf2754cc" Jul 06 19:05:55.436870 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:55.436833 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dl7lj"] Jul 06 19:05:55.437836 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:55.437810 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mz7xr"] Jul 06 19:05:55.437945 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:55.437919 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:55.438023 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:55.438005 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mz7xr" podUID="1f0f8d44-df3e-4fb6-afab-15fb76427980" Jul 06 19:05:55.438442 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:55.438419 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ggvtt"] Jul 06 19:05:55.641293 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:55.641219 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:05:55.641680 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:55.641228 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:55.641680 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:55.641319 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvtt" podUID="fc451a85-49aa-4966-b376-70bf49518b2d" Jul 06 19:05:55.641762 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:55.641720 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dl7lj" podUID="de165982-d4dc-4449-a40d-1fc3bf2754cc" Jul 06 19:05:56.646191 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:56.646004 2560 generic.go:356] "Generic (PLEG): container finished" podID="b6e4a6e5-96c2-4496-af5a-e162d61d9c20" containerID="081e4746d211b3d56568e72ec3bdab82ccaf1fe3e25b67573e9c3f6c206dc061" exitCode=0 Jul 06 19:05:56.646191 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:56.646080 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vj7fc" event={"ID":"b6e4a6e5-96c2-4496-af5a-e162d61d9c20","Type":"ContainerDied","Data":"081e4746d211b3d56568e72ec3bdab82ccaf1fe3e25b67573e9c3f6c206dc061"} Jul 06 19:05:57.385650 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:57.385550 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:05:57.385650 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:57.385608 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:57.385867 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:57.385550 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:57.385867 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:57.385681 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvtt" podUID="fc451a85-49aa-4966-b376-70bf49518b2d" Jul 06 19:05:57.385867 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:57.385773 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mz7xr" podUID="1f0f8d44-df3e-4fb6-afab-15fb76427980" Jul 06 19:05:57.385867 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:57.385847 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dl7lj" podUID="de165982-d4dc-4449-a40d-1fc3bf2754cc" Jul 06 19:05:58.652906 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:58.652875 2560 generic.go:356] "Generic (PLEG): container finished" podID="b6e4a6e5-96c2-4496-af5a-e162d61d9c20" containerID="aa9565f5bf980b5aa970be792c271bd5d5a095e6aff0a586b884f61405380078" exitCode=0 Jul 06 19:05:58.653519 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:58.652924 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vj7fc" event={"ID":"b6e4a6e5-96c2-4496-af5a-e162d61d9c20","Type":"ContainerDied","Data":"aa9565f5bf980b5aa970be792c271bd5d5a095e6aff0a586b884f61405380078"} Jul 06 19:05:58.859075 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:58.859048 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m5pz7_84c43c19-1748-4d6b-8c86-b9171445c9b8/dns-node-resolver/0.log" Jul 06 19:05:58.970117 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:58.970086 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0f8d44-df3e-4fb6-afab-15fb76427980-metrics-certs\") pod \"network-metrics-daemon-mz7xr\" (UID: \"1f0f8d44-df3e-4fb6-afab-15fb76427980\") " pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:58.970412 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:58.970125 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de165982-d4dc-4449-a40d-1fc3bf2754cc-original-pull-secret\") pod \"global-pull-secret-syncer-dl7lj\" (UID: \"de165982-d4dc-4449-a40d-1fc3bf2754cc\") " pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:58.970412 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:58.970217 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:05:58.970412 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:58.970230 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 19:05:58.970412 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:58.970272 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1f0f8d44-df3e-4fb6-afab-15fb76427980-metrics-certs podName:1f0f8d44-df3e-4fb6-afab-15fb76427980 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:30.970257356 +0000 UTC m=+66.161625011 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1f0f8d44-df3e-4fb6-afab-15fb76427980-metrics-certs") pod "network-metrics-daemon-mz7xr" (UID: "1f0f8d44-df3e-4fb6-afab-15fb76427980") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 19:05:58.970412 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:58.970285 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de165982-d4dc-4449-a40d-1fc3bf2754cc-original-pull-secret podName:de165982-d4dc-4449-a40d-1fc3bf2754cc nodeName:}" failed. No retries permitted until 2026-07-06 19:06:30.970278775 +0000 UTC m=+66.161646429 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de165982-d4dc-4449-a40d-1fc3bf2754cc-original-pull-secret") pod "global-pull-secret-syncer-dl7lj" (UID: "de165982-d4dc-4449-a40d-1fc3bf2754cc") : object "kube-system"/"original-pull-secret" not registered Jul 06 19:05:59.171399 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:59.171368 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfwxp\" (UniqueName: \"kubernetes.io/projected/fc451a85-49aa-4966-b376-70bf49518b2d-kube-api-access-pfwxp\") pod \"network-check-target-ggvtt\" (UID: \"fc451a85-49aa-4966-b376-70bf49518b2d\") " pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:05:59.171563 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:59.171550 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 19:05:59.171629 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:59.171568 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 19:05:59.171629 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:59.171598 2560 projected.go:196] Error preparing data for projected volume kube-api-access-pfwxp for pod openshift-network-diagnostics/network-check-target-ggvtt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:05:59.171712 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:59.171658 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc451a85-49aa-4966-b376-70bf49518b2d-kube-api-access-pfwxp podName:fc451a85-49aa-4966-b376-70bf49518b2d nodeName:}" failed. No retries permitted until 2026-07-06 19:06:31.171637996 +0000 UTC m=+66.363005655 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pfwxp" (UniqueName: "kubernetes.io/projected/fc451a85-49aa-4966-b376-70bf49518b2d-kube-api-access-pfwxp") pod "network-check-target-ggvtt" (UID: "fc451a85-49aa-4966-b376-70bf49518b2d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 19:05:59.385618 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:59.385516 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:05:59.385618 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:59.385552 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:05:59.385831 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:59.385516 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:05:59.385831 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:59.385659 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ggvtt" podUID="fc451a85-49aa-4966-b376-70bf49518b2d" Jul 06 19:05:59.385831 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:59.385758 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mz7xr" podUID="1f0f8d44-df3e-4fb6-afab-15fb76427980" Jul 06 19:05:59.385965 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:05:59.385847 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dl7lj" podUID="de165982-d4dc-4449-a40d-1fc3bf2754cc" Jul 06 19:05:59.439459 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:05:59.439433 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gghgg_fe072be2-5f94-440e-b0e4-83e1e4899fc8/node-ca/0.log" Jul 06 19:06:01.096686 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.096602 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-80.ec2.internal" event="NodeReady" Jul 06 19:06:01.097164 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.096761 2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 19:06:01.131452 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.131421 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-cbd47fd7c-4c5fz"] Jul 06 19:06:01.133944 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.133920 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:01.137087 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.137058 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 19:06:01.137552 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.137325 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 19:06:01.137552 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.137406 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 19:06:01.137786 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.137767 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dj2cn\"" Jul 06 19:06:01.143123 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.143102 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kr28m"] Jul 06 19:06:01.144296 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.144276 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 19:06:01.145406 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.145386 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kr28m" Jul 06 19:06:01.145542 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.145524 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-24gdx"] Jul 06 19:06:01.147856 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.147835 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-24gdx" Jul 06 19:06:01.148475 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.148445 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 19:06:01.148639 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.148596 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 19:06:01.148639 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.148616 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6qj8k\"" Jul 06 19:06:01.149299 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.149261 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 19:06:01.150466 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.150419 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cbd47fd7c-4c5fz"] Jul 06 19:06:01.151911 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.151812 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 19:06:01.152029 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.152007 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 19:06:01.152123 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.152070 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 19:06:01.152123 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.152109 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kgwxd\"" Jul 06 19:06:01.152354 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.152339 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 19:06:01.156959 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.156937 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kr28m"] Jul 06 19:06:01.166948 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.166925 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-72dgz"] Jul 06 19:06:01.168932 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.168916 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-72dgz" Jul 06 19:06:01.172510 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.172491 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 19:06:01.172624 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.172530 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mb8mb\"" Jul 06 19:06:01.172867 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.172851 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 19:06:01.175092 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.175072 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-24gdx"] Jul 06 19:06:01.180837 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.180812 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-72dgz"] Jul 06 19:06:01.286072 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.286037 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svsbn\" (UniqueName: \"kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-kube-api-access-svsbn\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:01.286072 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.286076 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9fe8873-82b2-4f13-82e3-d78b5f2d77b5-cert\") pod \"ingress-canary-kr28m\" (UID: \"a9fe8873-82b2-4f13-82e3-d78b5f2d77b5\") " pod="openshift-ingress-canary/ingress-canary-kr28m" Jul 06 19:06:01.286302 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.286101 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24gdx\" (UID: \"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6\") " pod="openshift-insights/insights-runtime-extractor-24gdx" Jul 06 19:06:01.286302 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.286141 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-registry-tls\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:01.286302 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.286165 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4b30d09-8527-4555-9df9-2b7fdb5918e2-ca-trust-extracted\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:01.286302 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.286224 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c23cd74-f515-4362-8f23-724a60a71d92-tmp-dir\") pod \"dns-default-72dgz\" (UID: \"5c23cd74-f515-4362-8f23-724a60a71d92\") " pod="openshift-dns/dns-default-72dgz" Jul 06 19:06:01.286302 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.286266 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4b30d09-8527-4555-9df9-2b7fdb5918e2-registry-certificates\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:01.286503 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.286329 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4b30d09-8527-4555-9df9-2b7fdb5918e2-installation-pull-secrets\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:01.286503 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.286373 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24gdx\" (UID: \"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6\") " pod="openshift-insights/insights-runtime-extractor-24gdx" Jul 06 19:06:01.286503 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.286401 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c23cd74-f515-4362-8f23-724a60a71d92-metrics-tls\") pod \"dns-default-72dgz\" (UID: \"5c23cd74-f515-4362-8f23-724a60a71d92\") " pod="openshift-dns/dns-default-72dgz" Jul 06 19:06:01.286503 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.286444 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf7jh\" (UniqueName: \"kubernetes.io/projected/5c23cd74-f515-4362-8f23-724a60a71d92-kube-api-access-cf7jh\") pod \"dns-default-72dgz\" (UID: \"5c23cd74-f515-4362-8f23-724a60a71d92\") " pod="openshift-dns/dns-default-72dgz" Jul 06 19:06:01.286503 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.286480 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-bound-sa-token\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:01.286723 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.286500 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgzch\" (UniqueName: \"kubernetes.io/projected/a9fe8873-82b2-4f13-82e3-d78b5f2d77b5-kube-api-access-wgzch\") pod \"ingress-canary-kr28m\" (UID: \"a9fe8873-82b2-4f13-82e3-d78b5f2d77b5\") " pod="openshift-ingress-canary/ingress-canary-kr28m" Jul 06 19:06:01.286723 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.286527 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-data-volume\") pod \"insights-runtime-extractor-24gdx\" (UID: \"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6\") " pod="openshift-insights/insights-runtime-extractor-24gdx" Jul 06 19:06:01.286723 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.286554 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b944m\" (UniqueName: \"kubernetes.io/projected/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-kube-api-access-b944m\") pod \"insights-runtime-extractor-24gdx\" (UID: \"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6\") " pod="openshift-insights/insights-runtime-extractor-24gdx" Jul 06 19:06:01.286723 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.286623 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4b30d09-8527-4555-9df9-2b7fdb5918e2-trusted-ca\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:01.286723 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.286641 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-crio-socket\") pod \"insights-runtime-extractor-24gdx\" (UID: \"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6\") " pod="openshift-insights/insights-runtime-extractor-24gdx" Jul 06 19:06:01.286723 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.286675 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5c23cd74-f515-4362-8f23-724a60a71d92-config-volume\") pod \"dns-default-72dgz\" (UID: \"5c23cd74-f515-4362-8f23-724a60a71d92\") " pod="openshift-dns/dns-default-72dgz" Jul 06 19:06:01.286723 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.286697 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4b30d09-8527-4555-9df9-2b7fdb5918e2-image-registry-private-configuration\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:01.385251 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.385168 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:06:01.385419 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.385174 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:06:01.385419 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.385305 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:06:01.387397 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.387371 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-data-volume\") pod \"insights-runtime-extractor-24gdx\" (UID: \"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6\") " pod="openshift-insights/insights-runtime-extractor-24gdx" Jul 06 19:06:01.387539 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.387410 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b944m\" (UniqueName: \"kubernetes.io/projected/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-kube-api-access-b944m\") pod \"insights-runtime-extractor-24gdx\" (UID: \"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6\") " pod="openshift-insights/insights-runtime-extractor-24gdx" Jul 06 19:06:01.387539 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.387454 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4b30d09-8527-4555-9df9-2b7fdb5918e2-trusted-ca\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:01.387539 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.387478 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-crio-socket\") pod \"insights-runtime-extractor-24gdx\" (UID: \"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6\") " pod="openshift-insights/insights-runtime-extractor-24gdx" Jul 06 19:06:01.387539 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.387515 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5c23cd74-f515-4362-8f23-724a60a71d92-config-volume\") pod \"dns-default-72dgz\" (UID: \"5c23cd74-f515-4362-8f23-724a60a71d92\") " pod="openshift-dns/dns-default-72dgz" Jul 06 19:06:01.387539 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.387538 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4b30d09-8527-4555-9df9-2b7fdb5918e2-image-registry-private-configuration\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:01.387804 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.387566 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svsbn\" (UniqueName: \"kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-kube-api-access-svsbn\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:01.387804 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.387611 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9fe8873-82b2-4f13-82e3-d78b5f2d77b5-cert\") pod \"ingress-canary-kr28m\" (UID: \"a9fe8873-82b2-4f13-82e3-d78b5f2d77b5\") " pod="openshift-ingress-canary/ingress-canary-kr28m" Jul 06 19:06:01.387804 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.387636 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24gdx\" (UID: \"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6\") " pod="openshift-insights/insights-runtime-extractor-24gdx" Jul 06 19:06:01.387804 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.387680 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-registry-tls\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:01.387804 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.387704 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4b30d09-8527-4555-9df9-2b7fdb5918e2-ca-trust-extracted\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:01.387804 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.387732 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c23cd74-f515-4362-8f23-724a60a71d92-tmp-dir\") pod \"dns-default-72dgz\" (UID: \"5c23cd74-f515-4362-8f23-724a60a71d92\") " pod="openshift-dns/dns-default-72dgz" Jul 06 19:06:01.387804 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.387751 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-data-volume\") pod \"insights-runtime-extractor-24gdx\" (UID: \"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6\") " pod="openshift-insights/insights-runtime-extractor-24gdx" Jul 06 19:06:01.387804 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.387758 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4b30d09-8527-4555-9df9-2b7fdb5918e2-registry-certificates\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:01.388164 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.387812 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4b30d09-8527-4555-9df9-2b7fdb5918e2-installation-pull-secrets\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:01.388164 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:01.387925 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:06:01.388164 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:01.387943 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cbd47fd7c-4c5fz: secret "image-registry-tls" not found Jul 06 19:06:01.388164 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:01.388007 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-registry-tls podName:a4b30d09-8527-4555-9df9-2b7fdb5918e2 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:01.887984636 +0000 UTC m=+37.079352302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-registry-tls") pod "image-registry-cbd47fd7c-4c5fz" (UID: "a4b30d09-8527-4555-9df9-2b7fdb5918e2") : secret "image-registry-tls" not found Jul 06 19:06:01.388354 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.388251 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4b30d09-8527-4555-9df9-2b7fdb5918e2-ca-trust-extracted\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:01.388354 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.388306 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24gdx\" (UID: \"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6\") " pod="openshift-insights/insights-runtime-extractor-24gdx" Jul 06 19:06:01.388354 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.388347 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c23cd74-f515-4362-8f23-724a60a71d92-metrics-tls\") pod \"dns-default-72dgz\" (UID: \"5c23cd74-f515-4362-8f23-724a60a71d92\") " pod="openshift-dns/dns-default-72dgz" Jul 06 19:06:01.388495 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.388407 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf7jh\" (UniqueName: \"kubernetes.io/projected/5c23cd74-f515-4362-8f23-724a60a71d92-kube-api-access-cf7jh\") pod \"dns-default-72dgz\" (UID: \"5c23cd74-f515-4362-8f23-724a60a71d92\") " pod="openshift-dns/dns-default-72dgz" Jul 06 19:06:01.388495 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.388436 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-bound-sa-token\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:01.388495 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.388471 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgzch\" (UniqueName: \"kubernetes.io/projected/a9fe8873-82b2-4f13-82e3-d78b5f2d77b5-kube-api-access-wgzch\") pod \"ingress-canary-kr28m\" (UID: \"a9fe8873-82b2-4f13-82e3-d78b5f2d77b5\") " pod="openshift-ingress-canary/ingress-canary-kr28m" Jul 06 19:06:01.388666 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.388636 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4b30d09-8527-4555-9df9-2b7fdb5918e2-registry-certificates\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:01.388666 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.388644 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:06:01.388769 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.388694 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:06:01.388881 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.388833 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:06:01.388952 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.388915 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c23cd74-f515-4362-8f23-724a60a71d92-tmp-dir\") pod \"dns-default-72dgz\" (UID: \"5c23cd74-f515-4362-8f23-724a60a71d92\") " pod="openshift-dns/dns-default-72dgz" Jul 06 19:06:01.389011 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.388969 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lqvqq\"" Jul 06 19:06:01.389011 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:01.388991 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:06:01.389113 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:01.389043 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-insights-runtime-extractor-tls podName:d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:01.889027983 +0000 UTC m=+37.080395636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-24gdx" (UID: "d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6") : secret "insights-runtime-extractor-tls" not found Jul 06 19:06:01.389113 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:01.389060 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:06:01.389113 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:01.389098 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:06:01.389258 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:01.389133 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c23cd74-f515-4362-8f23-724a60a71d92-metrics-tls podName:5c23cd74-f515-4362-8f23-724a60a71d92 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:01.889119318 +0000 UTC m=+37.080486970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c23cd74-f515-4362-8f23-724a60a71d92-metrics-tls") pod "dns-default-72dgz" (UID: "5c23cd74-f515-4362-8f23-724a60a71d92") : secret "dns-default-metrics-tls" not found Jul 06 19:06:01.389258 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.389132 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:06:01.389258 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:01.389177 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-kube-rbac-proxy-cm podName:d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:01.889160166 +0000 UTC m=+37.080527839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-24gdx" (UID: "d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6") : configmap "kube-rbac-proxy" not found Jul 06 19:06:01.389412 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.389373 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4b30d09-8527-4555-9df9-2b7fdb5918e2-trusted-ca\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:01.389741 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.389560 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-crio-socket\") pod \"insights-runtime-extractor-24gdx\" (UID: \"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6\") " pod="openshift-insights/insights-runtime-extractor-24gdx" Jul 06 19:06:01.389847 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.389753 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5c23cd74-f515-4362-8f23-724a60a71d92-config-volume\") pod \"dns-default-72dgz\" (UID: \"5c23cd74-f515-4362-8f23-724a60a71d92\") " pod="openshift-dns/dns-default-72dgz" Jul 06 19:06:01.389917 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.389848 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5h8dt\"" Jul 06 19:06:01.394243 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.394220 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4b30d09-8527-4555-9df9-2b7fdb5918e2-installation-pull-secrets\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:01.394243 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.394233 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9fe8873-82b2-4f13-82e3-d78b5f2d77b5-cert\") pod \"ingress-canary-kr28m\" (UID: \"a9fe8873-82b2-4f13-82e3-d78b5f2d77b5\") " pod="openshift-ingress-canary/ingress-canary-kr28m" Jul 06 19:06:01.394406 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.394220 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4b30d09-8527-4555-9df9-2b7fdb5918e2-image-registry-private-configuration\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:01.399773 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.398080 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b944m\" (UniqueName: \"kubernetes.io/projected/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-kube-api-access-b944m\") pod \"insights-runtime-extractor-24gdx\" (UID: \"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6\") " pod="openshift-insights/insights-runtime-extractor-24gdx" Jul 06 19:06:01.399773 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.399024 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgzch\" (UniqueName: \"kubernetes.io/projected/a9fe8873-82b2-4f13-82e3-d78b5f2d77b5-kube-api-access-wgzch\") pod \"ingress-canary-kr28m\" (UID: \"a9fe8873-82b2-4f13-82e3-d78b5f2d77b5\") " pod="openshift-ingress-canary/ingress-canary-kr28m" Jul 06 19:06:01.399773 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.399099 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-bound-sa-token\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:01.400019 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.399841 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf7jh\" (UniqueName: \"kubernetes.io/projected/5c23cd74-f515-4362-8f23-724a60a71d92-kube-api-access-cf7jh\") pod \"dns-default-72dgz\" (UID: \"5c23cd74-f515-4362-8f23-724a60a71d92\") " pod="openshift-dns/dns-default-72dgz" Jul 06 19:06:01.400076 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.400014 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svsbn\" (UniqueName: \"kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-kube-api-access-svsbn\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:01.458044 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.458009 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kr28m" Jul 06 19:06:01.893661 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.893617 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24gdx\" (UID: \"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6\") " pod="openshift-insights/insights-runtime-extractor-24gdx" Jul 06 19:06:01.893838 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.893689 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-registry-tls\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:01.893838 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.893747 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24gdx\" (UID: \"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6\") " pod="openshift-insights/insights-runtime-extractor-24gdx" Jul 06 19:06:01.893838 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:01.893753 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:06:01.893838 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:01.893839 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-kube-rbac-proxy-cm podName:d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:02.893818985 +0000 UTC m=+38.085186644 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-24gdx" (UID: "d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6") : configmap "kube-rbac-proxy" not found Jul 06 19:06:01.894045 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:01.893883 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:06:01.894045 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:01.893910 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cbd47fd7c-4c5fz: secret "image-registry-tls" not found Jul 06 19:06:01.894045 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:01.893978 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-registry-tls podName:a4b30d09-8527-4555-9df9-2b7fdb5918e2 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:02.893961929 +0000 UTC m=+38.085329584 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-registry-tls") pod "image-registry-cbd47fd7c-4c5fz" (UID: "a4b30d09-8527-4555-9df9-2b7fdb5918e2") : secret "image-registry-tls" not found Jul 06 19:06:01.894045 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:01.893886 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c23cd74-f515-4362-8f23-724a60a71d92-metrics-tls\") pod \"dns-default-72dgz\" (UID: \"5c23cd74-f515-4362-8f23-724a60a71d92\") " pod="openshift-dns/dns-default-72dgz" Jul 06 19:06:01.894231 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:01.893985 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:06:01.894231 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:01.894080 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c23cd74-f515-4362-8f23-724a60a71d92-metrics-tls podName:5c23cd74-f515-4362-8f23-724a60a71d92 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:02.894073412 +0000 UTC m=+38.085441063 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c23cd74-f515-4362-8f23-724a60a71d92-metrics-tls") pod "dns-default-72dgz" (UID: "5c23cd74-f515-4362-8f23-724a60a71d92") : secret "dns-default-metrics-tls" not found Jul 06 19:06:01.894231 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:01.894022 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:06:01.894231 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:01.894110 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-insights-runtime-extractor-tls podName:d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:02.894100669 +0000 UTC m=+38.085468321 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-24gdx" (UID: "d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6") : secret "insights-runtime-extractor-tls" not found Jul 06 19:06:02.696704 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:02.696372 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kr28m"] Jul 06 19:06:02.699847 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:06:02.699821 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9fe8873_82b2_4f13_82e3_d78b5f2d77b5.slice/crio-25b8be242f7d8b5541c16cfef8e26c38b277d441aff2dfcac99021539acb373e WatchSource:0}: Error finding container 25b8be242f7d8b5541c16cfef8e26c38b277d441aff2dfcac99021539acb373e: Status 404 returned error can't find the container with id 25b8be242f7d8b5541c16cfef8e26c38b277d441aff2dfcac99021539acb373e Jul 06 19:06:02.901634 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:02.901509 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24gdx\" (UID: \"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6\") " pod="openshift-insights/insights-runtime-extractor-24gdx" Jul 06 19:06:02.901634 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:02.901565 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c23cd74-f515-4362-8f23-724a60a71d92-metrics-tls\") pod \"dns-default-72dgz\" (UID: \"5c23cd74-f515-4362-8f23-724a60a71d92\") " pod="openshift-dns/dns-default-72dgz" Jul 06 19:06:02.901813 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:02.901659 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24gdx\" (UID: \"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6\") " pod="openshift-insights/insights-runtime-extractor-24gdx" Jul 06 19:06:02.901813 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:02.901670 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:06:02.901813 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:02.901704 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-registry-tls\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:02.901813 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:02.901731 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-insights-runtime-extractor-tls podName:d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:04.901715151 +0000 UTC m=+40.093082807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-24gdx" (UID: "d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6") : secret "insights-runtime-extractor-tls" not found Jul 06 19:06:02.901813 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:02.901751 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:06:02.901813 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:02.901778 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:06:02.901813 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:02.901790 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cbd47fd7c-4c5fz: secret "image-registry-tls" not found Jul 06 19:06:02.901813 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:02.901799 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:06:02.902099 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:02.901805 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c23cd74-f515-4362-8f23-724a60a71d92-metrics-tls podName:5c23cd74-f515-4362-8f23-724a60a71d92 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:04.901789263 +0000 UTC m=+40.093156929 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c23cd74-f515-4362-8f23-724a60a71d92-metrics-tls") pod "dns-default-72dgz" (UID: "5c23cd74-f515-4362-8f23-724a60a71d92") : secret "dns-default-metrics-tls" not found Jul 06 19:06:02.902099 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:02.901895 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-registry-tls podName:a4b30d09-8527-4555-9df9-2b7fdb5918e2 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:04.901866426 +0000 UTC m=+40.093234084 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-registry-tls") pod "image-registry-cbd47fd7c-4c5fz" (UID: "a4b30d09-8527-4555-9df9-2b7fdb5918e2") : secret "image-registry-tls" not found Jul 06 19:06:02.902099 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:02.901925 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-kube-rbac-proxy-cm podName:d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:04.901912867 +0000 UTC m=+40.093280534 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-24gdx" (UID: "d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6") : configmap "kube-rbac-proxy" not found Jul 06 19:06:03.670261 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:03.670224 2560 generic.go:356] "Generic (PLEG): container finished" podID="b6e4a6e5-96c2-4496-af5a-e162d61d9c20" containerID="6f8480d7c478b974d0a872d6975ce8b33626ae173f3a605a919132adb469729a" exitCode=0 Jul 06 19:06:03.670458 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:03.670264 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vj7fc" event={"ID":"b6e4a6e5-96c2-4496-af5a-e162d61d9c20","Type":"ContainerDied","Data":"6f8480d7c478b974d0a872d6975ce8b33626ae173f3a605a919132adb469729a"} Jul 06 19:06:03.671782 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:03.671675 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kr28m" event={"ID":"a9fe8873-82b2-4f13-82e3-d78b5f2d77b5","Type":"ContainerStarted","Data":"25b8be242f7d8b5541c16cfef8e26c38b277d441aff2dfcac99021539acb373e"} Jul 06 19:06:04.678343 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:04.678145 2560 generic.go:356] "Generic (PLEG): container finished" podID="b6e4a6e5-96c2-4496-af5a-e162d61d9c20" containerID="5ece2699ab020a7cff2903ff275bdd4e6dd2f0c5f16ce8fa24031a7b07ba7c04" exitCode=0 Jul 06 19:06:04.678343 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:04.678203 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vj7fc" event={"ID":"b6e4a6e5-96c2-4496-af5a-e162d61d9c20","Type":"ContainerDied","Data":"5ece2699ab020a7cff2903ff275bdd4e6dd2f0c5f16ce8fa24031a7b07ba7c04"} Jul 06 19:06:04.920702 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:04.920668 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24gdx\" (UID: \"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6\") " pod="openshift-insights/insights-runtime-extractor-24gdx" Jul 06 19:06:04.920911 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:04.920716 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c23cd74-f515-4362-8f23-724a60a71d92-metrics-tls\") pod \"dns-default-72dgz\" (UID: \"5c23cd74-f515-4362-8f23-724a60a71d92\") " pod="openshift-dns/dns-default-72dgz" Jul 06 19:06:04.920911 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:04.920792 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24gdx\" (UID: \"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6\") " pod="openshift-insights/insights-runtime-extractor-24gdx" Jul 06 19:06:04.920911 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:04.920835 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-registry-tls\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:04.920911 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:04.920835 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:06:04.921112 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:04.920868 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:06:04.921112 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:04.920948 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 19:06:04.921112 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:04.920967 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cbd47fd7c-4c5fz: secret "image-registry-tls" not found Jul 06 19:06:04.921112 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:04.920869 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 19:06:04.921112 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:04.920947 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-insights-runtime-extractor-tls podName:d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:08.920928523 +0000 UTC m=+44.112296192 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-24gdx" (UID: "d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6") : secret "insights-runtime-extractor-tls" not found Jul 06 19:06:04.921112 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:04.921041 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-kube-rbac-proxy-cm podName:d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:08.921022669 +0000 UTC m=+44.112390325 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-24gdx" (UID: "d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6") : configmap "kube-rbac-proxy" not found Jul 06 19:06:04.921112 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:04.921060 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-registry-tls podName:a4b30d09-8527-4555-9df9-2b7fdb5918e2 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:08.921049323 +0000 UTC m=+44.112416974 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-registry-tls") pod "image-registry-cbd47fd7c-4c5fz" (UID: "a4b30d09-8527-4555-9df9-2b7fdb5918e2") : secret "image-registry-tls" not found Jul 06 19:06:04.921112 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:04.921074 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c23cd74-f515-4362-8f23-724a60a71d92-metrics-tls podName:5c23cd74-f515-4362-8f23-724a60a71d92 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:08.921065858 +0000 UTC m=+44.112433513 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5c23cd74-f515-4362-8f23-724a60a71d92-metrics-tls") pod "dns-default-72dgz" (UID: "5c23cd74-f515-4362-8f23-724a60a71d92") : secret "dns-default-metrics-tls" not found Jul 06 19:06:05.682484 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:05.682434 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kr28m" event={"ID":"a9fe8873-82b2-4f13-82e3-d78b5f2d77b5","Type":"ContainerStarted","Data":"c841e991974b606f2ef225f8dca804c33e403a8d0dcadf8b123d3ecd554869d7"} Jul 06 19:06:05.685781 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:05.685711 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vj7fc" event={"ID":"b6e4a6e5-96c2-4496-af5a-e162d61d9c20","Type":"ContainerStarted","Data":"bed071f7fe0ba7ab3b044c9352f046f0f8e156d15536fa0216a61e73cd4df3d9"} Jul 06 19:06:05.727813 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:05.727770 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kr28m" podStartSLOduration=1.992303185 podStartE2EDuration="4.727756031s" totalImagesPullingTime="2.735452846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:06:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:05.703869894 +0000 UTC m=+40.895237567" watchObservedRunningTime="2026-07-06 19:06:05.727756031 +0000 UTC m=+40.919123716" Jul 06 19:06:05.727971 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:05.727892 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vj7fc" podStartSLOduration=9.792047294 podStartE2EDuration="40.727886972s" totalImagesPullingTime="30.935839678s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:05:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:05.726238994 +0000 UTC m=+40.917606667" watchObservedRunningTime="2026-07-06 19:06:05.727886972 +0000 UTC m=+40.919254644" Jul 06 19:06:08.952972 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:08.952930 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24gdx\" (UID: \"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6\") " pod="openshift-insights/insights-runtime-extractor-24gdx" Jul 06 19:06:08.952972 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:08.952975 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c23cd74-f515-4362-8f23-724a60a71d92-metrics-tls\") pod \"dns-default-72dgz\" (UID: \"5c23cd74-f515-4362-8f23-724a60a71d92\") " pod="openshift-dns/dns-default-72dgz" Jul 06 19:06:08.953365 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:08.953020 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24gdx\" (UID: \"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6\") " pod="openshift-insights/insights-runtime-extractor-24gdx" Jul 06 19:06:08.953365 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:08.953048 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-registry-tls\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:08.953365 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:08.953100 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 19:06:08.953365 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:08.953148 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 19:06:08.953365 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:08.953174 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-insights-runtime-extractor-tls podName:d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:16.953156555 +0000 UTC m=+52.144524217 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-24gdx" (UID: "d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6") : secret "insights-runtime-extractor-tls" not found Jul 06 19:06:08.953365 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:08.953204 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-kube-rbac-proxy-cm podName:d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6 nodeName:}" failed. No retries permitted until 2026-07-06 19:06:16.953188634 +0000 UTC m=+52.144556307 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-24gdx" (UID: "d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6") : configmap "kube-rbac-proxy" not found Jul 06 19:06:08.956668 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:08.956639 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5c23cd74-f515-4362-8f23-724a60a71d92-metrics-tls\") pod \"dns-default-72dgz\" (UID: \"5c23cd74-f515-4362-8f23-724a60a71d92\") " pod="openshift-dns/dns-default-72dgz" Jul 06 19:06:08.956786 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:08.956684 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-registry-tls\") pod \"image-registry-cbd47fd7c-4c5fz\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:08.979222 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:08.979198 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-72dgz" Jul 06 19:06:09.150020 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:09.149984 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-72dgz"] Jul 06 19:06:09.152944 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:06:09.152918 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c23cd74_f515_4362_8f23_724a60a71d92.slice/crio-4d1b5820a9a484972108c287641faa84688b69bcf3b38f10c490cf689dd7d87a WatchSource:0}: Error finding container 4d1b5820a9a484972108c287641faa84688b69bcf3b38f10c490cf689dd7d87a: Status 404 returned error can't find the container with id 4d1b5820a9a484972108c287641faa84688b69bcf3b38f10c490cf689dd7d87a Jul 06 19:06:09.244911 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:09.244816 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:09.389089 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:09.389068 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cbd47fd7c-4c5fz"] Jul 06 19:06:09.390967 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:06:09.390946 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b30d09_8527_4555_9df9_2b7fdb5918e2.slice/crio-ee355b6891409479250891b79b8c60c4aca7e63d4e076d8a7316691f3fd3e0bb WatchSource:0}: Error finding container ee355b6891409479250891b79b8c60c4aca7e63d4e076d8a7316691f3fd3e0bb: Status 404 returned error can't find the container with id ee355b6891409479250891b79b8c60c4aca7e63d4e076d8a7316691f3fd3e0bb Jul 06 19:06:09.697268 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:09.697235 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-72dgz" event={"ID":"5c23cd74-f515-4362-8f23-724a60a71d92","Type":"ContainerStarted","Data":"4d1b5820a9a484972108c287641faa84688b69bcf3b38f10c490cf689dd7d87a"} Jul 06 19:06:09.698788 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:09.698760 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" event={"ID":"a4b30d09-8527-4555-9df9-2b7fdb5918e2","Type":"ContainerStarted","Data":"e7e880194b65623f2319553208e2c8a536fdd8093bedb8bad72b2bc0d3a3dccf"} Jul 06 19:06:09.698883 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:09.698813 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" event={"ID":"a4b30d09-8527-4555-9df9-2b7fdb5918e2","Type":"ContainerStarted","Data":"ee355b6891409479250891b79b8c60c4aca7e63d4e076d8a7316691f3fd3e0bb"} Jul 06 19:06:09.698883 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:09.698879 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:09.719243 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:09.719189 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" podStartSLOduration=46.719172721 podStartE2EDuration="46.719172721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:09.717257568 +0000 UTC m=+44.908625277" watchObservedRunningTime="2026-07-06 19:06:09.719172721 +0000 UTC m=+44.910540395" Jul 06 19:06:12.707612 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:12.707555 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-72dgz" event={"ID":"5c23cd74-f515-4362-8f23-724a60a71d92","Type":"ContainerStarted","Data":"b955bebf2342df232c575026a598e8f981756fddcece2eabbe6c647de857ed5f"} Jul 06 19:06:12.708004 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:12.707638 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-72dgz" event={"ID":"5c23cd74-f515-4362-8f23-724a60a71d92","Type":"ContainerStarted","Data":"24318e709dbd7a989d803755054199b22829fad5c79f294d5c1efd700b071392"} Jul 06 19:06:12.708004 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:12.707720 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-72dgz" Jul 06 19:06:12.728181 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:12.728134 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-72dgz" podStartSLOduration=8.819657762 podStartE2EDuration="11.728119574s" totalImagesPullingTime="2.908461812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:06:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:12.726677975 +0000 UTC m=+47.918045667" watchObservedRunningTime="2026-07-06 19:06:12.728119574 +0000 UTC m=+47.919487247" Jul 06 19:06:17.010289 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:17.010255 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24gdx\" (UID: \"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6\") " pod="openshift-insights/insights-runtime-extractor-24gdx" Jul 06 19:06:17.010800 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:17.010320 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24gdx\" (UID: \"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6\") " pod="openshift-insights/insights-runtime-extractor-24gdx" Jul 06 19:06:17.010875 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:17.010853 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24gdx\" (UID: \"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6\") " pod="openshift-insights/insights-runtime-extractor-24gdx" Jul 06 19:06:17.012655 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:17.012633 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24gdx\" (UID: \"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6\") " pod="openshift-insights/insights-runtime-extractor-24gdx" Jul 06 19:06:17.064106 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:17.064069 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-24gdx" Jul 06 19:06:17.203240 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:17.203210 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-24gdx"] Jul 06 19:06:17.206093 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:06:17.206066 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88ae87e_56ea_4b82_b7cf_6d7bbb0612a6.slice/crio-f21e8a5815e012b7c108fe689398b7081f7c1477a06e190e77590941bf5315f5 WatchSource:0}: Error finding container f21e8a5815e012b7c108fe689398b7081f7c1477a06e190e77590941bf5315f5: Status 404 returned error can't find the container with id f21e8a5815e012b7c108fe689398b7081f7c1477a06e190e77590941bf5315f5 Jul 06 19:06:17.720670 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:17.720638 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-24gdx" event={"ID":"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6","Type":"ContainerStarted","Data":"f21e8a5815e012b7c108fe689398b7081f7c1477a06e190e77590941bf5315f5"} Jul 06 19:06:21.732406 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:21.732372 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-24gdx" event={"ID":"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6","Type":"ContainerStarted","Data":"28af3cbc36736a4a415c426b8fe217cf0ecd960313b80ebaa1b9e9271620aee9"} Jul 06 19:06:22.712652 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:22.712620 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-72dgz" Jul 06 19:06:22.740253 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:22.740219 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-24gdx" event={"ID":"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6","Type":"ContainerStarted","Data":"5232ce9936bab3ba5ca71eac23713b334cef8f41319609d8863b9b968ed3a0ae"} Jul 06 19:06:23.724693 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.724658 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cbd47fd7c-4c5fz"] Jul 06 19:06:23.782504 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.782452 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-68b5d89b7f-9mk89"] Jul 06 19:06:23.784859 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.784828 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429"] Jul 06 19:06:23.784991 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.784952 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68b5d89b7f-9mk89" Jul 06 19:06:23.786860 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.786837 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429" Jul 06 19:06:23.788035 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.787976 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 06 19:06:23.788137 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.788077 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 19:06:23.788137 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.788107 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 19:06:23.788457 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.788423 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 19:06:23.792189 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.792117 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 06 19:06:23.792308 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.792203 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 06 19:06:23.792369 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.792316 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 06 19:06:23.792640 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.792619 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 06 19:06:23.807285 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.807263 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-68b5d89b7f-9mk89"] Jul 06 19:06:23.811810 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.811760 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429"] Jul 06 19:06:23.862162 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.862126 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfbq2\" (UniqueName: \"kubernetes.io/projected/fe67fa23-2351-4a20-b3e6-2c7006cb45a2-kube-api-access-rfbq2\") pod \"klusterlet-addon-workmgr-68b5d89b7f-9mk89\" (UID: \"fe67fa23-2351-4a20-b3e6-2c7006cb45a2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68b5d89b7f-9mk89" Jul 06 19:06:23.862162 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.862166 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe67fa23-2351-4a20-b3e6-2c7006cb45a2-tmp\") pod \"klusterlet-addon-workmgr-68b5d89b7f-9mk89\" (UID: \"fe67fa23-2351-4a20-b3e6-2c7006cb45a2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68b5d89b7f-9mk89" Jul 06 19:06:23.862372 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.862205 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2fc9b675-2a15-4684-9792-57666e460829-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-848668db9f-vl429\" (UID: \"2fc9b675-2a15-4684-9792-57666e460829\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429" Jul 06 19:06:23.862372 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.862283 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn8nm\" (UniqueName: \"kubernetes.io/projected/2fc9b675-2a15-4684-9792-57666e460829-kube-api-access-jn8nm\") pod \"cluster-proxy-proxy-agent-848668db9f-vl429\" (UID: \"2fc9b675-2a15-4684-9792-57666e460829\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429" Jul 06 19:06:23.862372 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.862334 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fe67fa23-2351-4a20-b3e6-2c7006cb45a2-klusterlet-config\") pod \"klusterlet-addon-workmgr-68b5d89b7f-9mk89\" (UID: \"fe67fa23-2351-4a20-b3e6-2c7006cb45a2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68b5d89b7f-9mk89" Jul 06 19:06:23.862620 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.862395 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc9b675-2a15-4684-9792-57666e460829-additional-service-ca\") pod \"cluster-proxy-proxy-agent-848668db9f-vl429\" (UID: \"2fc9b675-2a15-4684-9792-57666e460829\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429" Jul 06 19:06:23.862620 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.862424 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2fc9b675-2a15-4684-9792-57666e460829-ca\") pod \"cluster-proxy-proxy-agent-848668db9f-vl429\" (UID: \"2fc9b675-2a15-4684-9792-57666e460829\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429" Jul 06 19:06:23.862620 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.862458 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2fc9b675-2a15-4684-9792-57666e460829-hub\") pod \"cluster-proxy-proxy-agent-848668db9f-vl429\" (UID: \"2fc9b675-2a15-4684-9792-57666e460829\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429" Jul 06 19:06:23.862620 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.862539 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2fc9b675-2a15-4684-9792-57666e460829-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-848668db9f-vl429\" (UID: \"2fc9b675-2a15-4684-9792-57666e460829\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429" Jul 06 19:06:23.962941 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.962903 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2fc9b675-2a15-4684-9792-57666e460829-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-848668db9f-vl429\" (UID: \"2fc9b675-2a15-4684-9792-57666e460829\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429" Jul 06 19:06:23.963139 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.962971 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn8nm\" (UniqueName: \"kubernetes.io/projected/2fc9b675-2a15-4684-9792-57666e460829-kube-api-access-jn8nm\") pod \"cluster-proxy-proxy-agent-848668db9f-vl429\" (UID: \"2fc9b675-2a15-4684-9792-57666e460829\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429" Jul 06 19:06:23.963139 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.963001 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fe67fa23-2351-4a20-b3e6-2c7006cb45a2-klusterlet-config\") pod \"klusterlet-addon-workmgr-68b5d89b7f-9mk89\" (UID: \"fe67fa23-2351-4a20-b3e6-2c7006cb45a2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68b5d89b7f-9mk89" Jul 06 19:06:23.963139 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.963038 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc9b675-2a15-4684-9792-57666e460829-additional-service-ca\") pod \"cluster-proxy-proxy-agent-848668db9f-vl429\" (UID: \"2fc9b675-2a15-4684-9792-57666e460829\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429" Jul 06 19:06:23.963139 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.963070 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2fc9b675-2a15-4684-9792-57666e460829-ca\") pod \"cluster-proxy-proxy-agent-848668db9f-vl429\" (UID: \"2fc9b675-2a15-4684-9792-57666e460829\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429" Jul 06 19:06:23.963139 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.963105 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2fc9b675-2a15-4684-9792-57666e460829-hub\") pod \"cluster-proxy-proxy-agent-848668db9f-vl429\" (UID: \"2fc9b675-2a15-4684-9792-57666e460829\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429" Jul 06 19:06:23.963383 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.963143 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2fc9b675-2a15-4684-9792-57666e460829-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-848668db9f-vl429\" (UID: \"2fc9b675-2a15-4684-9792-57666e460829\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429" Jul 06 19:06:23.963383 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.963172 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfbq2\" (UniqueName: \"kubernetes.io/projected/fe67fa23-2351-4a20-b3e6-2c7006cb45a2-kube-api-access-rfbq2\") pod \"klusterlet-addon-workmgr-68b5d89b7f-9mk89\" (UID: \"fe67fa23-2351-4a20-b3e6-2c7006cb45a2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68b5d89b7f-9mk89" Jul 06 19:06:23.963383 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.963200 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe67fa23-2351-4a20-b3e6-2c7006cb45a2-tmp\") pod \"klusterlet-addon-workmgr-68b5d89b7f-9mk89\" (UID: \"fe67fa23-2351-4a20-b3e6-2c7006cb45a2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68b5d89b7f-9mk89" Jul 06 19:06:23.963679 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.963619 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe67fa23-2351-4a20-b3e6-2c7006cb45a2-tmp\") pod \"klusterlet-addon-workmgr-68b5d89b7f-9mk89\" (UID: \"fe67fa23-2351-4a20-b3e6-2c7006cb45a2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68b5d89b7f-9mk89" Jul 06 19:06:23.964442 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.964396 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc9b675-2a15-4684-9792-57666e460829-additional-service-ca\") pod \"cluster-proxy-proxy-agent-848668db9f-vl429\" (UID: \"2fc9b675-2a15-4684-9792-57666e460829\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429" Jul 06 19:06:23.966227 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.966201 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2fc9b675-2a15-4684-9792-57666e460829-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-848668db9f-vl429\" (UID: \"2fc9b675-2a15-4684-9792-57666e460829\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429" Jul 06 19:06:23.966336 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.966223 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2fc9b675-2a15-4684-9792-57666e460829-hub\") pod \"cluster-proxy-proxy-agent-848668db9f-vl429\" (UID: \"2fc9b675-2a15-4684-9792-57666e460829\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429" Jul 06 19:06:23.966394 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.966340 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fe67fa23-2351-4a20-b3e6-2c7006cb45a2-klusterlet-config\") pod \"klusterlet-addon-workmgr-68b5d89b7f-9mk89\" (UID: \"fe67fa23-2351-4a20-b3e6-2c7006cb45a2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68b5d89b7f-9mk89" Jul 06 19:06:23.966686 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.966663 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2fc9b675-2a15-4684-9792-57666e460829-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-848668db9f-vl429\" (UID: \"2fc9b675-2a15-4684-9792-57666e460829\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429" Jul 06 19:06:23.966766 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.966707 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2fc9b675-2a15-4684-9792-57666e460829-ca\") pod \"cluster-proxy-proxy-agent-848668db9f-vl429\" (UID: \"2fc9b675-2a15-4684-9792-57666e460829\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429" Jul 06 19:06:23.980648 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.980566 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn8nm\" (UniqueName: \"kubernetes.io/projected/2fc9b675-2a15-4684-9792-57666e460829-kube-api-access-jn8nm\") pod \"cluster-proxy-proxy-agent-848668db9f-vl429\" (UID: \"2fc9b675-2a15-4684-9792-57666e460829\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429" Jul 06 19:06:23.981263 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:23.981239 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfbq2\" (UniqueName: \"kubernetes.io/projected/fe67fa23-2351-4a20-b3e6-2c7006cb45a2-kube-api-access-rfbq2\") pod \"klusterlet-addon-workmgr-68b5d89b7f-9mk89\" (UID: \"fe67fa23-2351-4a20-b3e6-2c7006cb45a2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68b5d89b7f-9mk89" Jul 06 19:06:24.098156 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:24.098122 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68b5d89b7f-9mk89" Jul 06 19:06:24.109965 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:24.109911 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429" Jul 06 19:06:25.164553 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:25.164525 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429"] Jul 06 19:06:25.168931 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:06:25.168903 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2fc9b675_2a15_4684_9792_57666e460829.slice/crio-0625523a61bb0cba7df7d9594a91e8c8ffc2011fcb67ee0abc5b9e38ed6b2990 WatchSource:0}: Error finding container 0625523a61bb0cba7df7d9594a91e8c8ffc2011fcb67ee0abc5b9e38ed6b2990: Status 404 returned error can't find the container with id 0625523a61bb0cba7df7d9594a91e8c8ffc2011fcb67ee0abc5b9e38ed6b2990 Jul 06 19:06:25.176958 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:25.176935 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-68b5d89b7f-9mk89"] Jul 06 19:06:25.179821 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:06:25.179799 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe67fa23_2351_4a20_b3e6_2c7006cb45a2.slice/crio-65988525892aefe0db6e7d91b5e3c72d065837f0a10f514e3a57b08131128ee6 WatchSource:0}: Error finding container 65988525892aefe0db6e7d91b5e3c72d065837f0a10f514e3a57b08131128ee6: Status 404 returned error can't find the container with id 65988525892aefe0db6e7d91b5e3c72d065837f0a10f514e3a57b08131128ee6 Jul 06 19:06:25.753045 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:25.753014 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68b5d89b7f-9mk89" event={"ID":"fe67fa23-2351-4a20-b3e6-2c7006cb45a2","Type":"ContainerStarted","Data":"65988525892aefe0db6e7d91b5e3c72d065837f0a10f514e3a57b08131128ee6"} Jul 06 19:06:25.754122 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:25.754097 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429" event={"ID":"2fc9b675-2a15-4684-9792-57666e460829","Type":"ContainerStarted","Data":"0625523a61bb0cba7df7d9594a91e8c8ffc2011fcb67ee0abc5b9e38ed6b2990"} Jul 06 19:06:25.755980 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:25.755957 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-24gdx" event={"ID":"d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6","Type":"ContainerStarted","Data":"524fcfb79afe4390ea9b8ff101340821bbe11d2a3998b8d5e75454052704f516"} Jul 06 19:06:25.774371 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:25.774321 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-24gdx" podStartSLOduration=17.034090533 podStartE2EDuration="24.774302969s" totalImagesPullingTime="7.740212436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:06:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:25.773435369 +0000 UTC m=+60.964803062" watchObservedRunningTime="2026-07-06 19:06:25.774302969 +0000 UTC m=+60.965670643" Jul 06 19:06:26.668062 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:26.667952 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ng9fp" Jul 06 19:06:30.778438 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:30.778392 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68b5d89b7f-9mk89" event={"ID":"fe67fa23-2351-4a20-b3e6-2c7006cb45a2","Type":"ContainerStarted","Data":"9e3ef4f36ac9fafc34c923d0b424ce63545b86b9338b4bb8313ab84e42b02098"} Jul 06 19:06:30.779092 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:30.778459 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68b5d89b7f-9mk89" Jul 06 19:06:30.780368 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:30.780337 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68b5d89b7f-9mk89" Jul 06 19:06:30.780917 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:30.780893 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429" event={"ID":"2fc9b675-2a15-4684-9792-57666e460829","Type":"ContainerStarted","Data":"7fa98efae0025792f8a55e4c7e8b0b46efeae212b5844e63c1c616b6fa2719c0"} Jul 06 19:06:30.780996 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:30.780938 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429" event={"ID":"2fc9b675-2a15-4684-9792-57666e460829","Type":"ContainerStarted","Data":"01ac8b50e85c9e1899c34be22cb21990f744ed57729539e3b7564748479a6af3"} Jul 06 19:06:30.780996 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:30.780959 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429" event={"ID":"2fc9b675-2a15-4684-9792-57666e460829","Type":"ContainerStarted","Data":"a4a66acdd3f4f77f8594ea0892f56d798aa3a1988dabbcb4e3ccba6097b841b8"} Jul 06 19:06:30.801770 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:30.801728 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-68b5d89b7f-9mk89" podStartSLOduration=2.807983139 podStartE2EDuration="7.801714248s" totalImagesPullingTime="4.993731109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:30.801378376 +0000 UTC m=+65.992746051" watchObservedRunningTime="2026-07-06 19:06:30.801714248 +0000 UTC m=+65.993081942" Jul 06 19:06:30.818433 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:30.818380 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-848668db9f-vl429" podStartSLOduration=2.815084561 podStartE2EDuration="7.81836539s" totalImagesPullingTime="5.003280829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:30.817950698 +0000 UTC m=+66.009318371" watchObservedRunningTime="2026-07-06 19:06:30.81836539 +0000 UTC m=+66.009733063" Jul 06 19:06:31.017655 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:31.017615 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0f8d44-df3e-4fb6-afab-15fb76427980-metrics-certs\") pod \"network-metrics-daemon-mz7xr\" (UID: \"1f0f8d44-df3e-4fb6-afab-15fb76427980\") " pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:06:31.017807 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:31.017691 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de165982-d4dc-4449-a40d-1fc3bf2754cc-original-pull-secret\") pod \"global-pull-secret-syncer-dl7lj\" (UID: \"de165982-d4dc-4449-a40d-1fc3bf2754cc\") " pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:06:31.020430 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:31.020412 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 19:06:31.020486 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:31.020454 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 19:06:31.030273 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:31.030212 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f0f8d44-df3e-4fb6-afab-15fb76427980-metrics-certs\") pod \"network-metrics-daemon-mz7xr\" (UID: \"1f0f8d44-df3e-4fb6-afab-15fb76427980\") " pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:06:31.030372 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:31.030304 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de165982-d4dc-4449-a40d-1fc3bf2754cc-original-pull-secret\") pod \"global-pull-secret-syncer-dl7lj\" (UID: \"de165982-d4dc-4449-a40d-1fc3bf2754cc\") " pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:06:31.124698 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:31.124663 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dl7lj" Jul 06 19:06:31.132762 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:31.132743 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lqvqq\"" Jul 06 19:06:31.140052 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:31.140033 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mz7xr" Jul 06 19:06:31.219265 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:31.219225 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfwxp\" (UniqueName: \"kubernetes.io/projected/fc451a85-49aa-4966-b376-70bf49518b2d-kube-api-access-pfwxp\") pod \"network-check-target-ggvtt\" (UID: \"fc451a85-49aa-4966-b376-70bf49518b2d\") " pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:06:31.222220 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:31.222187 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 19:06:31.232160 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:31.232070 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 19:06:31.242796 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:31.242771 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfwxp\" (UniqueName: \"kubernetes.io/projected/fc451a85-49aa-4966-b376-70bf49518b2d-kube-api-access-pfwxp\") pod \"network-check-target-ggvtt\" (UID: \"fc451a85-49aa-4966-b376-70bf49518b2d\") " pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:06:31.258863 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:31.258837 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dl7lj"] Jul 06 19:06:31.262046 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:06:31.262021 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde165982_d4dc_4449_a40d_1fc3bf2754cc.slice/crio-09443aa67248dc7ff559c0d92d68434c2a51ea3056ec77f202dc9227d47d8931 WatchSource:0}: Error finding container 09443aa67248dc7ff559c0d92d68434c2a51ea3056ec77f202dc9227d47d8931: Status 404 returned error can't find the container with id 09443aa67248dc7ff559c0d92d68434c2a51ea3056ec77f202dc9227d47d8931 Jul 06 19:06:31.278824 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:31.278792 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mz7xr"] Jul 06 19:06:31.282436 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:06:31.282415 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0f8d44_df3e_4fb6_afab_15fb76427980.slice/crio-4b15b7c962a13f844d83052a7dc99ff949916c5226739b9d84c1af2855e080ca WatchSource:0}: Error finding container 4b15b7c962a13f844d83052a7dc99ff949916c5226739b9d84c1af2855e080ca: Status 404 returned error can't find the container with id 4b15b7c962a13f844d83052a7dc99ff949916c5226739b9d84c1af2855e080ca Jul 06 19:06:31.421594 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:31.421553 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5h8dt\"" Jul 06 19:06:31.429016 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:31.429001 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:06:31.546500 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:31.546429 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ggvtt"] Jul 06 19:06:31.549121 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:06:31.549095 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc451a85_49aa_4966_b376_70bf49518b2d.slice/crio-04f1949d1422ddf95f24a791bf83791d97ea0fc1fc93946fd6b1a350018f2dfe WatchSource:0}: Error finding container 04f1949d1422ddf95f24a791bf83791d97ea0fc1fc93946fd6b1a350018f2dfe: Status 404 returned error can't find the container with id 04f1949d1422ddf95f24a791bf83791d97ea0fc1fc93946fd6b1a350018f2dfe Jul 06 19:06:31.786473 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:31.786432 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dl7lj" event={"ID":"de165982-d4dc-4449-a40d-1fc3bf2754cc","Type":"ContainerStarted","Data":"09443aa67248dc7ff559c0d92d68434c2a51ea3056ec77f202dc9227d47d8931"} Jul 06 19:06:31.787953 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:31.787920 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ggvtt" event={"ID":"fc451a85-49aa-4966-b376-70bf49518b2d","Type":"ContainerStarted","Data":"04f1949d1422ddf95f24a791bf83791d97ea0fc1fc93946fd6b1a350018f2dfe"} Jul 06 19:06:31.789190 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:31.789151 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mz7xr" event={"ID":"1f0f8d44-df3e-4fb6-afab-15fb76427980","Type":"ContainerStarted","Data":"4b15b7c962a13f844d83052a7dc99ff949916c5226739b9d84c1af2855e080ca"} Jul 06 19:06:32.796029 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:32.795861 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mz7xr" event={"ID":"1f0f8d44-df3e-4fb6-afab-15fb76427980","Type":"ContainerStarted","Data":"0a7c2a6f45f0f1743dc47c7d7ecf4fc4d14ed1971320dcfdaf2e210cd789cfab"} Jul 06 19:06:32.796029 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:32.795980 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mz7xr" event={"ID":"1f0f8d44-df3e-4fb6-afab-15fb76427980","Type":"ContainerStarted","Data":"cb4ee44d3794c0c5fd27a2d5c2e2ac5c522383cba0e625a0646aa19dde48945c"} Jul 06 19:06:32.812987 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:32.812921 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mz7xr" podStartSLOduration=66.644038435 podStartE2EDuration="1m7.812900735s" totalImagesPullingTime="1.1688623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:05:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:32.810982002 +0000 UTC m=+68.002349703" watchObservedRunningTime="2026-07-06 19:06:32.812900735 +0000 UTC m=+68.004268410" Jul 06 19:06:33.730330 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:33.730302 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:36.812218 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:36.812185 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dl7lj" event={"ID":"de165982-d4dc-4449-a40d-1fc3bf2754cc","Type":"ContainerStarted","Data":"da662ae5cbf3641138afdd7bc29cb7d8a6f183c018853c8baea3d9a2ea12f53d"} Jul 06 19:06:36.813598 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:36.813552 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ggvtt" event={"ID":"fc451a85-49aa-4966-b376-70bf49518b2d","Type":"ContainerStarted","Data":"3d6daa1e4bb9109251a2824c247343e8906c4875e770d04ab27c7412dbd0b29c"} Jul 06 19:06:36.813724 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:36.813672 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:06:36.826846 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:36.826798 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dl7lj" podStartSLOduration=65.863035919 podStartE2EDuration="1m10.82678244s" totalImagesPullingTime="4.963746521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:36.826368373 +0000 UTC m=+72.017736060" watchObservedRunningTime="2026-07-06 19:06:36.82678244 +0000 UTC m=+72.018150104" Jul 06 19:06:36.840720 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:36.840675 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ggvtt" podStartSLOduration=66.163256533 podStartE2EDuration="1m10.840660743s" totalImagesPullingTime="4.67740421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:36.839838274 +0000 UTC m=+72.031205947" watchObservedRunningTime="2026-07-06 19:06:36.840660743 +0000 UTC m=+72.032028416" Jul 06 19:06:42.968150 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:42.968121 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-d2tsw"] Jul 06 19:06:42.970870 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:42.970854 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:42.973921 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:42.973898 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 19:06:42.974054 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:42.973983 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 19:06:42.974054 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:42.974003 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 19:06:42.974054 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:42.974005 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 19:06:42.974204 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:42.974003 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 19:06:42.974204 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:42.974003 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 19:06:42.975178 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:42.975163 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hp6cr\"" Jul 06 19:06:43.004955 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.004932 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3897544-f579-477e-b95c-a9a0415a9cea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d2tsw\" (UID: \"a3897544-f579-477e-b95c-a9a0415a9cea\") " pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.005087 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.004961 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3897544-f579-477e-b95c-a9a0415a9cea-sys\") pod \"node-exporter-d2tsw\" (UID: \"a3897544-f579-477e-b95c-a9a0415a9cea\") " pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.005087 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.004987 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a3897544-f579-477e-b95c-a9a0415a9cea-node-exporter-textfile\") pod \"node-exporter-d2tsw\" (UID: \"a3897544-f579-477e-b95c-a9a0415a9cea\") " pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.005087 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.005003 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j7xq\" (UniqueName: \"kubernetes.io/projected/a3897544-f579-477e-b95c-a9a0415a9cea-kube-api-access-7j7xq\") pod \"node-exporter-d2tsw\" (UID: \"a3897544-f579-477e-b95c-a9a0415a9cea\") " pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.005087 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.005050 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a3897544-f579-477e-b95c-a9a0415a9cea-node-exporter-accelerators-collector-config\") pod \"node-exporter-d2tsw\" (UID: \"a3897544-f579-477e-b95c-a9a0415a9cea\") " pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.005231 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.005106 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a3897544-f579-477e-b95c-a9a0415a9cea-root\") pod \"node-exporter-d2tsw\" (UID: \"a3897544-f579-477e-b95c-a9a0415a9cea\") " pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.005231 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.005127 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3897544-f579-477e-b95c-a9a0415a9cea-metrics-client-ca\") pod \"node-exporter-d2tsw\" (UID: \"a3897544-f579-477e-b95c-a9a0415a9cea\") " pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.005231 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.005154 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a3897544-f579-477e-b95c-a9a0415a9cea-node-exporter-wtmp\") pod \"node-exporter-d2tsw\" (UID: \"a3897544-f579-477e-b95c-a9a0415a9cea\") " pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.005231 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.005211 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a3897544-f579-477e-b95c-a9a0415a9cea-node-exporter-tls\") pod \"node-exporter-d2tsw\" (UID: \"a3897544-f579-477e-b95c-a9a0415a9cea\") " pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.105701 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.105668 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a3897544-f579-477e-b95c-a9a0415a9cea-node-exporter-accelerators-collector-config\") pod \"node-exporter-d2tsw\" (UID: \"a3897544-f579-477e-b95c-a9a0415a9cea\") " pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.105875 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.105711 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a3897544-f579-477e-b95c-a9a0415a9cea-root\") pod \"node-exporter-d2tsw\" (UID: \"a3897544-f579-477e-b95c-a9a0415a9cea\") " pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.105875 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.105730 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3897544-f579-477e-b95c-a9a0415a9cea-metrics-client-ca\") pod \"node-exporter-d2tsw\" (UID: \"a3897544-f579-477e-b95c-a9a0415a9cea\") " pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.105875 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.105751 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a3897544-f579-477e-b95c-a9a0415a9cea-node-exporter-wtmp\") pod \"node-exporter-d2tsw\" (UID: \"a3897544-f579-477e-b95c-a9a0415a9cea\") " pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.105875 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.105768 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a3897544-f579-477e-b95c-a9a0415a9cea-node-exporter-tls\") pod \"node-exporter-d2tsw\" (UID: \"a3897544-f579-477e-b95c-a9a0415a9cea\") " pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.105875 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.105814 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a3897544-f579-477e-b95c-a9a0415a9cea-root\") pod \"node-exporter-d2tsw\" (UID: \"a3897544-f579-477e-b95c-a9a0415a9cea\") " pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.106135 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.105892 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3897544-f579-477e-b95c-a9a0415a9cea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d2tsw\" (UID: \"a3897544-f579-477e-b95c-a9a0415a9cea\") " pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.106135 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.105899 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a3897544-f579-477e-b95c-a9a0415a9cea-node-exporter-wtmp\") pod \"node-exporter-d2tsw\" (UID: \"a3897544-f579-477e-b95c-a9a0415a9cea\") " pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.106135 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.105935 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3897544-f579-477e-b95c-a9a0415a9cea-sys\") pod \"node-exporter-d2tsw\" (UID: \"a3897544-f579-477e-b95c-a9a0415a9cea\") " pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.106135 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.105987 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a3897544-f579-477e-b95c-a9a0415a9cea-node-exporter-textfile\") pod \"node-exporter-d2tsw\" (UID: \"a3897544-f579-477e-b95c-a9a0415a9cea\") " pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.106135 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.106012 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j7xq\" (UniqueName: \"kubernetes.io/projected/a3897544-f579-477e-b95c-a9a0415a9cea-kube-api-access-7j7xq\") pod \"node-exporter-d2tsw\" (UID: \"a3897544-f579-477e-b95c-a9a0415a9cea\") " pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.106135 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.106055 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3897544-f579-477e-b95c-a9a0415a9cea-sys\") pod \"node-exporter-d2tsw\" (UID: \"a3897544-f579-477e-b95c-a9a0415a9cea\") " pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.106396 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.106358 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a3897544-f579-477e-b95c-a9a0415a9cea-node-exporter-accelerators-collector-config\") pod \"node-exporter-d2tsw\" (UID: \"a3897544-f579-477e-b95c-a9a0415a9cea\") " pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.106396 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.106373 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a3897544-f579-477e-b95c-a9a0415a9cea-node-exporter-textfile\") pod \"node-exporter-d2tsw\" (UID: \"a3897544-f579-477e-b95c-a9a0415a9cea\") " pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.106493 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.106464 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3897544-f579-477e-b95c-a9a0415a9cea-metrics-client-ca\") pod \"node-exporter-d2tsw\" (UID: \"a3897544-f579-477e-b95c-a9a0415a9cea\") " pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.108587 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.108558 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a3897544-f579-477e-b95c-a9a0415a9cea-node-exporter-tls\") pod \"node-exporter-d2tsw\" (UID: \"a3897544-f579-477e-b95c-a9a0415a9cea\") " pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.108678 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.108659 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3897544-f579-477e-b95c-a9a0415a9cea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d2tsw\" (UID: \"a3897544-f579-477e-b95c-a9a0415a9cea\") " pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.115070 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.115028 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j7xq\" (UniqueName: \"kubernetes.io/projected/a3897544-f579-477e-b95c-a9a0415a9cea-kube-api-access-7j7xq\") pod \"node-exporter-d2tsw\" (UID: \"a3897544-f579-477e-b95c-a9a0415a9cea\") " pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.280292 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.280194 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d2tsw" Jul 06 19:06:43.292063 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:06:43.292034 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3897544_f579_477e_b95c_a9a0415a9cea.slice/crio-50e7681117e63532d23c8cc7a5d65d390a49492dd1a4670b3db6b79f563af6fc WatchSource:0}: Error finding container 50e7681117e63532d23c8cc7a5d65d390a49492dd1a4670b3db6b79f563af6fc: Status 404 returned error can't find the container with id 50e7681117e63532d23c8cc7a5d65d390a49492dd1a4670b3db6b79f563af6fc Jul 06 19:06:43.838233 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:43.838199 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d2tsw" event={"ID":"a3897544-f579-477e-b95c-a9a0415a9cea","Type":"ContainerStarted","Data":"50e7681117e63532d23c8cc7a5d65d390a49492dd1a4670b3db6b79f563af6fc"} Jul 06 19:06:44.842908 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:44.842873 2560 generic.go:356] "Generic (PLEG): container finished" podID="a3897544-f579-477e-b95c-a9a0415a9cea" containerID="b3feb8e17c84cc95d34e58161ba90281a8a73bcde38e2dfaf31d897d438c432e" exitCode=0 Jul 06 19:06:44.843291 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:44.842945 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d2tsw" event={"ID":"a3897544-f579-477e-b95c-a9a0415a9cea","Type":"ContainerDied","Data":"b3feb8e17c84cc95d34e58161ba90281a8a73bcde38e2dfaf31d897d438c432e"} Jul 06 19:06:45.848549 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:45.848509 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d2tsw" event={"ID":"a3897544-f579-477e-b95c-a9a0415a9cea","Type":"ContainerStarted","Data":"cf8e27832092d74fb8c3f5a2b0837b261d7618a958e39dda10c4cc634da45e3b"} Jul 06 19:06:45.849021 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:45.848689 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d2tsw" event={"ID":"a3897544-f579-477e-b95c-a9a0415a9cea","Type":"ContainerStarted","Data":"64bc2e08ea96c70386db4b76ff390a7e30d4b7f12d582c394283f93906a99b8d"} Jul 06 19:06:45.867966 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:45.867916 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-d2tsw" podStartSLOduration=2.485622167 podStartE2EDuration="3.867900968s" totalImagesPullingTime="1.317856809s" totalInitContainerRuntime="64.421992ms" isStatefulPod=true podCreationTimestamp="2026-07-06 19:06:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:06:45.866163659 +0000 UTC m=+81.057531345" watchObservedRunningTime="2026-07-06 19:06:45.867900968 +0000 UTC m=+81.059268640" Jul 06 19:06:48.751167 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:48.751124 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" podUID="a4b30d09-8527-4555-9df9-2b7fdb5918e2" containerName="registry" containerID="cri-o://e7e880194b65623f2319553208e2c8a536fdd8093bedb8bad72b2bc0d3a3dccf" gracePeriod=30 Jul 06 19:06:49.000664 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.000641 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:49.043804 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.043736 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-bound-sa-token\") pod \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " Jul 06 19:06:49.043804 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.043774 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4b30d09-8527-4555-9df9-2b7fdb5918e2-trusted-ca\") pod \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " Jul 06 19:06:49.043998 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.043809 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4b30d09-8527-4555-9df9-2b7fdb5918e2-image-registry-private-configuration\") pod \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " Jul 06 19:06:49.043998 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.043941 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svsbn\" (UniqueName: \"kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-kube-api-access-svsbn\") pod \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " Jul 06 19:06:49.044106 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.043996 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4b30d09-8527-4555-9df9-2b7fdb5918e2-installation-pull-secrets\") pod \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " Jul 06 19:06:49.044106 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.044031 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-registry-tls\") pod \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " Jul 06 19:06:49.044106 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.044085 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4b30d09-8527-4555-9df9-2b7fdb5918e2-registry-certificates\") pod \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " Jul 06 19:06:49.044244 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.044128 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4b30d09-8527-4555-9df9-2b7fdb5918e2-ca-trust-extracted\") pod \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\" (UID: \"a4b30d09-8527-4555-9df9-2b7fdb5918e2\") " Jul 06 19:06:49.044244 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.044172 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4b30d09-8527-4555-9df9-2b7fdb5918e2-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a4b30d09-8527-4555-9df9-2b7fdb5918e2" (UID: "a4b30d09-8527-4555-9df9-2b7fdb5918e2"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:06:49.044368 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.044354 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4b30d09-8527-4555-9df9-2b7fdb5918e2-trusted-ca\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:06:49.044687 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.044648 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4b30d09-8527-4555-9df9-2b7fdb5918e2-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a4b30d09-8527-4555-9df9-2b7fdb5918e2" (UID: "a4b30d09-8527-4555-9df9-2b7fdb5918e2"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:06:49.046509 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.046449 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4b30d09-8527-4555-9df9-2b7fdb5918e2-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a4b30d09-8527-4555-9df9-2b7fdb5918e2" (UID: "a4b30d09-8527-4555-9df9-2b7fdb5918e2"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:06:49.046633 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.046550 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4b30d09-8527-4555-9df9-2b7fdb5918e2-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a4b30d09-8527-4555-9df9-2b7fdb5918e2" (UID: "a4b30d09-8527-4555-9df9-2b7fdb5918e2"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:06:49.046633 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.046603 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a4b30d09-8527-4555-9df9-2b7fdb5918e2" (UID: "a4b30d09-8527-4555-9df9-2b7fdb5918e2"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:06:49.046750 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.046688 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-kube-api-access-svsbn" (OuterVolumeSpecName: "kube-api-access-svsbn") pod "a4b30d09-8527-4555-9df9-2b7fdb5918e2" (UID: "a4b30d09-8527-4555-9df9-2b7fdb5918e2"). InnerVolumeSpecName "kube-api-access-svsbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:06:49.046750 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.046722 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a4b30d09-8527-4555-9df9-2b7fdb5918e2" (UID: "a4b30d09-8527-4555-9df9-2b7fdb5918e2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:06:49.052964 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.052943 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b30d09-8527-4555-9df9-2b7fdb5918e2-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a4b30d09-8527-4555-9df9-2b7fdb5918e2" (UID: "a4b30d09-8527-4555-9df9-2b7fdb5918e2"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:06:49.145000 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.144968 2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a4b30d09-8527-4555-9df9-2b7fdb5918e2-registry-certificates\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:06:49.145000 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.144995 2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a4b30d09-8527-4555-9df9-2b7fdb5918e2-ca-trust-extracted\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:06:49.145000 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.145005 2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-bound-sa-token\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:06:49.145195 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.145013 2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a4b30d09-8527-4555-9df9-2b7fdb5918e2-image-registry-private-configuration\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:06:49.145195 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.145023 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svsbn\" (UniqueName: \"kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-kube-api-access-svsbn\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:06:49.145195 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.145031 2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a4b30d09-8527-4555-9df9-2b7fdb5918e2-installation-pull-secrets\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:06:49.145195 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.145040 2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a4b30d09-8527-4555-9df9-2b7fdb5918e2-registry-tls\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:06:49.268753 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.268721 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-m8s99"] Jul 06 19:06:49.268984 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.268972 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4b30d09-8527-4555-9df9-2b7fdb5918e2" containerName="registry" Jul 06 19:06:49.269024 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.268985 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4b30d09-8527-4555-9df9-2b7fdb5918e2" containerName="registry" Jul 06 19:06:49.269060 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.269030 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4b30d09-8527-4555-9df9-2b7fdb5918e2" containerName="registry" Jul 06 19:06:49.272236 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.272213 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-m8s99" Jul 06 19:06:49.274887 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.274863 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 19:06:49.275008 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.274987 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-b4hnx\"" Jul 06 19:06:49.275309 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.275293 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 19:06:49.282206 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.282182 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-m8s99"] Jul 06 19:06:49.346673 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.346600 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxhsm\" (UniqueName: \"kubernetes.io/projected/4f2e60c7-e7d9-4d33-88aa-1d9572b42567-kube-api-access-zxhsm\") pod \"downloads-685bfb5f64-m8s99\" (UID: \"4f2e60c7-e7d9-4d33-88aa-1d9572b42567\") " pod="openshift-console/downloads-685bfb5f64-m8s99" Jul 06 19:06:49.447371 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.447333 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxhsm\" (UniqueName: \"kubernetes.io/projected/4f2e60c7-e7d9-4d33-88aa-1d9572b42567-kube-api-access-zxhsm\") pod \"downloads-685bfb5f64-m8s99\" (UID: \"4f2e60c7-e7d9-4d33-88aa-1d9572b42567\") " pod="openshift-console/downloads-685bfb5f64-m8s99" Jul 06 19:06:49.454827 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.454800 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxhsm\" (UniqueName: \"kubernetes.io/projected/4f2e60c7-e7d9-4d33-88aa-1d9572b42567-kube-api-access-zxhsm\") pod \"downloads-685bfb5f64-m8s99\" (UID: \"4f2e60c7-e7d9-4d33-88aa-1d9572b42567\") " pod="openshift-console/downloads-685bfb5f64-m8s99" Jul 06 19:06:49.581335 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.581293 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-m8s99" Jul 06 19:06:49.705076 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.705042 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-m8s99"] Jul 06 19:06:49.708165 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:06:49.708140 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f2e60c7_e7d9_4d33_88aa_1d9572b42567.slice/crio-3139415357ba600e3db64c4bd4459446af01e243a93a1a4077f591038f27c9aa WatchSource:0}: Error finding container 3139415357ba600e3db64c4bd4459446af01e243a93a1a4077f591038f27c9aa: Status 404 returned error can't find the container with id 3139415357ba600e3db64c4bd4459446af01e243a93a1a4077f591038f27c9aa Jul 06 19:06:49.862876 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.862826 2560 generic.go:356] "Generic (PLEG): container finished" podID="a4b30d09-8527-4555-9df9-2b7fdb5918e2" containerID="e7e880194b65623f2319553208e2c8a536fdd8093bedb8bad72b2bc0d3a3dccf" exitCode=0 Jul 06 19:06:49.863347 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.862985 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" event={"ID":"a4b30d09-8527-4555-9df9-2b7fdb5918e2","Type":"ContainerDied","Data":"e7e880194b65623f2319553208e2c8a536fdd8093bedb8bad72b2bc0d3a3dccf"} Jul 06 19:06:49.863347 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.863061 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" event={"ID":"a4b30d09-8527-4555-9df9-2b7fdb5918e2","Type":"ContainerDied","Data":"ee355b6891409479250891b79b8c60c4aca7e63d4e076d8a7316691f3fd3e0bb"} Jul 06 19:06:49.863347 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.863104 2560 scope.go:117] "RemoveContainer" containerID="e7e880194b65623f2319553208e2c8a536fdd8093bedb8bad72b2bc0d3a3dccf" Jul 06 19:06:49.863347 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.863298 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cbd47fd7c-4c5fz" Jul 06 19:06:49.865391 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.865355 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-m8s99" event={"ID":"4f2e60c7-e7d9-4d33-88aa-1d9572b42567","Type":"ContainerStarted","Data":"3139415357ba600e3db64c4bd4459446af01e243a93a1a4077f591038f27c9aa"} Jul 06 19:06:49.871965 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.871945 2560 scope.go:117] "RemoveContainer" containerID="e7e880194b65623f2319553208e2c8a536fdd8093bedb8bad72b2bc0d3a3dccf" Jul 06 19:06:49.872228 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:06:49.872209 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7e880194b65623f2319553208e2c8a536fdd8093bedb8bad72b2bc0d3a3dccf\": container with ID starting with e7e880194b65623f2319553208e2c8a536fdd8093bedb8bad72b2bc0d3a3dccf not found: ID does not exist" containerID="e7e880194b65623f2319553208e2c8a536fdd8093bedb8bad72b2bc0d3a3dccf" Jul 06 19:06:49.872322 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.872239 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7e880194b65623f2319553208e2c8a536fdd8093bedb8bad72b2bc0d3a3dccf"} err="failed to get container status \"e7e880194b65623f2319553208e2c8a536fdd8093bedb8bad72b2bc0d3a3dccf\": rpc error: code = NotFound desc = could not find container \"e7e880194b65623f2319553208e2c8a536fdd8093bedb8bad72b2bc0d3a3dccf\": container with ID starting with e7e880194b65623f2319553208e2c8a536fdd8093bedb8bad72b2bc0d3a3dccf not found: ID does not exist" Jul 06 19:06:49.880224 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.880156 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cbd47fd7c-4c5fz"] Jul 06 19:06:49.883536 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:49.883518 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-cbd47fd7c-4c5fz"] Jul 06 19:06:51.391323 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:51.391291 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4b30d09-8527-4555-9df9-2b7fdb5918e2" path="/var/lib/kubelet/pods/a4b30d09-8527-4555-9df9-2b7fdb5918e2/volumes" Jul 06 19:06:59.329989 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.329955 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c6888777b-45tls"] Jul 06 19:06:59.332091 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.332062 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c6888777b-45tls" Jul 06 19:06:59.335051 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.335030 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vz865\"" Jul 06 19:06:59.335174 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.335054 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 19:06:59.336420 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.336361 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 19:06:59.336420 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.336361 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 19:06:59.336607 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.336444 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 19:06:59.336725 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.336692 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 19:06:59.344546 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.344524 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c6888777b-45tls"] Jul 06 19:06:59.418583 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.418535 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b52785b-e7e6-443b-8d6c-4101bf776560-console-oauth-config\") pod \"console-6c6888777b-45tls\" (UID: \"3b52785b-e7e6-443b-8d6c-4101bf776560\") " pod="openshift-console/console-6c6888777b-45tls" Jul 06 19:06:59.418583 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.418587 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b52785b-e7e6-443b-8d6c-4101bf776560-console-config\") pod \"console-6c6888777b-45tls\" (UID: \"3b52785b-e7e6-443b-8d6c-4101bf776560\") " pod="openshift-console/console-6c6888777b-45tls" Jul 06 19:06:59.418788 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.418608 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b52785b-e7e6-443b-8d6c-4101bf776560-oauth-serving-cert\") pod \"console-6c6888777b-45tls\" (UID: \"3b52785b-e7e6-443b-8d6c-4101bf776560\") " pod="openshift-console/console-6c6888777b-45tls" Jul 06 19:06:59.418788 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.418686 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b52785b-e7e6-443b-8d6c-4101bf776560-console-serving-cert\") pod \"console-6c6888777b-45tls\" (UID: \"3b52785b-e7e6-443b-8d6c-4101bf776560\") " pod="openshift-console/console-6c6888777b-45tls" Jul 06 19:06:59.418788 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.418732 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfjl2\" (UniqueName: \"kubernetes.io/projected/3b52785b-e7e6-443b-8d6c-4101bf776560-kube-api-access-sfjl2\") pod \"console-6c6888777b-45tls\" (UID: \"3b52785b-e7e6-443b-8d6c-4101bf776560\") " pod="openshift-console/console-6c6888777b-45tls" Jul 06 19:06:59.418788 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.418768 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b52785b-e7e6-443b-8d6c-4101bf776560-service-ca\") pod \"console-6c6888777b-45tls\" (UID: \"3b52785b-e7e6-443b-8d6c-4101bf776560\") " pod="openshift-console/console-6c6888777b-45tls" Jul 06 19:06:59.519799 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.519765 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b52785b-e7e6-443b-8d6c-4101bf776560-console-oauth-config\") pod \"console-6c6888777b-45tls\" (UID: \"3b52785b-e7e6-443b-8d6c-4101bf776560\") " pod="openshift-console/console-6c6888777b-45tls" Jul 06 19:06:59.519799 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.519804 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b52785b-e7e6-443b-8d6c-4101bf776560-console-config\") pod \"console-6c6888777b-45tls\" (UID: \"3b52785b-e7e6-443b-8d6c-4101bf776560\") " pod="openshift-console/console-6c6888777b-45tls" Jul 06 19:06:59.520054 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.519832 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b52785b-e7e6-443b-8d6c-4101bf776560-oauth-serving-cert\") pod \"console-6c6888777b-45tls\" (UID: \"3b52785b-e7e6-443b-8d6c-4101bf776560\") " pod="openshift-console/console-6c6888777b-45tls" Jul 06 19:06:59.520054 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.519866 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b52785b-e7e6-443b-8d6c-4101bf776560-console-serving-cert\") pod \"console-6c6888777b-45tls\" (UID: \"3b52785b-e7e6-443b-8d6c-4101bf776560\") " pod="openshift-console/console-6c6888777b-45tls" Jul 06 19:06:59.520054 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.519897 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfjl2\" (UniqueName: \"kubernetes.io/projected/3b52785b-e7e6-443b-8d6c-4101bf776560-kube-api-access-sfjl2\") pod \"console-6c6888777b-45tls\" (UID: \"3b52785b-e7e6-443b-8d6c-4101bf776560\") " pod="openshift-console/console-6c6888777b-45tls" Jul 06 19:06:59.520054 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.519921 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b52785b-e7e6-443b-8d6c-4101bf776560-service-ca\") pod \"console-6c6888777b-45tls\" (UID: \"3b52785b-e7e6-443b-8d6c-4101bf776560\") " pod="openshift-console/console-6c6888777b-45tls" Jul 06 19:06:59.520710 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.520681 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b52785b-e7e6-443b-8d6c-4101bf776560-console-config\") pod \"console-6c6888777b-45tls\" (UID: \"3b52785b-e7e6-443b-8d6c-4101bf776560\") " pod="openshift-console/console-6c6888777b-45tls" Jul 06 19:06:59.520850 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.520681 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b52785b-e7e6-443b-8d6c-4101bf776560-oauth-serving-cert\") pod \"console-6c6888777b-45tls\" (UID: \"3b52785b-e7e6-443b-8d6c-4101bf776560\") " pod="openshift-console/console-6c6888777b-45tls" Jul 06 19:06:59.520850 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.520761 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b52785b-e7e6-443b-8d6c-4101bf776560-service-ca\") pod \"console-6c6888777b-45tls\" (UID: \"3b52785b-e7e6-443b-8d6c-4101bf776560\") " pod="openshift-console/console-6c6888777b-45tls" Jul 06 19:06:59.522590 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.522550 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b52785b-e7e6-443b-8d6c-4101bf776560-console-oauth-config\") pod \"console-6c6888777b-45tls\" (UID: \"3b52785b-e7e6-443b-8d6c-4101bf776560\") " pod="openshift-console/console-6c6888777b-45tls" Jul 06 19:06:59.522700 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.522613 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b52785b-e7e6-443b-8d6c-4101bf776560-console-serving-cert\") pod \"console-6c6888777b-45tls\" (UID: \"3b52785b-e7e6-443b-8d6c-4101bf776560\") " pod="openshift-console/console-6c6888777b-45tls" Jul 06 19:06:59.528526 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.528496 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfjl2\" (UniqueName: \"kubernetes.io/projected/3b52785b-e7e6-443b-8d6c-4101bf776560-kube-api-access-sfjl2\") pod \"console-6c6888777b-45tls\" (UID: \"3b52785b-e7e6-443b-8d6c-4101bf776560\") " pod="openshift-console/console-6c6888777b-45tls" Jul 06 19:06:59.642870 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:06:59.642778 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c6888777b-45tls" Jul 06 19:07:06.178971 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:06.178937 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c6888777b-45tls"] Jul 06 19:07:06.181649 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:07:06.181622 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b52785b_e7e6_443b_8d6c_4101bf776560.slice/crio-73fb0c3b85aec7b25de3776c53769c9d902fb4bb4667921b004b7630485519e0 WatchSource:0}: Error finding container 73fb0c3b85aec7b25de3776c53769c9d902fb4bb4667921b004b7630485519e0: Status 404 returned error can't find the container with id 73fb0c3b85aec7b25de3776c53769c9d902fb4bb4667921b004b7630485519e0 Jul 06 19:07:06.933313 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:06.933057 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-m8s99" event={"ID":"4f2e60c7-e7d9-4d33-88aa-1d9572b42567","Type":"ContainerStarted","Data":"cef7d1b47d9d5e552306f22274182ba6a15a8e22166dc969aa65bd6df060fa85"} Jul 06 19:07:06.933816 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:06.933649 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-m8s99" Jul 06 19:07:06.935669 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:06.935612 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c6888777b-45tls" event={"ID":"3b52785b-e7e6-443b-8d6c-4101bf776560","Type":"ContainerStarted","Data":"73fb0c3b85aec7b25de3776c53769c9d902fb4bb4667921b004b7630485519e0"} Jul 06 19:07:06.945747 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:06.945727 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-m8s99" Jul 06 19:07:06.952300 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:06.952171 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-m8s99" podStartSLOduration=1.232409403 podStartE2EDuration="17.952155328s" totalImagesPullingTime="16.719745925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:06:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:07:06.94950096 +0000 UTC m=+102.140868656" watchObservedRunningTime="2026-07-06 19:07:06.952155328 +0000 UTC m=+102.143523001" Jul 06 19:07:07.765568 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:07.765527 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7dc4584b5-jn9h8"] Jul 06 19:07:07.792495 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:07.792464 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dc4584b5-jn9h8"] Jul 06 19:07:07.792671 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:07.792611 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:07:07.809179 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:07.808922 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 19:07:07.819043 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:07.819014 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ggvtt" Jul 06 19:07:07.890830 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:07.890730 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a13d6067-35b6-48ad-ae2d-80439502434e-service-ca\") pod \"console-7dc4584b5-jn9h8\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:07:07.890830 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:07.890787 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a13d6067-35b6-48ad-ae2d-80439502434e-oauth-serving-cert\") pod \"console-7dc4584b5-jn9h8\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:07:07.891074 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:07.890871 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrxnw\" (UniqueName: \"kubernetes.io/projected/a13d6067-35b6-48ad-ae2d-80439502434e-kube-api-access-nrxnw\") pod \"console-7dc4584b5-jn9h8\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:07:07.891074 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:07.890923 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a13d6067-35b6-48ad-ae2d-80439502434e-console-config\") pod \"console-7dc4584b5-jn9h8\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:07:07.891074 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:07.890976 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a13d6067-35b6-48ad-ae2d-80439502434e-console-serving-cert\") pod \"console-7dc4584b5-jn9h8\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:07:07.891074 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:07.891012 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a13d6067-35b6-48ad-ae2d-80439502434e-console-oauth-config\") pod \"console-7dc4584b5-jn9h8\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:07:07.891074 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:07.891045 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a13d6067-35b6-48ad-ae2d-80439502434e-trusted-ca-bundle\") pod \"console-7dc4584b5-jn9h8\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:07:07.992692 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:07.991964 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a13d6067-35b6-48ad-ae2d-80439502434e-console-oauth-config\") pod \"console-7dc4584b5-jn9h8\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:07:07.992692 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:07.992023 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a13d6067-35b6-48ad-ae2d-80439502434e-trusted-ca-bundle\") pod \"console-7dc4584b5-jn9h8\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:07:07.992692 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:07.992063 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a13d6067-35b6-48ad-ae2d-80439502434e-service-ca\") pod \"console-7dc4584b5-jn9h8\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:07:07.992692 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:07.992097 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a13d6067-35b6-48ad-ae2d-80439502434e-oauth-serving-cert\") pod \"console-7dc4584b5-jn9h8\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:07:07.992692 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:07.992140 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrxnw\" (UniqueName: \"kubernetes.io/projected/a13d6067-35b6-48ad-ae2d-80439502434e-kube-api-access-nrxnw\") pod \"console-7dc4584b5-jn9h8\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:07:07.992692 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:07.992188 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a13d6067-35b6-48ad-ae2d-80439502434e-console-config\") pod \"console-7dc4584b5-jn9h8\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:07:07.992692 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:07.992226 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a13d6067-35b6-48ad-ae2d-80439502434e-console-serving-cert\") pod \"console-7dc4584b5-jn9h8\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:07:07.993426 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:07.993392 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a13d6067-35b6-48ad-ae2d-80439502434e-oauth-serving-cert\") pod \"console-7dc4584b5-jn9h8\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:07:07.993538 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:07.993438 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a13d6067-35b6-48ad-ae2d-80439502434e-console-config\") pod \"console-7dc4584b5-jn9h8\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:07:07.993626 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:07.993610 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a13d6067-35b6-48ad-ae2d-80439502434e-trusted-ca-bundle\") pod \"console-7dc4584b5-jn9h8\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:07:07.994031 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:07.994004 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a13d6067-35b6-48ad-ae2d-80439502434e-service-ca\") pod \"console-7dc4584b5-jn9h8\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:07:07.996461 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:07.996428 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a13d6067-35b6-48ad-ae2d-80439502434e-console-oauth-config\") pod \"console-7dc4584b5-jn9h8\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:07:07.996547 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:07.996432 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a13d6067-35b6-48ad-ae2d-80439502434e-console-serving-cert\") pod \"console-7dc4584b5-jn9h8\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:07:08.001778 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:08.001734 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrxnw\" (UniqueName: \"kubernetes.io/projected/a13d6067-35b6-48ad-ae2d-80439502434e-kube-api-access-nrxnw\") pod \"console-7dc4584b5-jn9h8\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:07:08.106108 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:08.106012 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:07:08.301938 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:08.301904 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dc4584b5-jn9h8"] Jul 06 19:07:08.306854 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:07:08.306820 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13d6067_35b6_48ad_ae2d_80439502434e.slice/crio-6865e37932ac1b22847a765dd00efc85faa734bef03be27264f93802d706f2eb WatchSource:0}: Error finding container 6865e37932ac1b22847a765dd00efc85faa734bef03be27264f93802d706f2eb: Status 404 returned error can't find the container with id 6865e37932ac1b22847a765dd00efc85faa734bef03be27264f93802d706f2eb Jul 06 19:07:08.950063 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:08.950015 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dc4584b5-jn9h8" event={"ID":"a13d6067-35b6-48ad-ae2d-80439502434e","Type":"ContainerStarted","Data":"6865e37932ac1b22847a765dd00efc85faa734bef03be27264f93802d706f2eb"} Jul 06 19:07:10.960422 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:10.960379 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c6888777b-45tls" event={"ID":"3b52785b-e7e6-443b-8d6c-4101bf776560","Type":"ContainerStarted","Data":"6ab1b179f2a62bf635bc3f83e5bc815fb0eb49acf80fa7e6eaa4598b445a6728"} Jul 06 19:07:10.962163 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:10.962139 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dc4584b5-jn9h8" event={"ID":"a13d6067-35b6-48ad-ae2d-80439502434e","Type":"ContainerStarted","Data":"a76c92b75d32e46b5e814fd5bc0917bb1ae3599b22b29ca2ae2471feda30b0df"} Jul 06 19:07:10.979854 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:10.979803 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c6888777b-45tls" podStartSLOduration=7.87400976 podStartE2EDuration="11.97978709s" totalImagesPullingTime="4.10577733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:06:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:07:10.978281558 +0000 UTC m=+106.169649258" watchObservedRunningTime="2026-07-06 19:07:10.97978709 +0000 UTC m=+106.171154796" Jul 06 19:07:10.998513 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:10.998457 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7dc4584b5-jn9h8" podStartSLOduration=2.017768304 podStartE2EDuration="3.998437003s" totalImagesPullingTime="1.980668699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:07:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:07:10.997087722 +0000 UTC m=+106.188455397" watchObservedRunningTime="2026-07-06 19:07:10.998437003 +0000 UTC m=+106.189804679" Jul 06 19:07:18.106243 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:18.106206 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:07:18.106799 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:18.106276 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:07:18.110799 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:18.110775 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:07:18.994261 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:18.994234 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:07:19.039283 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:19.039247 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c6888777b-45tls"] Jul 06 19:07:19.643596 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:19.643541 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c6888777b-45tls" Jul 06 19:07:44.059207 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:44.059163 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6c6888777b-45tls" podUID="3b52785b-e7e6-443b-8d6c-4101bf776560" containerName="console" containerID="cri-o://6ab1b179f2a62bf635bc3f83e5bc815fb0eb49acf80fa7e6eaa4598b445a6728" gracePeriod=15 Jul 06 19:07:44.347888 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:44.347814 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c6888777b-45tls" Jul 06 19:07:44.359788 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:44.358997 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b52785b-e7e6-443b-8d6c-4101bf776560-console-serving-cert\") pod \"3b52785b-e7e6-443b-8d6c-4101bf776560\" (UID: \"3b52785b-e7e6-443b-8d6c-4101bf776560\") " Jul 06 19:07:44.359788 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:44.359066 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b52785b-e7e6-443b-8d6c-4101bf776560-console-oauth-config\") pod \"3b52785b-e7e6-443b-8d6c-4101bf776560\" (UID: \"3b52785b-e7e6-443b-8d6c-4101bf776560\") " Jul 06 19:07:44.359788 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:44.359095 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b52785b-e7e6-443b-8d6c-4101bf776560-oauth-serving-cert\") pod \"3b52785b-e7e6-443b-8d6c-4101bf776560\" (UID: \"3b52785b-e7e6-443b-8d6c-4101bf776560\") " Jul 06 19:07:44.359788 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:44.359136 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b52785b-e7e6-443b-8d6c-4101bf776560-service-ca\") pod \"3b52785b-e7e6-443b-8d6c-4101bf776560\" (UID: \"3b52785b-e7e6-443b-8d6c-4101bf776560\") " Jul 06 19:07:44.359788 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:44.359204 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b52785b-e7e6-443b-8d6c-4101bf776560-console-config\") pod \"3b52785b-e7e6-443b-8d6c-4101bf776560\" (UID: \"3b52785b-e7e6-443b-8d6c-4101bf776560\") " Jul 06 19:07:44.359788 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:44.359241 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfjl2\" (UniqueName: \"kubernetes.io/projected/3b52785b-e7e6-443b-8d6c-4101bf776560-kube-api-access-sfjl2\") pod \"3b52785b-e7e6-443b-8d6c-4101bf776560\" (UID: \"3b52785b-e7e6-443b-8d6c-4101bf776560\") " Jul 06 19:07:44.360782 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:44.360523 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b52785b-e7e6-443b-8d6c-4101bf776560-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3b52785b-e7e6-443b-8d6c-4101bf776560" (UID: "3b52785b-e7e6-443b-8d6c-4101bf776560"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:07:44.363191 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:44.362946 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b52785b-e7e6-443b-8d6c-4101bf776560-service-ca" (OuterVolumeSpecName: "service-ca") pod "3b52785b-e7e6-443b-8d6c-4101bf776560" (UID: "3b52785b-e7e6-443b-8d6c-4101bf776560"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:07:44.363191 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:44.363127 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b52785b-e7e6-443b-8d6c-4101bf776560-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3b52785b-e7e6-443b-8d6c-4101bf776560" (UID: "3b52785b-e7e6-443b-8d6c-4101bf776560"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:07:44.363525 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:44.363491 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b52785b-e7e6-443b-8d6c-4101bf776560-console-config" (OuterVolumeSpecName: "console-config") pod "3b52785b-e7e6-443b-8d6c-4101bf776560" (UID: "3b52785b-e7e6-443b-8d6c-4101bf776560"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:07:44.365553 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:44.365523 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b52785b-e7e6-443b-8d6c-4101bf776560-kube-api-access-sfjl2" (OuterVolumeSpecName: "kube-api-access-sfjl2") pod "3b52785b-e7e6-443b-8d6c-4101bf776560" (UID: "3b52785b-e7e6-443b-8d6c-4101bf776560"). InnerVolumeSpecName "kube-api-access-sfjl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:07:44.367771 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:44.367735 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b52785b-e7e6-443b-8d6c-4101bf776560-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3b52785b-e7e6-443b-8d6c-4101bf776560" (UID: "3b52785b-e7e6-443b-8d6c-4101bf776560"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:07:44.460763 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:44.460719 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b52785b-e7e6-443b-8d6c-4101bf776560-console-config\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:07:44.460763 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:44.460763 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfjl2\" (UniqueName: \"kubernetes.io/projected/3b52785b-e7e6-443b-8d6c-4101bf776560-kube-api-access-sfjl2\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:07:44.461036 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:44.460781 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b52785b-e7e6-443b-8d6c-4101bf776560-console-serving-cert\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:07:44.461036 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:44.460797 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b52785b-e7e6-443b-8d6c-4101bf776560-console-oauth-config\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:07:44.461036 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:44.460812 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b52785b-e7e6-443b-8d6c-4101bf776560-oauth-serving-cert\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:07:44.461036 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:44.460827 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b52785b-e7e6-443b-8d6c-4101bf776560-service-ca\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:07:45.079995 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:45.079965 2560 generic.go:356] "Generic (PLEG): container finished" podID="3b52785b-e7e6-443b-8d6c-4101bf776560" containerID="6ab1b179f2a62bf635bc3f83e5bc815fb0eb49acf80fa7e6eaa4598b445a6728" exitCode=0 Jul 06 19:07:45.080410 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:45.080010 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c6888777b-45tls" Jul 06 19:07:45.080410 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:45.080014 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c6888777b-45tls" event={"ID":"3b52785b-e7e6-443b-8d6c-4101bf776560","Type":"ContainerDied","Data":"6ab1b179f2a62bf635bc3f83e5bc815fb0eb49acf80fa7e6eaa4598b445a6728"} Jul 06 19:07:45.080410 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:45.080056 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c6888777b-45tls" event={"ID":"3b52785b-e7e6-443b-8d6c-4101bf776560","Type":"ContainerDied","Data":"73fb0c3b85aec7b25de3776c53769c9d902fb4bb4667921b004b7630485519e0"} Jul 06 19:07:45.080410 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:45.080076 2560 scope.go:117] "RemoveContainer" containerID="6ab1b179f2a62bf635bc3f83e5bc815fb0eb49acf80fa7e6eaa4598b445a6728" Jul 06 19:07:45.100845 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:45.100819 2560 scope.go:117] "RemoveContainer" containerID="6ab1b179f2a62bf635bc3f83e5bc815fb0eb49acf80fa7e6eaa4598b445a6728" Jul 06 19:07:45.101124 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:07:45.101103 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ab1b179f2a62bf635bc3f83e5bc815fb0eb49acf80fa7e6eaa4598b445a6728\": container with ID starting with 6ab1b179f2a62bf635bc3f83e5bc815fb0eb49acf80fa7e6eaa4598b445a6728 not found: ID does not exist" containerID="6ab1b179f2a62bf635bc3f83e5bc815fb0eb49acf80fa7e6eaa4598b445a6728" Jul 06 19:07:45.101224 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:45.101131 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ab1b179f2a62bf635bc3f83e5bc815fb0eb49acf80fa7e6eaa4598b445a6728"} err="failed to get container status \"6ab1b179f2a62bf635bc3f83e5bc815fb0eb49acf80fa7e6eaa4598b445a6728\": rpc error: code = NotFound desc = could not find container \"6ab1b179f2a62bf635bc3f83e5bc815fb0eb49acf80fa7e6eaa4598b445a6728\": container with ID starting with 6ab1b179f2a62bf635bc3f83e5bc815fb0eb49acf80fa7e6eaa4598b445a6728 not found: ID does not exist" Jul 06 19:07:45.101433 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:45.101413 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c6888777b-45tls"] Jul 06 19:07:45.105351 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:45.105326 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c6888777b-45tls"] Jul 06 19:07:45.392998 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:07:45.392928 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b52785b-e7e6-443b-8d6c-4101bf776560" path="/var/lib/kubelet/pods/3b52785b-e7e6-443b-8d6c-4101bf776560/volumes" Jul 06 19:08:11.110145 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.110111 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f67c86f4b-bptrf"] Jul 06 19:08:11.110644 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.110326 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b52785b-e7e6-443b-8d6c-4101bf776560" containerName="console" Jul 06 19:08:11.110644 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.110337 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b52785b-e7e6-443b-8d6c-4101bf776560" containerName="console" Jul 06 19:08:11.110644 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.110387 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b52785b-e7e6-443b-8d6c-4101bf776560" containerName="console" Jul 06 19:08:11.112386 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.112371 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:08:11.125035 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.125009 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f67c86f4b-bptrf"] Jul 06 19:08:11.250785 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.250755 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b110f25-97f4-4799-9f94-46ec012ab2ce-trusted-ca-bundle\") pod \"console-5f67c86f4b-bptrf\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:08:11.250785 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.250794 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b110f25-97f4-4799-9f94-46ec012ab2ce-console-serving-cert\") pod \"console-5f67c86f4b-bptrf\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:08:11.251024 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.250816 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b110f25-97f4-4799-9f94-46ec012ab2ce-console-config\") pod \"console-5f67c86f4b-bptrf\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:08:11.251024 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.250864 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b110f25-97f4-4799-9f94-46ec012ab2ce-service-ca\") pod \"console-5f67c86f4b-bptrf\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:08:11.251024 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.250879 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b110f25-97f4-4799-9f94-46ec012ab2ce-oauth-serving-cert\") pod \"console-5f67c86f4b-bptrf\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:08:11.251024 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.250898 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw4ln\" (UniqueName: \"kubernetes.io/projected/2b110f25-97f4-4799-9f94-46ec012ab2ce-kube-api-access-xw4ln\") pod \"console-5f67c86f4b-bptrf\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:08:11.251024 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.250977 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b110f25-97f4-4799-9f94-46ec012ab2ce-console-oauth-config\") pod \"console-5f67c86f4b-bptrf\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:08:11.351836 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.351801 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b110f25-97f4-4799-9f94-46ec012ab2ce-console-serving-cert\") pod \"console-5f67c86f4b-bptrf\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:08:11.351836 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.351839 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b110f25-97f4-4799-9f94-46ec012ab2ce-console-config\") pod \"console-5f67c86f4b-bptrf\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:08:11.351994 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.351860 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b110f25-97f4-4799-9f94-46ec012ab2ce-service-ca\") pod \"console-5f67c86f4b-bptrf\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:08:11.352078 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.352058 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b110f25-97f4-4799-9f94-46ec012ab2ce-oauth-serving-cert\") pod \"console-5f67c86f4b-bptrf\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:08:11.352112 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.352097 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw4ln\" (UniqueName: \"kubernetes.io/projected/2b110f25-97f4-4799-9f94-46ec012ab2ce-kube-api-access-xw4ln\") pod \"console-5f67c86f4b-bptrf\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:08:11.352167 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.352154 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b110f25-97f4-4799-9f94-46ec012ab2ce-console-oauth-config\") pod \"console-5f67c86f4b-bptrf\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:08:11.352223 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.352211 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b110f25-97f4-4799-9f94-46ec012ab2ce-trusted-ca-bundle\") pod \"console-5f67c86f4b-bptrf\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:08:11.352691 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.352658 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b110f25-97f4-4799-9f94-46ec012ab2ce-service-ca\") pod \"console-5f67c86f4b-bptrf\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:08:11.352812 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.352658 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b110f25-97f4-4799-9f94-46ec012ab2ce-console-config\") pod \"console-5f67c86f4b-bptrf\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:08:11.352971 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.352946 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b110f25-97f4-4799-9f94-46ec012ab2ce-oauth-serving-cert\") pod \"console-5f67c86f4b-bptrf\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:08:11.353035 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.352986 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b110f25-97f4-4799-9f94-46ec012ab2ce-trusted-ca-bundle\") pod \"console-5f67c86f4b-bptrf\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:08:11.354473 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.354453 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b110f25-97f4-4799-9f94-46ec012ab2ce-console-serving-cert\") pod \"console-5f67c86f4b-bptrf\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:08:11.354585 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.354480 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b110f25-97f4-4799-9f94-46ec012ab2ce-console-oauth-config\") pod \"console-5f67c86f4b-bptrf\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:08:11.361115 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.361069 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw4ln\" (UniqueName: \"kubernetes.io/projected/2b110f25-97f4-4799-9f94-46ec012ab2ce-kube-api-access-xw4ln\") pod \"console-5f67c86f4b-bptrf\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:08:11.421625 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.421597 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:08:11.543657 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:11.543621 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f67c86f4b-bptrf"] Jul 06 19:08:11.546752 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:08:11.546724 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b110f25_97f4_4799_9f94_46ec012ab2ce.slice/crio-fcc1d43faf28b3d557ebe02803f0285d34cfdec06ef00b6e309e724b7e03a732 WatchSource:0}: Error finding container fcc1d43faf28b3d557ebe02803f0285d34cfdec06ef00b6e309e724b7e03a732: Status 404 returned error can't find the container with id fcc1d43faf28b3d557ebe02803f0285d34cfdec06ef00b6e309e724b7e03a732 Jul 06 19:08:12.171079 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:12.171046 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f67c86f4b-bptrf" event={"ID":"2b110f25-97f4-4799-9f94-46ec012ab2ce","Type":"ContainerStarted","Data":"1aedcbeff5ae8f8d4d3ae41d25d872a05d61cb0ddefc7c6f441db342176f75a3"} Jul 06 19:08:12.171527 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:12.171098 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f67c86f4b-bptrf" event={"ID":"2b110f25-97f4-4799-9f94-46ec012ab2ce","Type":"ContainerStarted","Data":"fcc1d43faf28b3d557ebe02803f0285d34cfdec06ef00b6e309e724b7e03a732"} Jul 06 19:08:12.191986 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:12.191941 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f67c86f4b-bptrf" podStartSLOduration=1.1919263820000001 podStartE2EDuration="1.191926382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:08:12.190592485 +0000 UTC m=+167.381960174" watchObservedRunningTime="2026-07-06 19:08:12.191926382 +0000 UTC m=+167.383294055" Jul 06 19:08:21.422019 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:21.421919 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:08:21.422019 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:21.422011 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:08:21.426486 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:21.426468 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:08:22.208334 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:22.208306 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:08:22.250413 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:22.250383 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dc4584b5-jn9h8"] Jul 06 19:08:47.269722 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:47.269663 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7dc4584b5-jn9h8" podUID="a13d6067-35b6-48ad-ae2d-80439502434e" containerName="console" containerID="cri-o://a76c92b75d32e46b5e814fd5bc0917bb1ae3599b22b29ca2ae2471feda30b0df" gracePeriod=15 Jul 06 19:08:47.505511 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:47.505486 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:08:47.621632 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:47.621554 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a13d6067-35b6-48ad-ae2d-80439502434e-service-ca\") pod \"a13d6067-35b6-48ad-ae2d-80439502434e\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " Jul 06 19:08:47.621632 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:47.621610 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a13d6067-35b6-48ad-ae2d-80439502434e-console-config\") pod \"a13d6067-35b6-48ad-ae2d-80439502434e\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " Jul 06 19:08:47.621837 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:47.621635 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a13d6067-35b6-48ad-ae2d-80439502434e-trusted-ca-bundle\") pod \"a13d6067-35b6-48ad-ae2d-80439502434e\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " Jul 06 19:08:47.621837 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:47.621658 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a13d6067-35b6-48ad-ae2d-80439502434e-console-serving-cert\") pod \"a13d6067-35b6-48ad-ae2d-80439502434e\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " Jul 06 19:08:47.621837 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:47.621682 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrxnw\" (UniqueName: \"kubernetes.io/projected/a13d6067-35b6-48ad-ae2d-80439502434e-kube-api-access-nrxnw\") pod \"a13d6067-35b6-48ad-ae2d-80439502434e\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " Jul 06 19:08:47.621837 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:47.621700 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a13d6067-35b6-48ad-ae2d-80439502434e-console-oauth-config\") pod \"a13d6067-35b6-48ad-ae2d-80439502434e\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " Jul 06 19:08:47.621837 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:47.621745 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a13d6067-35b6-48ad-ae2d-80439502434e-oauth-serving-cert\") pod \"a13d6067-35b6-48ad-ae2d-80439502434e\" (UID: \"a13d6067-35b6-48ad-ae2d-80439502434e\") " Jul 06 19:08:47.622066 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:47.621902 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13d6067-35b6-48ad-ae2d-80439502434e-service-ca" (OuterVolumeSpecName: "service-ca") pod "a13d6067-35b6-48ad-ae2d-80439502434e" (UID: "a13d6067-35b6-48ad-ae2d-80439502434e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:08:47.622066 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:47.622011 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13d6067-35b6-48ad-ae2d-80439502434e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a13d6067-35b6-48ad-ae2d-80439502434e" (UID: "a13d6067-35b6-48ad-ae2d-80439502434e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:08:47.622246 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:47.622215 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13d6067-35b6-48ad-ae2d-80439502434e-console-config" (OuterVolumeSpecName: "console-config") pod "a13d6067-35b6-48ad-ae2d-80439502434e" (UID: "a13d6067-35b6-48ad-ae2d-80439502434e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:08:47.622308 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:47.622239 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13d6067-35b6-48ad-ae2d-80439502434e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a13d6067-35b6-48ad-ae2d-80439502434e" (UID: "a13d6067-35b6-48ad-ae2d-80439502434e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:08:47.623904 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:47.623879 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a13d6067-35b6-48ad-ae2d-80439502434e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a13d6067-35b6-48ad-ae2d-80439502434e" (UID: "a13d6067-35b6-48ad-ae2d-80439502434e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:08:47.624012 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:47.623941 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a13d6067-35b6-48ad-ae2d-80439502434e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a13d6067-35b6-48ad-ae2d-80439502434e" (UID: "a13d6067-35b6-48ad-ae2d-80439502434e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:08:47.624077 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:47.624058 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a13d6067-35b6-48ad-ae2d-80439502434e-kube-api-access-nrxnw" (OuterVolumeSpecName: "kube-api-access-nrxnw") pod "a13d6067-35b6-48ad-ae2d-80439502434e" (UID: "a13d6067-35b6-48ad-ae2d-80439502434e"). InnerVolumeSpecName "kube-api-access-nrxnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:08:47.723032 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:47.722986 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrxnw\" (UniqueName: \"kubernetes.io/projected/a13d6067-35b6-48ad-ae2d-80439502434e-kube-api-access-nrxnw\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:08:47.723032 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:47.723028 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a13d6067-35b6-48ad-ae2d-80439502434e-console-oauth-config\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:08:47.723032 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:47.723038 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a13d6067-35b6-48ad-ae2d-80439502434e-oauth-serving-cert\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:08:47.723032 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:47.723055 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a13d6067-35b6-48ad-ae2d-80439502434e-service-ca\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:08:47.723300 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:47.723064 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a13d6067-35b6-48ad-ae2d-80439502434e-console-config\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:08:47.723300 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:47.723072 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a13d6067-35b6-48ad-ae2d-80439502434e-trusted-ca-bundle\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:08:47.723300 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:47.723080 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a13d6067-35b6-48ad-ae2d-80439502434e-console-serving-cert\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:08:48.288014 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:48.287975 2560 generic.go:356] "Generic (PLEG): container finished" podID="a13d6067-35b6-48ad-ae2d-80439502434e" containerID="a76c92b75d32e46b5e814fd5bc0917bb1ae3599b22b29ca2ae2471feda30b0df" exitCode=0 Jul 06 19:08:48.288405 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:48.288024 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dc4584b5-jn9h8" Jul 06 19:08:48.288405 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:48.288055 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dc4584b5-jn9h8" event={"ID":"a13d6067-35b6-48ad-ae2d-80439502434e","Type":"ContainerDied","Data":"a76c92b75d32e46b5e814fd5bc0917bb1ae3599b22b29ca2ae2471feda30b0df"} Jul 06 19:08:48.288405 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:48.288104 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dc4584b5-jn9h8" event={"ID":"a13d6067-35b6-48ad-ae2d-80439502434e","Type":"ContainerDied","Data":"6865e37932ac1b22847a765dd00efc85faa734bef03be27264f93802d706f2eb"} Jul 06 19:08:48.288405 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:48.288124 2560 scope.go:117] "RemoveContainer" containerID="a76c92b75d32e46b5e814fd5bc0917bb1ae3599b22b29ca2ae2471feda30b0df" Jul 06 19:08:48.295691 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:48.295673 2560 scope.go:117] "RemoveContainer" containerID="a76c92b75d32e46b5e814fd5bc0917bb1ae3599b22b29ca2ae2471feda30b0df" Jul 06 19:08:48.295961 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:08:48.295937 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a76c92b75d32e46b5e814fd5bc0917bb1ae3599b22b29ca2ae2471feda30b0df\": container with ID starting with a76c92b75d32e46b5e814fd5bc0917bb1ae3599b22b29ca2ae2471feda30b0df not found: ID does not exist" containerID="a76c92b75d32e46b5e814fd5bc0917bb1ae3599b22b29ca2ae2471feda30b0df" Jul 06 19:08:48.296014 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:48.295970 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a76c92b75d32e46b5e814fd5bc0917bb1ae3599b22b29ca2ae2471feda30b0df"} err="failed to get container status \"a76c92b75d32e46b5e814fd5bc0917bb1ae3599b22b29ca2ae2471feda30b0df\": rpc error: code = NotFound desc = could not find container \"a76c92b75d32e46b5e814fd5bc0917bb1ae3599b22b29ca2ae2471feda30b0df\": container with ID starting with a76c92b75d32e46b5e814fd5bc0917bb1ae3599b22b29ca2ae2471feda30b0df not found: ID does not exist" Jul 06 19:08:48.308442 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:48.308415 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dc4584b5-jn9h8"] Jul 06 19:08:48.310289 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:48.310269 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7dc4584b5-jn9h8"] Jul 06 19:08:49.389590 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:08:49.389552 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a13d6067-35b6-48ad-ae2d-80439502434e" path="/var/lib/kubelet/pods/a13d6067-35b6-48ad-ae2d-80439502434e/volumes" Jul 06 19:09:11.564351 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:11.564315 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cxst9"] Jul 06 19:09:11.564779 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:11.564563 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a13d6067-35b6-48ad-ae2d-80439502434e" containerName="console" Jul 06 19:09:11.564779 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:11.564588 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a13d6067-35b6-48ad-ae2d-80439502434e" containerName="console" Jul 06 19:09:11.564779 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:11.564630 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a13d6067-35b6-48ad-ae2d-80439502434e" containerName="console" Jul 06 19:09:11.567360 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:11.567341 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cxst9" Jul 06 19:09:11.570282 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:11.570256 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 06 19:09:11.570411 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:11.570302 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:09:11.570411 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:11.570364 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-j52n6\"" Jul 06 19:09:11.579888 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:11.579861 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cxst9"] Jul 06 19:09:11.580832 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:11.580808 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06a4409d-83ac-488d-a55c-5055e5ede07e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-cxst9\" (UID: \"06a4409d-83ac-488d-a55c-5055e5ede07e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cxst9" Jul 06 19:09:11.580951 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:11.580870 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnpcc\" (UniqueName: \"kubernetes.io/projected/06a4409d-83ac-488d-a55c-5055e5ede07e-kube-api-access-cnpcc\") pod \"cert-manager-operator-controller-manager-55c444bf86-cxst9\" (UID: \"06a4409d-83ac-488d-a55c-5055e5ede07e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cxst9" Jul 06 19:09:11.681977 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:11.681943 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06a4409d-83ac-488d-a55c-5055e5ede07e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-cxst9\" (UID: \"06a4409d-83ac-488d-a55c-5055e5ede07e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cxst9" Jul 06 19:09:11.682169 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:11.681996 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnpcc\" (UniqueName: \"kubernetes.io/projected/06a4409d-83ac-488d-a55c-5055e5ede07e-kube-api-access-cnpcc\") pod \"cert-manager-operator-controller-manager-55c444bf86-cxst9\" (UID: \"06a4409d-83ac-488d-a55c-5055e5ede07e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cxst9" Jul 06 19:09:11.682381 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:11.682355 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06a4409d-83ac-488d-a55c-5055e5ede07e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-cxst9\" (UID: \"06a4409d-83ac-488d-a55c-5055e5ede07e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cxst9" Jul 06 19:09:11.690882 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:11.690854 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnpcc\" (UniqueName: \"kubernetes.io/projected/06a4409d-83ac-488d-a55c-5055e5ede07e-kube-api-access-cnpcc\") pod \"cert-manager-operator-controller-manager-55c444bf86-cxst9\" (UID: \"06a4409d-83ac-488d-a55c-5055e5ede07e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cxst9" Jul 06 19:09:11.877212 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:11.877114 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cxst9" Jul 06 19:09:12.011814 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:12.011781 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cxst9"] Jul 06 19:09:12.014724 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:09:12.014696 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a4409d_83ac_488d_a55c_5055e5ede07e.slice/crio-98faab8822619668853e39c5a1543fa2cbf64e99a00749916eb2ec22d6273326 WatchSource:0}: Error finding container 98faab8822619668853e39c5a1543fa2cbf64e99a00749916eb2ec22d6273326: Status 404 returned error can't find the container with id 98faab8822619668853e39c5a1543fa2cbf64e99a00749916eb2ec22d6273326 Jul 06 19:09:12.368749 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:12.368713 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cxst9" event={"ID":"06a4409d-83ac-488d-a55c-5055e5ede07e","Type":"ContainerStarted","Data":"98faab8822619668853e39c5a1543fa2cbf64e99a00749916eb2ec22d6273326"} Jul 06 19:09:15.380807 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:15.380770 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cxst9" event={"ID":"06a4409d-83ac-488d-a55c-5055e5ede07e","Type":"ContainerStarted","Data":"19c7536000fbf463b3aad3cd30a0b73f49dfd85c23db8dbe80a075582191fa37"} Jul 06 19:09:15.403540 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:15.403493 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-cxst9" podStartSLOduration=1.96742039 podStartE2EDuration="4.403479063s" totalImagesPullingTime="2.436058673s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:09:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:09:15.400943195 +0000 UTC m=+230.592310868" watchObservedRunningTime="2026-07-06 19:09:15.403479063 +0000 UTC m=+230.594846736" Jul 06 19:09:15.991984 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:15.991948 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl"] Jul 06 19:09:15.994029 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:15.994008 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl" Jul 06 19:09:15.997012 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:15.996994 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 06 19:09:15.998267 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:15.998250 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 06 19:09:15.998325 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:15.998250 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-lpf5r\"" Jul 06 19:09:16.004397 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:16.004376 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl"] Jul 06 19:09:16.014852 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:16.014830 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwf2v\" (UniqueName: \"kubernetes.io/projected/444f0728-1c31-44eb-88e9-d85fbb9913c9-kube-api-access-mwf2v\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl\" (UID: \"444f0728-1c31-44eb-88e9-d85fbb9913c9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl" Jul 06 19:09:16.014963 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:16.014860 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/444f0728-1c31-44eb-88e9-d85fbb9913c9-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl\" (UID: \"444f0728-1c31-44eb-88e9-d85fbb9913c9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl" Jul 06 19:09:16.014963 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:16.014900 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/444f0728-1c31-44eb-88e9-d85fbb9913c9-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl\" (UID: \"444f0728-1c31-44eb-88e9-d85fbb9913c9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl" Jul 06 19:09:16.115974 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:16.115932 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwf2v\" (UniqueName: \"kubernetes.io/projected/444f0728-1c31-44eb-88e9-d85fbb9913c9-kube-api-access-mwf2v\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl\" (UID: \"444f0728-1c31-44eb-88e9-d85fbb9913c9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl" Jul 06 19:09:16.115974 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:16.115976 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/444f0728-1c31-44eb-88e9-d85fbb9913c9-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl\" (UID: \"444f0728-1c31-44eb-88e9-d85fbb9913c9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl" Jul 06 19:09:16.116186 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:16.116013 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/444f0728-1c31-44eb-88e9-d85fbb9913c9-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl\" (UID: \"444f0728-1c31-44eb-88e9-d85fbb9913c9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl" Jul 06 19:09:16.116337 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:16.116322 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/444f0728-1c31-44eb-88e9-d85fbb9913c9-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl\" (UID: \"444f0728-1c31-44eb-88e9-d85fbb9913c9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl" Jul 06 19:09:16.116373 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:16.116353 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/444f0728-1c31-44eb-88e9-d85fbb9913c9-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl\" (UID: \"444f0728-1c31-44eb-88e9-d85fbb9913c9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl" Jul 06 19:09:16.125523 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:16.125493 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwf2v\" (UniqueName: \"kubernetes.io/projected/444f0728-1c31-44eb-88e9-d85fbb9913c9-kube-api-access-mwf2v\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl\" (UID: \"444f0728-1c31-44eb-88e9-d85fbb9913c9\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl" Jul 06 19:09:16.303017 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:16.302931 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl" Jul 06 19:09:16.454037 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:16.454006 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl"] Jul 06 19:09:16.456762 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:09:16.456730 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod444f0728_1c31_44eb_88e9_d85fbb9913c9.slice/crio-6f50aa4d15f8c86f89ba25d755d5badc3bc1313b70cabcb7129db2db13dc2145 WatchSource:0}: Error finding container 6f50aa4d15f8c86f89ba25d755d5badc3bc1313b70cabcb7129db2db13dc2145: Status 404 returned error can't find the container with id 6f50aa4d15f8c86f89ba25d755d5badc3bc1313b70cabcb7129db2db13dc2145 Jul 06 19:09:17.392087 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:17.392049 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl" event={"ID":"444f0728-1c31-44eb-88e9-d85fbb9913c9","Type":"ContainerStarted","Data":"6f50aa4d15f8c86f89ba25d755d5badc3bc1313b70cabcb7129db2db13dc2145"} Jul 06 19:09:24.424111 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:24.424077 2560 generic.go:356] "Generic (PLEG): container finished" podID="444f0728-1c31-44eb-88e9-d85fbb9913c9" containerID="48c0e82b9129dcd7b371ce1f81e5666d0417fb6c47d30f837e83657e5eb19c7d" exitCode=0 Jul 06 19:09:24.424548 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:24.424150 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl" event={"ID":"444f0728-1c31-44eb-88e9-d85fbb9913c9","Type":"ContainerDied","Data":"48c0e82b9129dcd7b371ce1f81e5666d0417fb6c47d30f837e83657e5eb19c7d"} Jul 06 19:09:25.661562 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:25.661478 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-svwmx"] Jul 06 19:09:25.663542 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:25.663508 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-svwmx" Jul 06 19:09:25.666237 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:25.666215 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-wrbsc\"" Jul 06 19:09:25.666368 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:25.666215 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 06 19:09:25.667510 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:25.667486 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 06 19:09:25.674851 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:25.674827 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-svwmx"] Jul 06 19:09:25.688354 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:25.688332 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/281ee2d5-5da5-42a8-a0a3-d2b4541f150a-bound-sa-token\") pod \"cert-manager-55b4d954d5-svwmx\" (UID: \"281ee2d5-5da5-42a8-a0a3-d2b4541f150a\") " pod="cert-manager/cert-manager-55b4d954d5-svwmx" Jul 06 19:09:25.688476 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:25.688390 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp27v\" (UniqueName: \"kubernetes.io/projected/281ee2d5-5da5-42a8-a0a3-d2b4541f150a-kube-api-access-hp27v\") pod \"cert-manager-55b4d954d5-svwmx\" (UID: \"281ee2d5-5da5-42a8-a0a3-d2b4541f150a\") " pod="cert-manager/cert-manager-55b4d954d5-svwmx" Jul 06 19:09:25.788879 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:25.788839 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp27v\" (UniqueName: \"kubernetes.io/projected/281ee2d5-5da5-42a8-a0a3-d2b4541f150a-kube-api-access-hp27v\") pod \"cert-manager-55b4d954d5-svwmx\" (UID: \"281ee2d5-5da5-42a8-a0a3-d2b4541f150a\") " pod="cert-manager/cert-manager-55b4d954d5-svwmx" Jul 06 19:09:25.789036 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:25.788909 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/281ee2d5-5da5-42a8-a0a3-d2b4541f150a-bound-sa-token\") pod \"cert-manager-55b4d954d5-svwmx\" (UID: \"281ee2d5-5da5-42a8-a0a3-d2b4541f150a\") " pod="cert-manager/cert-manager-55b4d954d5-svwmx" Jul 06 19:09:25.797532 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:25.797501 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/281ee2d5-5da5-42a8-a0a3-d2b4541f150a-bound-sa-token\") pod \"cert-manager-55b4d954d5-svwmx\" (UID: \"281ee2d5-5da5-42a8-a0a3-d2b4541f150a\") " pod="cert-manager/cert-manager-55b4d954d5-svwmx" Jul 06 19:09:25.797779 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:25.797761 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp27v\" (UniqueName: \"kubernetes.io/projected/281ee2d5-5da5-42a8-a0a3-d2b4541f150a-kube-api-access-hp27v\") pod \"cert-manager-55b4d954d5-svwmx\" (UID: \"281ee2d5-5da5-42a8-a0a3-d2b4541f150a\") " pod="cert-manager/cert-manager-55b4d954d5-svwmx" Jul 06 19:09:25.974173 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:25.974138 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-svwmx" Jul 06 19:09:26.745233 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:26.745193 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-svwmx"] Jul 06 19:09:26.749213 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:09:26.749183 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod281ee2d5_5da5_42a8_a0a3_d2b4541f150a.slice/crio-775c16b86a853c85182733d37ce9f3ed803ab9b6034113bbb3bec9a6caeae107 WatchSource:0}: Error finding container 775c16b86a853c85182733d37ce9f3ed803ab9b6034113bbb3bec9a6caeae107: Status 404 returned error can't find the container with id 775c16b86a853c85182733d37ce9f3ed803ab9b6034113bbb3bec9a6caeae107 Jul 06 19:09:27.436718 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:27.436682 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-svwmx" event={"ID":"281ee2d5-5da5-42a8-a0a3-d2b4541f150a","Type":"ContainerStarted","Data":"775c16b86a853c85182733d37ce9f3ed803ab9b6034113bbb3bec9a6caeae107"} Jul 06 19:09:27.438414 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:27.438386 2560 generic.go:356] "Generic (PLEG): container finished" podID="444f0728-1c31-44eb-88e9-d85fbb9913c9" containerID="207dc0a9b3df5b0e9d925203ef2c9a50e450ffb1ff2ea3136fbdd37eb734753b" exitCode=0 Jul 06 19:09:27.438531 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:27.438437 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl" event={"ID":"444f0728-1c31-44eb-88e9-d85fbb9913c9","Type":"ContainerDied","Data":"207dc0a9b3df5b0e9d925203ef2c9a50e450ffb1ff2ea3136fbdd37eb734753b"} Jul 06 19:09:32.460445 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:32.460408 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-svwmx" event={"ID":"281ee2d5-5da5-42a8-a0a3-d2b4541f150a","Type":"ContainerStarted","Data":"e2e983caf6f38db977ff4065fac784dd161b42bea1062147649a5f0678202362"} Jul 06 19:09:32.477821 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:32.477755 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-svwmx" podStartSLOduration=2.734836125 podStartE2EDuration="7.477737839s" totalImagesPullingTime="4.742901714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:09:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:09:32.477015595 +0000 UTC m=+247.668383269" watchObservedRunningTime="2026-07-06 19:09:32.477737839 +0000 UTC m=+247.669105510" Jul 06 19:09:36.477544 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:36.477509 2560 generic.go:356] "Generic (PLEG): container finished" podID="444f0728-1c31-44eb-88e9-d85fbb9913c9" containerID="02bf338e45e4aaa2fe55e9d81375a43297afdc7e1e1e8ec814df120dcaef4040" exitCode=0 Jul 06 19:09:36.477899 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:36.477547 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl" event={"ID":"444f0728-1c31-44eb-88e9-d85fbb9913c9","Type":"ContainerDied","Data":"02bf338e45e4aaa2fe55e9d81375a43297afdc7e1e1e8ec814df120dcaef4040"} Jul 06 19:09:37.605481 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:37.605456 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl" Jul 06 19:09:37.686766 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:37.686734 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/444f0728-1c31-44eb-88e9-d85fbb9913c9-bundle\") pod \"444f0728-1c31-44eb-88e9-d85fbb9913c9\" (UID: \"444f0728-1c31-44eb-88e9-d85fbb9913c9\") " Jul 06 19:09:37.686766 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:37.686770 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/444f0728-1c31-44eb-88e9-d85fbb9913c9-util\") pod \"444f0728-1c31-44eb-88e9-d85fbb9913c9\" (UID: \"444f0728-1c31-44eb-88e9-d85fbb9913c9\") " Jul 06 19:09:37.686997 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:37.686793 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwf2v\" (UniqueName: \"kubernetes.io/projected/444f0728-1c31-44eb-88e9-d85fbb9913c9-kube-api-access-mwf2v\") pod \"444f0728-1c31-44eb-88e9-d85fbb9913c9\" (UID: \"444f0728-1c31-44eb-88e9-d85fbb9913c9\") " Jul 06 19:09:37.687157 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:37.687125 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/444f0728-1c31-44eb-88e9-d85fbb9913c9-bundle" (OuterVolumeSpecName: "bundle") pod "444f0728-1c31-44eb-88e9-d85fbb9913c9" (UID: "444f0728-1c31-44eb-88e9-d85fbb9913c9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:09:37.689173 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:37.689151 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/444f0728-1c31-44eb-88e9-d85fbb9913c9-kube-api-access-mwf2v" (OuterVolumeSpecName: "kube-api-access-mwf2v") pod "444f0728-1c31-44eb-88e9-d85fbb9913c9" (UID: "444f0728-1c31-44eb-88e9-d85fbb9913c9"). InnerVolumeSpecName "kube-api-access-mwf2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:09:37.690823 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:37.690801 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/444f0728-1c31-44eb-88e9-d85fbb9913c9-util" (OuterVolumeSpecName: "util") pod "444f0728-1c31-44eb-88e9-d85fbb9913c9" (UID: "444f0728-1c31-44eb-88e9-d85fbb9913c9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:09:37.787450 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:37.787366 2560 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/444f0728-1c31-44eb-88e9-d85fbb9913c9-bundle\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:09:37.787450 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:37.787398 2560 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/444f0728-1c31-44eb-88e9-d85fbb9913c9-util\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:09:37.787450 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:37.787408 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwf2v\" (UniqueName: \"kubernetes.io/projected/444f0728-1c31-44eb-88e9-d85fbb9913c9-kube-api-access-mwf2v\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:09:38.486478 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:38.486441 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl" event={"ID":"444f0728-1c31-44eb-88e9-d85fbb9913c9","Type":"ContainerDied","Data":"6f50aa4d15f8c86f89ba25d755d5badc3bc1313b70cabcb7129db2db13dc2145"} Jul 06 19:09:38.486666 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:38.486491 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f50aa4d15f8c86f89ba25d755d5badc3bc1313b70cabcb7129db2db13dc2145" Jul 06 19:09:38.486666 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:38.486453 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxx7gl" Jul 06 19:09:57.506464 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.506430 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7995d775b8-ddtnd"] Jul 06 19:09:57.506979 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.506672 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="444f0728-1c31-44eb-88e9-d85fbb9913c9" containerName="util" Jul 06 19:09:57.506979 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.506683 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="444f0728-1c31-44eb-88e9-d85fbb9913c9" containerName="util" Jul 06 19:09:57.506979 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.506692 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="444f0728-1c31-44eb-88e9-d85fbb9913c9" containerName="extract" Jul 06 19:09:57.506979 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.506697 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="444f0728-1c31-44eb-88e9-d85fbb9913c9" containerName="extract" Jul 06 19:09:57.506979 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.506704 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="444f0728-1c31-44eb-88e9-d85fbb9913c9" containerName="pull" Jul 06 19:09:57.506979 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.506710 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="444f0728-1c31-44eb-88e9-d85fbb9913c9" containerName="pull" Jul 06 19:09:57.506979 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.506747 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="444f0728-1c31-44eb-88e9-d85fbb9913c9" containerName="extract" Jul 06 19:09:57.508461 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.508436 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7995d775b8-ddtnd" Jul 06 19:09:57.511684 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.511666 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 06 19:09:57.513189 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.513171 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 06 19:09:57.513272 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.513213 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 06 19:09:57.513272 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.513232 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rqntc\"" Jul 06 19:09:57.513375 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.513279 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 06 19:09:57.513375 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.513293 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 06 19:09:57.521561 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.521539 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7995d775b8-ddtnd"] Jul 06 19:09:57.628508 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.628467 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cfbd124b-5f35-45c8-abc6-4c321941360c-metrics-cert\") pod \"lws-controller-manager-7995d775b8-ddtnd\" (UID: \"cfbd124b-5f35-45c8-abc6-4c321941360c\") " pod="openshift-lws-operator/lws-controller-manager-7995d775b8-ddtnd" Jul 06 19:09:57.628696 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.628547 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wvs8\" (UniqueName: \"kubernetes.io/projected/cfbd124b-5f35-45c8-abc6-4c321941360c-kube-api-access-2wvs8\") pod \"lws-controller-manager-7995d775b8-ddtnd\" (UID: \"cfbd124b-5f35-45c8-abc6-4c321941360c\") " pod="openshift-lws-operator/lws-controller-manager-7995d775b8-ddtnd" Jul 06 19:09:57.628696 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.628596 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cfbd124b-5f35-45c8-abc6-4c321941360c-manager-config\") pod \"lws-controller-manager-7995d775b8-ddtnd\" (UID: \"cfbd124b-5f35-45c8-abc6-4c321941360c\") " pod="openshift-lws-operator/lws-controller-manager-7995d775b8-ddtnd" Jul 06 19:09:57.628696 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.628618 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfbd124b-5f35-45c8-abc6-4c321941360c-cert\") pod \"lws-controller-manager-7995d775b8-ddtnd\" (UID: \"cfbd124b-5f35-45c8-abc6-4c321941360c\") " pod="openshift-lws-operator/lws-controller-manager-7995d775b8-ddtnd" Jul 06 19:09:57.729963 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.729924 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cfbd124b-5f35-45c8-abc6-4c321941360c-manager-config\") pod \"lws-controller-manager-7995d775b8-ddtnd\" (UID: \"cfbd124b-5f35-45c8-abc6-4c321941360c\") " pod="openshift-lws-operator/lws-controller-manager-7995d775b8-ddtnd" Jul 06 19:09:57.729963 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.729963 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfbd124b-5f35-45c8-abc6-4c321941360c-cert\") pod \"lws-controller-manager-7995d775b8-ddtnd\" (UID: \"cfbd124b-5f35-45c8-abc6-4c321941360c\") " pod="openshift-lws-operator/lws-controller-manager-7995d775b8-ddtnd" Jul 06 19:09:57.730142 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.729984 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cfbd124b-5f35-45c8-abc6-4c321941360c-metrics-cert\") pod \"lws-controller-manager-7995d775b8-ddtnd\" (UID: \"cfbd124b-5f35-45c8-abc6-4c321941360c\") " pod="openshift-lws-operator/lws-controller-manager-7995d775b8-ddtnd" Jul 06 19:09:57.730142 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.730120 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wvs8\" (UniqueName: \"kubernetes.io/projected/cfbd124b-5f35-45c8-abc6-4c321941360c-kube-api-access-2wvs8\") pod \"lws-controller-manager-7995d775b8-ddtnd\" (UID: \"cfbd124b-5f35-45c8-abc6-4c321941360c\") " pod="openshift-lws-operator/lws-controller-manager-7995d775b8-ddtnd" Jul 06 19:09:57.730652 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.730634 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cfbd124b-5f35-45c8-abc6-4c321941360c-manager-config\") pod \"lws-controller-manager-7995d775b8-ddtnd\" (UID: \"cfbd124b-5f35-45c8-abc6-4c321941360c\") " pod="openshift-lws-operator/lws-controller-manager-7995d775b8-ddtnd" Jul 06 19:09:57.732484 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.732463 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cfbd124b-5f35-45c8-abc6-4c321941360c-metrics-cert\") pod \"lws-controller-manager-7995d775b8-ddtnd\" (UID: \"cfbd124b-5f35-45c8-abc6-4c321941360c\") " pod="openshift-lws-operator/lws-controller-manager-7995d775b8-ddtnd" Jul 06 19:09:57.732666 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.732644 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cfbd124b-5f35-45c8-abc6-4c321941360c-cert\") pod \"lws-controller-manager-7995d775b8-ddtnd\" (UID: \"cfbd124b-5f35-45c8-abc6-4c321941360c\") " pod="openshift-lws-operator/lws-controller-manager-7995d775b8-ddtnd" Jul 06 19:09:57.739617 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.739598 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wvs8\" (UniqueName: \"kubernetes.io/projected/cfbd124b-5f35-45c8-abc6-4c321941360c-kube-api-access-2wvs8\") pod \"lws-controller-manager-7995d775b8-ddtnd\" (UID: \"cfbd124b-5f35-45c8-abc6-4c321941360c\") " pod="openshift-lws-operator/lws-controller-manager-7995d775b8-ddtnd" Jul 06 19:09:57.817700 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.817615 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7995d775b8-ddtnd" Jul 06 19:09:57.941863 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:57.941805 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7995d775b8-ddtnd"] Jul 06 19:09:57.943996 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:09:57.943972 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfbd124b_5f35_45c8_abc6_4c321941360c.slice/crio-c80f492155259970c03f56b12eeac846787c90e3265b8325e88dd56238b6531a WatchSource:0}: Error finding container c80f492155259970c03f56b12eeac846787c90e3265b8325e88dd56238b6531a: Status 404 returned error can't find the container with id c80f492155259970c03f56b12eeac846787c90e3265b8325e88dd56238b6531a Jul 06 19:09:58.554082 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:58.554050 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7995d775b8-ddtnd" event={"ID":"cfbd124b-5f35-45c8-abc6-4c321941360c","Type":"ContainerStarted","Data":"c80f492155259970c03f56b12eeac846787c90e3265b8325e88dd56238b6531a"} Jul 06 19:09:59.558916 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:59.558876 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7995d775b8-ddtnd" event={"ID":"cfbd124b-5f35-45c8-abc6-4c321941360c","Type":"ContainerStarted","Data":"7bddf7fba4d6a0aee019e4490cd086e673711c094e2878289981174826760c19"} Jul 06 19:09:59.559337 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:59.559008 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7995d775b8-ddtnd" Jul 06 19:09:59.576428 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:09:59.576378 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7995d775b8-ddtnd" podStartSLOduration=1.100962469 podStartE2EDuration="2.576362231s" totalImagesPullingTime="1.475399762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:09:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:09:59.574663492 +0000 UTC m=+274.766031176" watchObservedRunningTime="2026-07-06 19:09:59.576362231 +0000 UTC m=+274.767729904" Jul 06 19:10:10.565103 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:10.565075 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7995d775b8-ddtnd" Jul 06 19:10:14.332162 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:14.332130 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn"] Jul 06 19:10:14.334354 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:14.334339 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn" Jul 06 19:10:14.338692 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:14.338667 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 19:10:14.339473 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:14.339451 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-d2b6f\"" Jul 06 19:10:14.339626 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:14.339603 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 19:10:14.339860 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:14.339845 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 19:10:14.340002 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:14.339984 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 19:10:14.350304 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:14.350274 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn"] Jul 06 19:10:14.454418 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:14.454380 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fca2463e-c1ba-42e0-85ab-5b321c61940c-tmp\") pod \"opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn\" (UID: \"fca2463e-c1ba-42e0-85ab-5b321c61940c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn" Jul 06 19:10:14.454418 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:14.454420 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fca2463e-c1ba-42e0-85ab-5b321c61940c-manifests\") pod \"opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn\" (UID: \"fca2463e-c1ba-42e0-85ab-5b321c61940c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn" Jul 06 19:10:14.454695 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:14.454447 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fca2463e-c1ba-42e0-85ab-5b321c61940c-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn\" (UID: \"fca2463e-c1ba-42e0-85ab-5b321c61940c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn" Jul 06 19:10:14.454695 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:14.454515 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fca2463e-c1ba-42e0-85ab-5b321c61940c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn\" (UID: \"fca2463e-c1ba-42e0-85ab-5b321c61940c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn" Jul 06 19:10:14.454695 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:14.454626 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gmw6\" (UniqueName: \"kubernetes.io/projected/fca2463e-c1ba-42e0-85ab-5b321c61940c-kube-api-access-2gmw6\") pod \"opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn\" (UID: \"fca2463e-c1ba-42e0-85ab-5b321c61940c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn" Jul 06 19:10:14.554981 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:14.554937 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fca2463e-c1ba-42e0-85ab-5b321c61940c-manifests\") pod \"opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn\" (UID: \"fca2463e-c1ba-42e0-85ab-5b321c61940c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn" Jul 06 19:10:14.555147 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:14.554988 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fca2463e-c1ba-42e0-85ab-5b321c61940c-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn\" (UID: \"fca2463e-c1ba-42e0-85ab-5b321c61940c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn" Jul 06 19:10:14.555147 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:14.555107 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fca2463e-c1ba-42e0-85ab-5b321c61940c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn\" (UID: \"fca2463e-c1ba-42e0-85ab-5b321c61940c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn" Jul 06 19:10:14.555220 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:14.555177 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gmw6\" (UniqueName: \"kubernetes.io/projected/fca2463e-c1ba-42e0-85ab-5b321c61940c-kube-api-access-2gmw6\") pod \"opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn\" (UID: \"fca2463e-c1ba-42e0-85ab-5b321c61940c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn" Jul 06 19:10:14.555220 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:14.555215 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fca2463e-c1ba-42e0-85ab-5b321c61940c-tmp\") pod \"opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn\" (UID: \"fca2463e-c1ba-42e0-85ab-5b321c61940c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn" Jul 06 19:10:14.555399 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:14.555378 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fca2463e-c1ba-42e0-85ab-5b321c61940c-manifests\") pod \"opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn\" (UID: \"fca2463e-c1ba-42e0-85ab-5b321c61940c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn" Jul 06 19:10:14.557348 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:14.557318 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fca2463e-c1ba-42e0-85ab-5b321c61940c-tmp\") pod \"opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn\" (UID: \"fca2463e-c1ba-42e0-85ab-5b321c61940c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn" Jul 06 19:10:14.557485 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:14.557355 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fca2463e-c1ba-42e0-85ab-5b321c61940c-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn\" (UID: \"fca2463e-c1ba-42e0-85ab-5b321c61940c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn" Jul 06 19:10:14.557485 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:14.557407 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fca2463e-c1ba-42e0-85ab-5b321c61940c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn\" (UID: \"fca2463e-c1ba-42e0-85ab-5b321c61940c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn" Jul 06 19:10:14.565052 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:14.565025 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gmw6\" (UniqueName: \"kubernetes.io/projected/fca2463e-c1ba-42e0-85ab-5b321c61940c-kube-api-access-2gmw6\") pod \"opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn\" (UID: \"fca2463e-c1ba-42e0-85ab-5b321c61940c\") " pod="opendatahub/opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn" Jul 06 19:10:14.643745 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:14.643665 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn" Jul 06 19:10:14.801269 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:14.801244 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn"] Jul 06 19:10:14.803752 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:10:14.803722 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca2463e_c1ba_42e0_85ab_5b321c61940c.slice/crio-2f4ab1ae06e37e44211767f3e9c297de347a7c9aaf8037362160bd00db238c98 WatchSource:0}: Error finding container 2f4ab1ae06e37e44211767f3e9c297de347a7c9aaf8037362160bd00db238c98: Status 404 returned error can't find the container with id 2f4ab1ae06e37e44211767f3e9c297de347a7c9aaf8037362160bd00db238c98 Jul 06 19:10:15.619438 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:15.619398 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn" event={"ID":"fca2463e-c1ba-42e0-85ab-5b321c61940c","Type":"ContainerStarted","Data":"2f4ab1ae06e37e44211767f3e9c297de347a7c9aaf8037362160bd00db238c98"} Jul 06 19:10:18.634112 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:18.634073 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn" event={"ID":"fca2463e-c1ba-42e0-85ab-5b321c61940c","Type":"ContainerStarted","Data":"1170ffd37d5dec25d52752cd264934c87162208e3c676230f115fa6096c3d31e"} Jul 06 19:10:19.639143 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:19.639104 2560 generic.go:356] "Generic (PLEG): container finished" podID="fca2463e-c1ba-42e0-85ab-5b321c61940c" containerID="1170ffd37d5dec25d52752cd264934c87162208e3c676230f115fa6096c3d31e" exitCode=0 Jul 06 19:10:19.639528 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:19.639173 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn" event={"ID":"fca2463e-c1ba-42e0-85ab-5b321c61940c","Type":"ContainerDied","Data":"1170ffd37d5dec25d52752cd264934c87162208e3c676230f115fa6096c3d31e"} Jul 06 19:10:20.645048 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:20.645017 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn" event={"ID":"fca2463e-c1ba-42e0-85ab-5b321c61940c","Type":"ContainerStarted","Data":"c2737a4b165d6f11c570a2ef047d9ea359c0c43ee4d45e7de238b207f90b1219"} Jul 06 19:10:20.645444 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:20.645140 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn" Jul 06 19:10:20.668847 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:20.668792 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn" podStartSLOduration=2.385812898 podStartE2EDuration="6.668777681s" totalImagesPullingTime="3.205944297s" totalInitContainerRuntime="1.077020486s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:10:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:10:20.666365093 +0000 UTC m=+295.857732784" watchObservedRunningTime="2026-07-06 19:10:20.668777681 +0000 UTC m=+295.860145354" Jul 06 19:10:25.220156 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:25.220125 2560 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 19:10:31.651330 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:31.651297 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn" Jul 06 19:10:34.044492 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:34.044452 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d5b9884b5-78zsz"] Jul 06 19:10:34.047040 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:34.047010 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d5b9884b5-78zsz" Jul 06 19:10:34.050437 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:34.050415 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 06 19:10:34.051968 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:34.051948 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 06 19:10:34.052051 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:34.051974 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 19:10:34.052051 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:34.051987 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8rxxj\"" Jul 06 19:10:34.052347 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:34.052331 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 19:10:34.074556 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:34.074530 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d5b9884b5-78zsz"] Jul 06 19:10:34.100060 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:34.100027 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cf4ca25-f9f2-412e-bf20-9e918f48a804-tmp\") pod \"kube-auth-proxy-5d5b9884b5-78zsz\" (UID: \"0cf4ca25-f9f2-412e-bf20-9e918f48a804\") " pod="openshift-ingress/kube-auth-proxy-5d5b9884b5-78zsz" Jul 06 19:10:34.100237 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:34.100072 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cf4ca25-f9f2-412e-bf20-9e918f48a804-tls-certs\") pod \"kube-auth-proxy-5d5b9884b5-78zsz\" (UID: \"0cf4ca25-f9f2-412e-bf20-9e918f48a804\") " pod="openshift-ingress/kube-auth-proxy-5d5b9884b5-78zsz" Jul 06 19:10:34.100237 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:34.100119 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2knlg\" (UniqueName: \"kubernetes.io/projected/0cf4ca25-f9f2-412e-bf20-9e918f48a804-kube-api-access-2knlg\") pod \"kube-auth-proxy-5d5b9884b5-78zsz\" (UID: \"0cf4ca25-f9f2-412e-bf20-9e918f48a804\") " pod="openshift-ingress/kube-auth-proxy-5d5b9884b5-78zsz" Jul 06 19:10:34.200839 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:34.200802 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cf4ca25-f9f2-412e-bf20-9e918f48a804-tls-certs\") pod \"kube-auth-proxy-5d5b9884b5-78zsz\" (UID: \"0cf4ca25-f9f2-412e-bf20-9e918f48a804\") " pod="openshift-ingress/kube-auth-proxy-5d5b9884b5-78zsz" Jul 06 19:10:34.201023 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:34.200845 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2knlg\" (UniqueName: \"kubernetes.io/projected/0cf4ca25-f9f2-412e-bf20-9e918f48a804-kube-api-access-2knlg\") pod \"kube-auth-proxy-5d5b9884b5-78zsz\" (UID: \"0cf4ca25-f9f2-412e-bf20-9e918f48a804\") " pod="openshift-ingress/kube-auth-proxy-5d5b9884b5-78zsz" Jul 06 19:10:34.201023 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:34.200892 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cf4ca25-f9f2-412e-bf20-9e918f48a804-tmp\") pod \"kube-auth-proxy-5d5b9884b5-78zsz\" (UID: \"0cf4ca25-f9f2-412e-bf20-9e918f48a804\") " pod="openshift-ingress/kube-auth-proxy-5d5b9884b5-78zsz" Jul 06 19:10:34.203275 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:34.203249 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cf4ca25-f9f2-412e-bf20-9e918f48a804-tmp\") pod \"kube-auth-proxy-5d5b9884b5-78zsz\" (UID: \"0cf4ca25-f9f2-412e-bf20-9e918f48a804\") " pod="openshift-ingress/kube-auth-proxy-5d5b9884b5-78zsz" Jul 06 19:10:34.203630 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:34.203605 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cf4ca25-f9f2-412e-bf20-9e918f48a804-tls-certs\") pod \"kube-auth-proxy-5d5b9884b5-78zsz\" (UID: \"0cf4ca25-f9f2-412e-bf20-9e918f48a804\") " pod="openshift-ingress/kube-auth-proxy-5d5b9884b5-78zsz" Jul 06 19:10:34.223650 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:34.223616 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2knlg\" (UniqueName: \"kubernetes.io/projected/0cf4ca25-f9f2-412e-bf20-9e918f48a804-kube-api-access-2knlg\") pod \"kube-auth-proxy-5d5b9884b5-78zsz\" (UID: \"0cf4ca25-f9f2-412e-bf20-9e918f48a804\") " pod="openshift-ingress/kube-auth-proxy-5d5b9884b5-78zsz" Jul 06 19:10:34.356276 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:34.356197 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d5b9884b5-78zsz" Jul 06 19:10:34.517334 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:34.517305 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d5b9884b5-78zsz"] Jul 06 19:10:34.519897 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:10:34.519870 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf4ca25_f9f2_412e_bf20_9e918f48a804.slice/crio-a2821e29741b54717f7c523455eef165872a250b94ed065989223e1d324a28d1 WatchSource:0}: Error finding container a2821e29741b54717f7c523455eef165872a250b94ed065989223e1d324a28d1: Status 404 returned error can't find the container with id a2821e29741b54717f7c523455eef165872a250b94ed065989223e1d324a28d1 Jul 06 19:10:34.521766 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:34.521749 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:10:34.700059 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:34.700018 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d5b9884b5-78zsz" event={"ID":"0cf4ca25-f9f2-412e-bf20-9e918f48a804","Type":"ContainerStarted","Data":"a2821e29741b54717f7c523455eef165872a250b94ed065989223e1d324a28d1"} Jul 06 19:10:39.723026 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:39.722971 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d5b9884b5-78zsz" event={"ID":"0cf4ca25-f9f2-412e-bf20-9e918f48a804","Type":"ContainerStarted","Data":"3ac31918dd0a22319060efe3fe510e6e97b23ece237e8e689d0d548307660cc4"} Jul 06 19:10:39.739875 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:10:39.739826 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d5b9884b5-78zsz" podStartSLOduration=2.013344014 podStartE2EDuration="6.739810637s" totalImagesPullingTime="4.726466623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:10:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:10:39.738560104 +0000 UTC m=+314.929927777" watchObservedRunningTime="2026-07-06 19:10:39.739810637 +0000 UTC m=+314.931178339" Jul 06 19:12:00.971307 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:00.971272 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68f967bf9b-5cn9r"] Jul 06 19:12:00.973634 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:00.973616 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f967bf9b-5cn9r" Jul 06 19:12:00.990544 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:00.990522 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68f967bf9b-5cn9r"] Jul 06 19:12:01.068597 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:01.068543 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc451369-ba32-4e0f-a332-b6d2c0702f4e-trusted-ca-bundle\") pod \"console-68f967bf9b-5cn9r\" (UID: \"fc451369-ba32-4e0f-a332-b6d2c0702f4e\") " pod="openshift-console/console-68f967bf9b-5cn9r" Jul 06 19:12:01.068778 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:01.068602 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bgp2\" (UniqueName: \"kubernetes.io/projected/fc451369-ba32-4e0f-a332-b6d2c0702f4e-kube-api-access-6bgp2\") pod \"console-68f967bf9b-5cn9r\" (UID: \"fc451369-ba32-4e0f-a332-b6d2c0702f4e\") " pod="openshift-console/console-68f967bf9b-5cn9r" Jul 06 19:12:01.068778 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:01.068636 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc451369-ba32-4e0f-a332-b6d2c0702f4e-service-ca\") pod \"console-68f967bf9b-5cn9r\" (UID: \"fc451369-ba32-4e0f-a332-b6d2c0702f4e\") " pod="openshift-console/console-68f967bf9b-5cn9r" Jul 06 19:12:01.068778 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:01.068680 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc451369-ba32-4e0f-a332-b6d2c0702f4e-oauth-serving-cert\") pod \"console-68f967bf9b-5cn9r\" (UID: \"fc451369-ba32-4e0f-a332-b6d2c0702f4e\") " pod="openshift-console/console-68f967bf9b-5cn9r" Jul 06 19:12:01.068778 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:01.068751 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc451369-ba32-4e0f-a332-b6d2c0702f4e-console-config\") pod \"console-68f967bf9b-5cn9r\" (UID: \"fc451369-ba32-4e0f-a332-b6d2c0702f4e\") " pod="openshift-console/console-68f967bf9b-5cn9r" Jul 06 19:12:01.068778 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:01.068776 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc451369-ba32-4e0f-a332-b6d2c0702f4e-console-oauth-config\") pod \"console-68f967bf9b-5cn9r\" (UID: \"fc451369-ba32-4e0f-a332-b6d2c0702f4e\") " pod="openshift-console/console-68f967bf9b-5cn9r" Jul 06 19:12:01.068979 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:01.068801 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc451369-ba32-4e0f-a332-b6d2c0702f4e-console-serving-cert\") pod \"console-68f967bf9b-5cn9r\" (UID: \"fc451369-ba32-4e0f-a332-b6d2c0702f4e\") " pod="openshift-console/console-68f967bf9b-5cn9r" Jul 06 19:12:01.169843 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:01.169809 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc451369-ba32-4e0f-a332-b6d2c0702f4e-console-serving-cert\") pod \"console-68f967bf9b-5cn9r\" (UID: \"fc451369-ba32-4e0f-a332-b6d2c0702f4e\") " pod="openshift-console/console-68f967bf9b-5cn9r" Jul 06 19:12:01.169843 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:01.169852 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc451369-ba32-4e0f-a332-b6d2c0702f4e-trusted-ca-bundle\") pod \"console-68f967bf9b-5cn9r\" (UID: \"fc451369-ba32-4e0f-a332-b6d2c0702f4e\") " pod="openshift-console/console-68f967bf9b-5cn9r" Jul 06 19:12:01.170099 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:01.169870 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bgp2\" (UniqueName: \"kubernetes.io/projected/fc451369-ba32-4e0f-a332-b6d2c0702f4e-kube-api-access-6bgp2\") pod \"console-68f967bf9b-5cn9r\" (UID: \"fc451369-ba32-4e0f-a332-b6d2c0702f4e\") " pod="openshift-console/console-68f967bf9b-5cn9r" Jul 06 19:12:01.170099 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:01.169895 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc451369-ba32-4e0f-a332-b6d2c0702f4e-service-ca\") pod \"console-68f967bf9b-5cn9r\" (UID: \"fc451369-ba32-4e0f-a332-b6d2c0702f4e\") " pod="openshift-console/console-68f967bf9b-5cn9r" Jul 06 19:12:01.170099 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:01.169912 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc451369-ba32-4e0f-a332-b6d2c0702f4e-oauth-serving-cert\") pod \"console-68f967bf9b-5cn9r\" (UID: \"fc451369-ba32-4e0f-a332-b6d2c0702f4e\") " pod="openshift-console/console-68f967bf9b-5cn9r" Jul 06 19:12:01.170099 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:01.169955 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc451369-ba32-4e0f-a332-b6d2c0702f4e-console-config\") pod \"console-68f967bf9b-5cn9r\" (UID: \"fc451369-ba32-4e0f-a332-b6d2c0702f4e\") " pod="openshift-console/console-68f967bf9b-5cn9r" Jul 06 19:12:01.170099 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:01.169978 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc451369-ba32-4e0f-a332-b6d2c0702f4e-console-oauth-config\") pod \"console-68f967bf9b-5cn9r\" (UID: \"fc451369-ba32-4e0f-a332-b6d2c0702f4e\") " pod="openshift-console/console-68f967bf9b-5cn9r" Jul 06 19:12:01.170708 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:01.170677 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc451369-ba32-4e0f-a332-b6d2c0702f4e-service-ca\") pod \"console-68f967bf9b-5cn9r\" (UID: \"fc451369-ba32-4e0f-a332-b6d2c0702f4e\") " pod="openshift-console/console-68f967bf9b-5cn9r" Jul 06 19:12:01.170816 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:01.170726 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc451369-ba32-4e0f-a332-b6d2c0702f4e-console-config\") pod \"console-68f967bf9b-5cn9r\" (UID: \"fc451369-ba32-4e0f-a332-b6d2c0702f4e\") " pod="openshift-console/console-68f967bf9b-5cn9r" Jul 06 19:12:01.170886 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:01.170862 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc451369-ba32-4e0f-a332-b6d2c0702f4e-oauth-serving-cert\") pod \"console-68f967bf9b-5cn9r\" (UID: \"fc451369-ba32-4e0f-a332-b6d2c0702f4e\") " pod="openshift-console/console-68f967bf9b-5cn9r" Jul 06 19:12:01.170922 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:01.170904 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc451369-ba32-4e0f-a332-b6d2c0702f4e-trusted-ca-bundle\") pod \"console-68f967bf9b-5cn9r\" (UID: \"fc451369-ba32-4e0f-a332-b6d2c0702f4e\") " pod="openshift-console/console-68f967bf9b-5cn9r" Jul 06 19:12:01.172925 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:01.172906 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc451369-ba32-4e0f-a332-b6d2c0702f4e-console-oauth-config\") pod \"console-68f967bf9b-5cn9r\" (UID: \"fc451369-ba32-4e0f-a332-b6d2c0702f4e\") " pod="openshift-console/console-68f967bf9b-5cn9r" Jul 06 19:12:01.173139 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:01.173112 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc451369-ba32-4e0f-a332-b6d2c0702f4e-console-serving-cert\") pod \"console-68f967bf9b-5cn9r\" (UID: \"fc451369-ba32-4e0f-a332-b6d2c0702f4e\") " pod="openshift-console/console-68f967bf9b-5cn9r" Jul 06 19:12:01.180303 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:01.180283 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bgp2\" (UniqueName: \"kubernetes.io/projected/fc451369-ba32-4e0f-a332-b6d2c0702f4e-kube-api-access-6bgp2\") pod \"console-68f967bf9b-5cn9r\" (UID: \"fc451369-ba32-4e0f-a332-b6d2c0702f4e\") " pod="openshift-console/console-68f967bf9b-5cn9r" Jul 06 19:12:01.282711 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:01.282623 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68f967bf9b-5cn9r" Jul 06 19:12:01.453667 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:01.453634 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68f967bf9b-5cn9r"] Jul 06 19:12:01.456023 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:12:01.456002 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc451369_ba32_4e0f_a332_b6d2c0702f4e.slice/crio-c1ed08441be417e256b414cdf2782fd0381fdaaf55ff9bfcd36b717a4eb06ff0 WatchSource:0}: Error finding container c1ed08441be417e256b414cdf2782fd0381fdaaf55ff9bfcd36b717a4eb06ff0: Status 404 returned error can't find the container with id c1ed08441be417e256b414cdf2782fd0381fdaaf55ff9bfcd36b717a4eb06ff0 Jul 06 19:12:02.038872 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:02.038835 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f967bf9b-5cn9r" event={"ID":"fc451369-ba32-4e0f-a332-b6d2c0702f4e","Type":"ContainerStarted","Data":"d4093773b35db821b29d5723a736b1e607cb628fcd1914d96126a55a60d7d868"} Jul 06 19:12:02.039307 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:02.038885 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68f967bf9b-5cn9r" event={"ID":"fc451369-ba32-4e0f-a332-b6d2c0702f4e","Type":"ContainerStarted","Data":"c1ed08441be417e256b414cdf2782fd0381fdaaf55ff9bfcd36b717a4eb06ff0"} Jul 06 19:12:02.062617 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:02.062545 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68f967bf9b-5cn9r" podStartSLOduration=2.062529609 podStartE2EDuration="2.062529609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:02.061350144 +0000 UTC m=+397.252717839" watchObservedRunningTime="2026-07-06 19:12:02.062529609 +0000 UTC m=+397.253897283" Jul 06 19:12:11.283476 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:11.283435 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68f967bf9b-5cn9r" Jul 06 19:12:11.283905 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:11.283498 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68f967bf9b-5cn9r" Jul 06 19:12:11.288350 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:11.288330 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68f967bf9b-5cn9r" Jul 06 19:12:12.081214 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:12.081186 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68f967bf9b-5cn9r" Jul 06 19:12:12.176706 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:12.176671 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f67c86f4b-bptrf"] Jul 06 19:12:22.380320 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:22.380287 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-b429r"] Jul 06 19:12:22.382301 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:22.382285 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b429r" Jul 06 19:12:22.384972 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:22.384952 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 06 19:12:22.385123 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:22.385104 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 19:12:22.386374 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:22.386355 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zhmjg\"" Jul 06 19:12:22.386473 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:22.386355 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 06 19:12:22.386473 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:22.386395 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 19:12:22.396271 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:22.396244 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-b429r"] Jul 06 19:12:22.446824 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:22.446790 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4b3029aa-1e23-4bb9-9f21-cbb38517cb0c-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-b429r\" (UID: \"4b3029aa-1e23-4bb9-9f21-cbb38517cb0c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b429r" Jul 06 19:12:22.446961 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:22.446847 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b3029aa-1e23-4bb9-9f21-cbb38517cb0c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-b429r\" (UID: \"4b3029aa-1e23-4bb9-9f21-cbb38517cb0c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b429r" Jul 06 19:12:22.446961 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:22.446883 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpqsb\" (UniqueName: \"kubernetes.io/projected/4b3029aa-1e23-4bb9-9f21-cbb38517cb0c-kube-api-access-hpqsb\") pod \"kuadrant-console-plugin-66b5774985-b429r\" (UID: \"4b3029aa-1e23-4bb9-9f21-cbb38517cb0c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b429r" Jul 06 19:12:22.547282 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:22.547246 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b3029aa-1e23-4bb9-9f21-cbb38517cb0c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-b429r\" (UID: \"4b3029aa-1e23-4bb9-9f21-cbb38517cb0c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b429r" Jul 06 19:12:22.547282 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:22.547283 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpqsb\" (UniqueName: \"kubernetes.io/projected/4b3029aa-1e23-4bb9-9f21-cbb38517cb0c-kube-api-access-hpqsb\") pod \"kuadrant-console-plugin-66b5774985-b429r\" (UID: \"4b3029aa-1e23-4bb9-9f21-cbb38517cb0c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b429r" Jul 06 19:12:22.547477 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:22.547327 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4b3029aa-1e23-4bb9-9f21-cbb38517cb0c-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-b429r\" (UID: \"4b3029aa-1e23-4bb9-9f21-cbb38517cb0c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b429r" Jul 06 19:12:22.547477 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:12:22.547398 2560 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 06 19:12:22.547545 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:12:22.547485 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b3029aa-1e23-4bb9-9f21-cbb38517cb0c-plugin-serving-cert podName:4b3029aa-1e23-4bb9-9f21-cbb38517cb0c nodeName:}" failed. No retries permitted until 2026-07-06 19:12:23.047465222 +0000 UTC m=+418.238832891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/4b3029aa-1e23-4bb9-9f21-cbb38517cb0c-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-b429r" (UID: "4b3029aa-1e23-4bb9-9f21-cbb38517cb0c") : secret "plugin-serving-cert" not found Jul 06 19:12:22.547912 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:22.547897 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4b3029aa-1e23-4bb9-9f21-cbb38517cb0c-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-b429r\" (UID: \"4b3029aa-1e23-4bb9-9f21-cbb38517cb0c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b429r" Jul 06 19:12:22.562160 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:22.562132 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpqsb\" (UniqueName: \"kubernetes.io/projected/4b3029aa-1e23-4bb9-9f21-cbb38517cb0c-kube-api-access-hpqsb\") pod \"kuadrant-console-plugin-66b5774985-b429r\" (UID: \"4b3029aa-1e23-4bb9-9f21-cbb38517cb0c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b429r" Jul 06 19:12:23.053376 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:23.053337 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b3029aa-1e23-4bb9-9f21-cbb38517cb0c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-b429r\" (UID: \"4b3029aa-1e23-4bb9-9f21-cbb38517cb0c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b429r" Jul 06 19:12:23.053535 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:12:23.053482 2560 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 06 19:12:23.053628 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:12:23.053557 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b3029aa-1e23-4bb9-9f21-cbb38517cb0c-plugin-serving-cert podName:4b3029aa-1e23-4bb9-9f21-cbb38517cb0c nodeName:}" failed. No retries permitted until 2026-07-06 19:12:24.053540828 +0000 UTC m=+419.244908478 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/4b3029aa-1e23-4bb9-9f21-cbb38517cb0c-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-b429r" (UID: "4b3029aa-1e23-4bb9-9f21-cbb38517cb0c") : secret "plugin-serving-cert" not found Jul 06 19:12:24.061268 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:24.061231 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b3029aa-1e23-4bb9-9f21-cbb38517cb0c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-b429r\" (UID: \"4b3029aa-1e23-4bb9-9f21-cbb38517cb0c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b429r" Jul 06 19:12:24.063832 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:24.063796 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b3029aa-1e23-4bb9-9f21-cbb38517cb0c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-b429r\" (UID: \"4b3029aa-1e23-4bb9-9f21-cbb38517cb0c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b429r" Jul 06 19:12:24.191770 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:24.191737 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b429r" Jul 06 19:12:24.318792 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:24.318767 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-b429r"] Jul 06 19:12:24.321055 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:12:24.321030 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b3029aa_1e23_4bb9_9f21_cbb38517cb0c.slice/crio-16d4781194cdba556d7ac92070a87795e80ea668eed26b4be6e77cfaf1252685 WatchSource:0}: Error finding container 16d4781194cdba556d7ac92070a87795e80ea668eed26b4be6e77cfaf1252685: Status 404 returned error can't find the container with id 16d4781194cdba556d7ac92070a87795e80ea668eed26b4be6e77cfaf1252685 Jul 06 19:12:25.128500 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:25.128463 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b429r" event={"ID":"4b3029aa-1e23-4bb9-9f21-cbb38517cb0c","Type":"ContainerStarted","Data":"16d4781194cdba556d7ac92070a87795e80ea668eed26b4be6e77cfaf1252685"} Jul 06 19:12:37.201819 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:37.201735 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-5f67c86f4b-bptrf" podUID="2b110f25-97f4-4799-9f94-46ec012ab2ce" containerName="console" containerID="cri-o://1aedcbeff5ae8f8d4d3ae41d25d872a05d61cb0ddefc7c6f441db342176f75a3" gracePeriod=15 Jul 06 19:12:42.205383 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:42.205339 2560 patch_prober.go:28] interesting pod/console-5f67c86f4b-bptrf container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.15:8443/health\": dial tcp 10.133.0.15:8443: connect: connection refused" start-of-body= Jul 06 19:12:42.205820 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:42.205408 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-5f67c86f4b-bptrf" podUID="2b110f25-97f4-4799-9f94-46ec012ab2ce" containerName="console" probeResult="failure" output="Get \"https://10.133.0.15:8443/health\": dial tcp 10.133.0.15:8443: connect: connection refused" Jul 06 19:12:48.121948 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.121923 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:12:48.158771 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.158748 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xw4ln\" (UniqueName: \"kubernetes.io/projected/2b110f25-97f4-4799-9f94-46ec012ab2ce-kube-api-access-xw4ln\") pod \"2b110f25-97f4-4799-9f94-46ec012ab2ce\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " Jul 06 19:12:48.158906 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.158801 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b110f25-97f4-4799-9f94-46ec012ab2ce-console-serving-cert\") pod \"2b110f25-97f4-4799-9f94-46ec012ab2ce\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " Jul 06 19:12:48.158906 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.158835 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b110f25-97f4-4799-9f94-46ec012ab2ce-service-ca\") pod \"2b110f25-97f4-4799-9f94-46ec012ab2ce\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " Jul 06 19:12:48.158906 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.158859 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b110f25-97f4-4799-9f94-46ec012ab2ce-console-config\") pod \"2b110f25-97f4-4799-9f94-46ec012ab2ce\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " Jul 06 19:12:48.159073 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.159027 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b110f25-97f4-4799-9f94-46ec012ab2ce-console-oauth-config\") pod \"2b110f25-97f4-4799-9f94-46ec012ab2ce\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " Jul 06 19:12:48.159289 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.159238 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b110f25-97f4-4799-9f94-46ec012ab2ce-service-ca" (OuterVolumeSpecName: "service-ca") pod "2b110f25-97f4-4799-9f94-46ec012ab2ce" (UID: "2b110f25-97f4-4799-9f94-46ec012ab2ce"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:12:48.159289 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.159246 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b110f25-97f4-4799-9f94-46ec012ab2ce-console-config" (OuterVolumeSpecName: "console-config") pod "2b110f25-97f4-4799-9f94-46ec012ab2ce" (UID: "2b110f25-97f4-4799-9f94-46ec012ab2ce"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:12:48.159289 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.159283 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b110f25-97f4-4799-9f94-46ec012ab2ce-oauth-serving-cert\") pod \"2b110f25-97f4-4799-9f94-46ec012ab2ce\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " Jul 06 19:12:48.159769 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.159338 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b110f25-97f4-4799-9f94-46ec012ab2ce-trusted-ca-bundle\") pod \"2b110f25-97f4-4799-9f94-46ec012ab2ce\" (UID: \"2b110f25-97f4-4799-9f94-46ec012ab2ce\") " Jul 06 19:12:48.159769 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.159613 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b110f25-97f4-4799-9f94-46ec012ab2ce-service-ca\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:12:48.159769 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.159633 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b110f25-97f4-4799-9f94-46ec012ab2ce-console-config\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:12:48.159769 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.159722 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b110f25-97f4-4799-9f94-46ec012ab2ce-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2b110f25-97f4-4799-9f94-46ec012ab2ce" (UID: "2b110f25-97f4-4799-9f94-46ec012ab2ce"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:12:48.160164 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.160138 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b110f25-97f4-4799-9f94-46ec012ab2ce-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2b110f25-97f4-4799-9f94-46ec012ab2ce" (UID: "2b110f25-97f4-4799-9f94-46ec012ab2ce"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 19:12:48.160854 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.160831 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b110f25-97f4-4799-9f94-46ec012ab2ce-kube-api-access-xw4ln" (OuterVolumeSpecName: "kube-api-access-xw4ln") pod "2b110f25-97f4-4799-9f94-46ec012ab2ce" (UID: "2b110f25-97f4-4799-9f94-46ec012ab2ce"). InnerVolumeSpecName "kube-api-access-xw4ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:12:48.161545 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.161524 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b110f25-97f4-4799-9f94-46ec012ab2ce-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2b110f25-97f4-4799-9f94-46ec012ab2ce" (UID: "2b110f25-97f4-4799-9f94-46ec012ab2ce"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:12:48.161647 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.161624 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b110f25-97f4-4799-9f94-46ec012ab2ce-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2b110f25-97f4-4799-9f94-46ec012ab2ce" (UID: "2b110f25-97f4-4799-9f94-46ec012ab2ce"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:12:48.236447 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.236414 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b429r" event={"ID":"4b3029aa-1e23-4bb9-9f21-cbb38517cb0c","Type":"ContainerStarted","Data":"84e9e6bc567bf078c1391747689cf9eea11282a1d5e0daaa44d6403610eec436"} Jul 06 19:12:48.237694 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.237673 2560 generic.go:356] "Generic (PLEG): container finished" podID="2b110f25-97f4-4799-9f94-46ec012ab2ce" containerID="1aedcbeff5ae8f8d4d3ae41d25d872a05d61cb0ddefc7c6f441db342176f75a3" exitCode=0 Jul 06 19:12:48.237798 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.237708 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f67c86f4b-bptrf" Jul 06 19:12:48.237798 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.237736 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f67c86f4b-bptrf" event={"ID":"2b110f25-97f4-4799-9f94-46ec012ab2ce","Type":"ContainerDied","Data":"1aedcbeff5ae8f8d4d3ae41d25d872a05d61cb0ddefc7c6f441db342176f75a3"} Jul 06 19:12:48.237798 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.237769 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f67c86f4b-bptrf" event={"ID":"2b110f25-97f4-4799-9f94-46ec012ab2ce","Type":"ContainerDied","Data":"fcc1d43faf28b3d557ebe02803f0285d34cfdec06ef00b6e309e724b7e03a732"} Jul 06 19:12:48.237798 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.237788 2560 scope.go:117] "RemoveContainer" containerID="1aedcbeff5ae8f8d4d3ae41d25d872a05d61cb0ddefc7c6f441db342176f75a3" Jul 06 19:12:48.245699 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.245681 2560 scope.go:117] "RemoveContainer" containerID="1aedcbeff5ae8f8d4d3ae41d25d872a05d61cb0ddefc7c6f441db342176f75a3" Jul 06 19:12:48.245947 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:12:48.245929 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aedcbeff5ae8f8d4d3ae41d25d872a05d61cb0ddefc7c6f441db342176f75a3\": container with ID starting with 1aedcbeff5ae8f8d4d3ae41d25d872a05d61cb0ddefc7c6f441db342176f75a3 not found: ID does not exist" containerID="1aedcbeff5ae8f8d4d3ae41d25d872a05d61cb0ddefc7c6f441db342176f75a3" Jul 06 19:12:48.246000 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.245954 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aedcbeff5ae8f8d4d3ae41d25d872a05d61cb0ddefc7c6f441db342176f75a3"} err="failed to get container status \"1aedcbeff5ae8f8d4d3ae41d25d872a05d61cb0ddefc7c6f441db342176f75a3\": rpc error: code = NotFound desc = could not find container \"1aedcbeff5ae8f8d4d3ae41d25d872a05d61cb0ddefc7c6f441db342176f75a3\": container with ID starting with 1aedcbeff5ae8f8d4d3ae41d25d872a05d61cb0ddefc7c6f441db342176f75a3 not found: ID does not exist" Jul 06 19:12:48.260731 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.260701 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b110f25-97f4-4799-9f94-46ec012ab2ce-console-oauth-config\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:12:48.260731 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.260724 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b110f25-97f4-4799-9f94-46ec012ab2ce-oauth-serving-cert\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:12:48.260907 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.260709 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-b429r" podStartSLOduration=2.441436641 podStartE2EDuration="26.260694584s" totalImagesPullingTime="23.819257943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:12:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:12:48.25892359 +0000 UTC m=+443.450291264" watchObservedRunningTime="2026-07-06 19:12:48.260694584 +0000 UTC m=+443.452062257" Jul 06 19:12:48.260907 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.260738 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b110f25-97f4-4799-9f94-46ec012ab2ce-trusted-ca-bundle\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:12:48.260907 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.260886 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw4ln\" (UniqueName: \"kubernetes.io/projected/2b110f25-97f4-4799-9f94-46ec012ab2ce-kube-api-access-xw4ln\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:12:48.260907 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.260899 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b110f25-97f4-4799-9f94-46ec012ab2ce-console-serving-cert\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:12:48.290566 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.290535 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f67c86f4b-bptrf"] Jul 06 19:12:48.299940 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:48.299913 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f67c86f4b-bptrf"] Jul 06 19:12:49.390686 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:12:49.390655 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b110f25-97f4-4799-9f94-46ec012ab2ce" path="/var/lib/kubelet/pods/2b110f25-97f4-4799-9f94-46ec012ab2ce/volumes" Jul 06 19:13:12.244748 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:12.244708 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:13:12.245251 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:12.245004 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b110f25-97f4-4799-9f94-46ec012ab2ce" containerName="console" Jul 06 19:13:12.245251 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:12.245016 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b110f25-97f4-4799-9f94-46ec012ab2ce" containerName="console" Jul 06 19:13:12.245251 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:12.245067 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b110f25-97f4-4799-9f94-46ec012ab2ce" containerName="console" Jul 06 19:13:12.282632 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:12.282600 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:13:12.282800 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:12.282719 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-2mmvv" Jul 06 19:13:12.287295 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:12.287263 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:13:12.287676 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:12.287660 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 06 19:13:12.352348 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:12.352318 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhpqw\" (UniqueName: \"kubernetes.io/projected/3445ab7e-f3f4-4c28-8b1d-5c3771c211d1-kube-api-access-qhpqw\") pod \"limitador-limitador-54d6d988d5-2mmvv\" (UID: \"3445ab7e-f3f4-4c28-8b1d-5c3771c211d1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2mmvv" Jul 06 19:13:12.352524 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:12.352362 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3445ab7e-f3f4-4c28-8b1d-5c3771c211d1-config-file\") pod \"limitador-limitador-54d6d988d5-2mmvv\" (UID: \"3445ab7e-f3f4-4c28-8b1d-5c3771c211d1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2mmvv" Jul 06 19:13:12.453390 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:12.453352 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhpqw\" (UniqueName: \"kubernetes.io/projected/3445ab7e-f3f4-4c28-8b1d-5c3771c211d1-kube-api-access-qhpqw\") pod \"limitador-limitador-54d6d988d5-2mmvv\" (UID: \"3445ab7e-f3f4-4c28-8b1d-5c3771c211d1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2mmvv" Jul 06 19:13:12.453606 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:12.453411 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3445ab7e-f3f4-4c28-8b1d-5c3771c211d1-config-file\") pod \"limitador-limitador-54d6d988d5-2mmvv\" (UID: \"3445ab7e-f3f4-4c28-8b1d-5c3771c211d1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2mmvv" Jul 06 19:13:12.454088 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:12.454064 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3445ab7e-f3f4-4c28-8b1d-5c3771c211d1-config-file\") pod \"limitador-limitador-54d6d988d5-2mmvv\" (UID: \"3445ab7e-f3f4-4c28-8b1d-5c3771c211d1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2mmvv" Jul 06 19:13:12.462919 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:12.462895 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhpqw\" (UniqueName: \"kubernetes.io/projected/3445ab7e-f3f4-4c28-8b1d-5c3771c211d1-kube-api-access-qhpqw\") pod \"limitador-limitador-54d6d988d5-2mmvv\" (UID: \"3445ab7e-f3f4-4c28-8b1d-5c3771c211d1\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-2mmvv" Jul 06 19:13:12.593406 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:12.593325 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-2mmvv" Jul 06 19:13:12.738478 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:12.738426 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:13:12.741477 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:13:12.741438 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3445ab7e_f3f4_4c28_8b1d_5c3771c211d1.slice/crio-51e5dfe82e3243e88fbfc2d97cecaa9b9142ee536f642eedd36bb683d47f592a WatchSource:0}: Error finding container 51e5dfe82e3243e88fbfc2d97cecaa9b9142ee536f642eedd36bb683d47f592a: Status 404 returned error can't find the container with id 51e5dfe82e3243e88fbfc2d97cecaa9b9142ee536f642eedd36bb683d47f592a Jul 06 19:13:12.840924 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:12.840891 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-jvqzw"] Jul 06 19:13:12.884759 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:12.884675 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jvqzw"] Jul 06 19:13:12.884759 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:12.884732 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-jvqzw" Jul 06 19:13:12.888050 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:12.888030 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mfwsm\"" Jul 06 19:13:12.957050 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:12.957023 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ffds\" (UniqueName: \"kubernetes.io/projected/ee4c9939-5e44-401f-8feb-efcad555e59e-kube-api-access-4ffds\") pod \"authorino-75d556bd77-jvqzw\" (UID: \"ee4c9939-5e44-401f-8feb-efcad555e59e\") " pod="kuadrant-system/authorino-75d556bd77-jvqzw" Jul 06 19:13:13.058068 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:13.058037 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ffds\" (UniqueName: \"kubernetes.io/projected/ee4c9939-5e44-401f-8feb-efcad555e59e-kube-api-access-4ffds\") pod \"authorino-75d556bd77-jvqzw\" (UID: \"ee4c9939-5e44-401f-8feb-efcad555e59e\") " pod="kuadrant-system/authorino-75d556bd77-jvqzw" Jul 06 19:13:13.068297 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:13.068270 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ffds\" (UniqueName: \"kubernetes.io/projected/ee4c9939-5e44-401f-8feb-efcad555e59e-kube-api-access-4ffds\") pod \"authorino-75d556bd77-jvqzw\" (UID: \"ee4c9939-5e44-401f-8feb-efcad555e59e\") " pod="kuadrant-system/authorino-75d556bd77-jvqzw" Jul 06 19:13:13.194436 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:13.194347 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-jvqzw" Jul 06 19:13:13.339813 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:13.339776 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-2mmvv" event={"ID":"3445ab7e-f3f4-4c28-8b1d-5c3771c211d1","Type":"ContainerStarted","Data":"51e5dfe82e3243e88fbfc2d97cecaa9b9142ee536f642eedd36bb683d47f592a"} Jul 06 19:13:13.396350 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:13.396322 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jvqzw"] Jul 06 19:13:13.398627 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:13:13.398603 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee4c9939_5e44_401f_8feb_efcad555e59e.slice/crio-f4e474e90dfc42feaa5c8647fbd93d32408d77bec84585684077da1b6506497f WatchSource:0}: Error finding container f4e474e90dfc42feaa5c8647fbd93d32408d77bec84585684077da1b6506497f: Status 404 returned error can't find the container with id f4e474e90dfc42feaa5c8647fbd93d32408d77bec84585684077da1b6506497f Jul 06 19:13:14.347089 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:14.347033 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-jvqzw" event={"ID":"ee4c9939-5e44-401f-8feb-efcad555e59e","Type":"ContainerStarted","Data":"f4e474e90dfc42feaa5c8647fbd93d32408d77bec84585684077da1b6506497f"} Jul 06 19:13:16.358890 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:16.358856 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-2mmvv" event={"ID":"3445ab7e-f3f4-4c28-8b1d-5c3771c211d1","Type":"ContainerStarted","Data":"276f102d8e30d89bef6bc8ea2cde27e27a580e64b2a6c3204a61962c798a2558"} Jul 06 19:13:16.359368 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:16.358959 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-2mmvv" Jul 06 19:13:16.377604 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:16.377545 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-2mmvv" podStartSLOduration=0.880765774 podStartE2EDuration="4.377529049s" totalImagesPullingTime="3.496763275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:13:16.375056858 +0000 UTC m=+471.566424547" watchObservedRunningTime="2026-07-06 19:13:16.377529049 +0000 UTC m=+471.568896734" Jul 06 19:13:18.369354 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:18.369316 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-jvqzw" event={"ID":"ee4c9939-5e44-401f-8feb-efcad555e59e","Type":"ContainerStarted","Data":"91665082190d7b5f1dbe8c64fcdffaa80250cba889ca169c1b1edac2bcc170d7"} Jul 06 19:13:18.385627 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:18.385556 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-jvqzw" podStartSLOduration=1.862437441 podStartE2EDuration="6.385541909s" totalImagesPullingTime="4.523104468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:13:18.385373617 +0000 UTC m=+473.576741293" watchObservedRunningTime="2026-07-06 19:13:18.385541909 +0000 UTC m=+473.576909582" Jul 06 19:13:27.365017 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:27.364987 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-2mmvv" Jul 06 19:13:46.730967 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:46.730931 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jvqzw"] Jul 06 19:13:46.731446 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:46.731115 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-jvqzw" podUID="ee4c9939-5e44-401f-8feb-efcad555e59e" containerName="authorino" containerID="cri-o://91665082190d7b5f1dbe8c64fcdffaa80250cba889ca169c1b1edac2bcc170d7" gracePeriod=30 Jul 06 19:13:47.486638 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:47.486557 2560 generic.go:356] "Generic (PLEG): container finished" podID="ee4c9939-5e44-401f-8feb-efcad555e59e" containerID="91665082190d7b5f1dbe8c64fcdffaa80250cba889ca169c1b1edac2bcc170d7" exitCode=0 Jul 06 19:13:47.486738 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:47.486635 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-jvqzw" event={"ID":"ee4c9939-5e44-401f-8feb-efcad555e59e","Type":"ContainerDied","Data":"91665082190d7b5f1dbe8c64fcdffaa80250cba889ca169c1b1edac2bcc170d7"} Jul 06 19:13:47.486738 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:47.486673 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-jvqzw" event={"ID":"ee4c9939-5e44-401f-8feb-efcad555e59e","Type":"ContainerDied","Data":"f4e474e90dfc42feaa5c8647fbd93d32408d77bec84585684077da1b6506497f"} Jul 06 19:13:47.486738 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:47.486684 2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4e474e90dfc42feaa5c8647fbd93d32408d77bec84585684077da1b6506497f" Jul 06 19:13:47.505776 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:47.505751 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-jvqzw" Jul 06 19:13:47.532502 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:47.532480 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ffds\" (UniqueName: \"kubernetes.io/projected/ee4c9939-5e44-401f-8feb-efcad555e59e-kube-api-access-4ffds\") pod \"ee4c9939-5e44-401f-8feb-efcad555e59e\" (UID: \"ee4c9939-5e44-401f-8feb-efcad555e59e\") " Jul 06 19:13:47.534728 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:47.534693 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4c9939-5e44-401f-8feb-efcad555e59e-kube-api-access-4ffds" (OuterVolumeSpecName: "kube-api-access-4ffds") pod "ee4c9939-5e44-401f-8feb-efcad555e59e" (UID: "ee4c9939-5e44-401f-8feb-efcad555e59e"). InnerVolumeSpecName "kube-api-access-4ffds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:13:47.633058 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:47.633017 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ffds\" (UniqueName: \"kubernetes.io/projected/ee4c9939-5e44-401f-8feb-efcad555e59e-kube-api-access-4ffds\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:13:48.491001 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:48.490960 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-jvqzw" Jul 06 19:13:48.515844 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:48.515808 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jvqzw"] Jul 06 19:13:48.516037 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:48.515865 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jvqzw"] Jul 06 19:13:49.389868 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:13:49.389836 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee4c9939-5e44-401f-8feb-efcad555e59e" path="/var/lib/kubelet/pods/ee4c9939-5e44-401f-8feb-efcad555e59e/volumes" Jul 06 19:14:26.145925 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:14:26.145841 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:14:43.583931 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:14:43.583891 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:15:00.915815 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:00.915777 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:15:13.671531 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:13.671495 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:15:26.668397 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:26.668369 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:15:33.475418 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.475376 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs"] Jul 06 19:15:33.477199 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.475695 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee4c9939-5e44-401f-8feb-efcad555e59e" containerName="authorino" Jul 06 19:15:33.477199 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.475708 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee4c9939-5e44-401f-8feb-efcad555e59e" containerName="authorino" Jul 06 19:15:33.477199 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.475764 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee4c9939-5e44-401f-8feb-efcad555e59e" containerName="authorino" Jul 06 19:15:33.477913 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.477892 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs" Jul 06 19:15:33.480866 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.480849 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 19:15:33.482438 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.482411 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 19:15:33.482438 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.482428 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lqm7w\"" Jul 06 19:15:33.482590 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.482461 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 06 19:15:33.489818 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.489793 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs"] Jul 06 19:15:33.522485 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.522451 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b361b590-f458-4a0f-ac61-178547d6892c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6l6gs\" (UID: \"b361b590-f458-4a0f-ac61-178547d6892c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs" Jul 06 19:15:33.522659 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.522496 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b361b590-f458-4a0f-ac61-178547d6892c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6l6gs\" (UID: \"b361b590-f458-4a0f-ac61-178547d6892c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs" Jul 06 19:15:33.522659 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.522556 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b361b590-f458-4a0f-ac61-178547d6892c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6l6gs\" (UID: \"b361b590-f458-4a0f-ac61-178547d6892c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs" Jul 06 19:15:33.522659 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.522628 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b361b590-f458-4a0f-ac61-178547d6892c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6l6gs\" (UID: \"b361b590-f458-4a0f-ac61-178547d6892c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs" Jul 06 19:15:33.522659 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.522652 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rp8s\" (UniqueName: \"kubernetes.io/projected/b361b590-f458-4a0f-ac61-178547d6892c-kube-api-access-4rp8s\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6l6gs\" (UID: \"b361b590-f458-4a0f-ac61-178547d6892c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs" Jul 06 19:15:33.522814 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.522698 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b361b590-f458-4a0f-ac61-178547d6892c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6l6gs\" (UID: \"b361b590-f458-4a0f-ac61-178547d6892c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs" Jul 06 19:15:33.623040 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.623007 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b361b590-f458-4a0f-ac61-178547d6892c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6l6gs\" (UID: \"b361b590-f458-4a0f-ac61-178547d6892c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs" Jul 06 19:15:33.623208 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.623054 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b361b590-f458-4a0f-ac61-178547d6892c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6l6gs\" (UID: \"b361b590-f458-4a0f-ac61-178547d6892c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs" Jul 06 19:15:33.623208 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.623081 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b361b590-f458-4a0f-ac61-178547d6892c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6l6gs\" (UID: \"b361b590-f458-4a0f-ac61-178547d6892c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs" Jul 06 19:15:33.623208 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.623116 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b361b590-f458-4a0f-ac61-178547d6892c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6l6gs\" (UID: \"b361b590-f458-4a0f-ac61-178547d6892c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs" Jul 06 19:15:33.623208 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.623134 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b361b590-f458-4a0f-ac61-178547d6892c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6l6gs\" (UID: \"b361b590-f458-4a0f-ac61-178547d6892c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs" Jul 06 19:15:33.623208 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.623160 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rp8s\" (UniqueName: \"kubernetes.io/projected/b361b590-f458-4a0f-ac61-178547d6892c-kube-api-access-4rp8s\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6l6gs\" (UID: \"b361b590-f458-4a0f-ac61-178547d6892c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs" Jul 06 19:15:33.623506 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.623473 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b361b590-f458-4a0f-ac61-178547d6892c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6l6gs\" (UID: \"b361b590-f458-4a0f-ac61-178547d6892c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs" Jul 06 19:15:33.623647 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.623522 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b361b590-f458-4a0f-ac61-178547d6892c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6l6gs\" (UID: \"b361b590-f458-4a0f-ac61-178547d6892c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs" Jul 06 19:15:33.623647 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.623552 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b361b590-f458-4a0f-ac61-178547d6892c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6l6gs\" (UID: \"b361b590-f458-4a0f-ac61-178547d6892c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs" Jul 06 19:15:33.625363 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.625347 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b361b590-f458-4a0f-ac61-178547d6892c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6l6gs\" (UID: \"b361b590-f458-4a0f-ac61-178547d6892c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs" Jul 06 19:15:33.625653 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.625637 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b361b590-f458-4a0f-ac61-178547d6892c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6l6gs\" (UID: \"b361b590-f458-4a0f-ac61-178547d6892c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs" Jul 06 19:15:33.632043 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.632017 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rp8s\" (UniqueName: \"kubernetes.io/projected/b361b590-f458-4a0f-ac61-178547d6892c-kube-api-access-4rp8s\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6l6gs\" (UID: \"b361b590-f458-4a0f-ac61-178547d6892c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs" Jul 06 19:15:33.787640 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.787544 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs" Jul 06 19:15:33.916493 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:33.916470 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs"] Jul 06 19:15:33.918631 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:15:33.918603 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb361b590_f458_4a0f_ac61_178547d6892c.slice/crio-16ac17358e656713e953e5b9fff6416c46779dd0bd0cd2db5b2be4c9197ea4f9 WatchSource:0}: Error finding container 16ac17358e656713e953e5b9fff6416c46779dd0bd0cd2db5b2be4c9197ea4f9: Status 404 returned error can't find the container with id 16ac17358e656713e953e5b9fff6416c46779dd0bd0cd2db5b2be4c9197ea4f9 Jul 06 19:15:34.916889 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:34.916852 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs" event={"ID":"b361b590-f458-4a0f-ac61-178547d6892c","Type":"ContainerStarted","Data":"16ac17358e656713e953e5b9fff6416c46779dd0bd0cd2db5b2be4c9197ea4f9"} Jul 06 19:15:35.275993 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:35.275958 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:15:35.921654 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:35.921621 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs" event={"ID":"b361b590-f458-4a0f-ac61-178547d6892c","Type":"ContainerStarted","Data":"2e90c73e04768e5735042e20c1e748cb7f450649c3bbc850b1fa0948293269fe"} Jul 06 19:15:35.942498 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:35.942451 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs" podStartSLOduration=1.8763514300000002 podStartE2EDuration="2.942436727s" totalImagesPullingTime="1.066085297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:15:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:15:35.940246152 +0000 UTC m=+611.131613826" watchObservedRunningTime="2026-07-06 19:15:35.942436727 +0000 UTC m=+611.133804399" Jul 06 19:15:43.788198 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:43.788142 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs" Jul 06 19:15:43.788636 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:43.788254 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs" Jul 06 19:15:43.800722 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:43.800693 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs" Jul 06 19:15:43.965867 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:43.965835 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6l6gs" Jul 06 19:15:49.789093 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:49.789016 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67"] Jul 06 19:15:49.791894 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:49.791877 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67" Jul 06 19:15:49.794760 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:49.794742 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 06 19:15:49.801409 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:49.801389 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67"] Jul 06 19:15:49.843998 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:49.843961 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c99200dd-114d-4f40-8946-608b63e10695-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67\" (UID: \"c99200dd-114d-4f40-8946-608b63e10695\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67" Jul 06 19:15:49.844179 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:49.844016 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z8pr\" (UniqueName: \"kubernetes.io/projected/c99200dd-114d-4f40-8946-608b63e10695-kube-api-access-6z8pr\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67\" (UID: \"c99200dd-114d-4f40-8946-608b63e10695\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67" Jul 06 19:15:49.844179 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:49.844071 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c99200dd-114d-4f40-8946-608b63e10695-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67\" (UID: \"c99200dd-114d-4f40-8946-608b63e10695\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67" Jul 06 19:15:49.844179 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:49.844124 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c99200dd-114d-4f40-8946-608b63e10695-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67\" (UID: \"c99200dd-114d-4f40-8946-608b63e10695\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67" Jul 06 19:15:49.844179 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:49.844154 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c99200dd-114d-4f40-8946-608b63e10695-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67\" (UID: \"c99200dd-114d-4f40-8946-608b63e10695\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67" Jul 06 19:15:49.844179 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:49.844176 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c99200dd-114d-4f40-8946-608b63e10695-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67\" (UID: \"c99200dd-114d-4f40-8946-608b63e10695\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67" Jul 06 19:15:49.945293 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:49.945255 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c99200dd-114d-4f40-8946-608b63e10695-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67\" (UID: \"c99200dd-114d-4f40-8946-608b63e10695\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67" Jul 06 19:15:49.945293 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:49.945293 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c99200dd-114d-4f40-8946-608b63e10695-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67\" (UID: \"c99200dd-114d-4f40-8946-608b63e10695\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67" Jul 06 19:15:49.945533 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:49.945314 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c99200dd-114d-4f40-8946-608b63e10695-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67\" (UID: \"c99200dd-114d-4f40-8946-608b63e10695\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67" Jul 06 19:15:49.945533 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:49.945340 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c99200dd-114d-4f40-8946-608b63e10695-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67\" (UID: \"c99200dd-114d-4f40-8946-608b63e10695\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67" Jul 06 19:15:49.945682 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:49.945531 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z8pr\" (UniqueName: \"kubernetes.io/projected/c99200dd-114d-4f40-8946-608b63e10695-kube-api-access-6z8pr\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67\" (UID: \"c99200dd-114d-4f40-8946-608b63e10695\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67" Jul 06 19:15:49.945682 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:49.945634 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c99200dd-114d-4f40-8946-608b63e10695-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67\" (UID: \"c99200dd-114d-4f40-8946-608b63e10695\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67" Jul 06 19:15:49.945851 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:49.945829 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c99200dd-114d-4f40-8946-608b63e10695-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67\" (UID: \"c99200dd-114d-4f40-8946-608b63e10695\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67" Jul 06 19:15:49.945907 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:49.945874 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c99200dd-114d-4f40-8946-608b63e10695-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67\" (UID: \"c99200dd-114d-4f40-8946-608b63e10695\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67" Jul 06 19:15:49.945985 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:49.945967 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c99200dd-114d-4f40-8946-608b63e10695-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67\" (UID: \"c99200dd-114d-4f40-8946-608b63e10695\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67" Jul 06 19:15:49.947590 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:49.947552 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c99200dd-114d-4f40-8946-608b63e10695-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67\" (UID: \"c99200dd-114d-4f40-8946-608b63e10695\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67" Jul 06 19:15:49.948009 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:49.947987 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c99200dd-114d-4f40-8946-608b63e10695-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67\" (UID: \"c99200dd-114d-4f40-8946-608b63e10695\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67" Jul 06 19:15:49.953476 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:49.953451 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z8pr\" (UniqueName: \"kubernetes.io/projected/c99200dd-114d-4f40-8946-608b63e10695-kube-api-access-6z8pr\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67\" (UID: \"c99200dd-114d-4f40-8946-608b63e10695\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67" Jul 06 19:15:50.102008 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:50.101916 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67" Jul 06 19:15:50.234985 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:50.234956 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67"] Jul 06 19:15:50.237337 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:15:50.237310 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc99200dd_114d_4f40_8946_608b63e10695.slice/crio-3684b9850b40d8429282e0d521163ea5f4f8cc84d18346ce30457e92798de20a WatchSource:0}: Error finding container 3684b9850b40d8429282e0d521163ea5f4f8cc84d18346ce30457e92798de20a: Status 404 returned error can't find the container with id 3684b9850b40d8429282e0d521163ea5f4f8cc84d18346ce30457e92798de20a Jul 06 19:15:50.238716 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:50.238694 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 19:15:50.985783 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:50.985744 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67" event={"ID":"c99200dd-114d-4f40-8946-608b63e10695","Type":"ContainerStarted","Data":"0fb4e3465c62d2da7fb0ebc24e5989cec484442c7c906cf59ece9148a0b4b507"} Jul 06 19:15:50.985783 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:50.985785 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67" event={"ID":"c99200dd-114d-4f40-8946-608b63e10695","Type":"ContainerStarted","Data":"3684b9850b40d8429282e0d521163ea5f4f8cc84d18346ce30457e92798de20a"} Jul 06 19:15:51.001712 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:51.001659 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67" podStartSLOduration=1.796632872 podStartE2EDuration="2.001638747s" totalImagesPullingTime="205.005875ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:15:51.001354555 +0000 UTC m=+626.192722227" watchObservedRunningTime="2026-07-06 19:15:51.001638747 +0000 UTC m=+626.193006421" Jul 06 19:15:52.278767 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:15:52.278735 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:16:00.103044 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:16:00.103003 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67" Jul 06 19:16:00.103480 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:16:00.103056 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67" Jul 06 19:16:00.115605 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:16:00.115564 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67" Jul 06 19:16:01.037955 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:16:01.037924 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67" Jul 06 19:16:53.024052 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:16:53.024015 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b65467f89-px8rm"] Jul 06 19:16:53.027169 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:16:53.027150 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b65467f89-px8rm" Jul 06 19:16:53.031311 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:16:53.031289 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mfwsm\"" Jul 06 19:16:53.031311 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:16:53.031299 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 06 19:16:53.039038 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:16:53.039012 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b65467f89-px8rm"] Jul 06 19:16:53.172346 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:16:53.172305 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/305566d2-d32b-4527-b87e-4e96c8fa0c3b-tls-cert\") pod \"authorino-5b65467f89-px8rm\" (UID: \"305566d2-d32b-4527-b87e-4e96c8fa0c3b\") " pod="kuadrant-system/authorino-5b65467f89-px8rm" Jul 06 19:16:53.172524 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:16:53.172353 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5jl6\" (UniqueName: \"kubernetes.io/projected/305566d2-d32b-4527-b87e-4e96c8fa0c3b-kube-api-access-s5jl6\") pod \"authorino-5b65467f89-px8rm\" (UID: \"305566d2-d32b-4527-b87e-4e96c8fa0c3b\") " pod="kuadrant-system/authorino-5b65467f89-px8rm" Jul 06 19:16:53.273681 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:16:53.273642 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/305566d2-d32b-4527-b87e-4e96c8fa0c3b-tls-cert\") pod \"authorino-5b65467f89-px8rm\" (UID: \"305566d2-d32b-4527-b87e-4e96c8fa0c3b\") " pod="kuadrant-system/authorino-5b65467f89-px8rm" Jul 06 19:16:53.273681 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:16:53.273693 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5jl6\" (UniqueName: \"kubernetes.io/projected/305566d2-d32b-4527-b87e-4e96c8fa0c3b-kube-api-access-s5jl6\") pod \"authorino-5b65467f89-px8rm\" (UID: \"305566d2-d32b-4527-b87e-4e96c8fa0c3b\") " pod="kuadrant-system/authorino-5b65467f89-px8rm" Jul 06 19:16:53.276304 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:16:53.276252 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/305566d2-d32b-4527-b87e-4e96c8fa0c3b-tls-cert\") pod \"authorino-5b65467f89-px8rm\" (UID: \"305566d2-d32b-4527-b87e-4e96c8fa0c3b\") " pod="kuadrant-system/authorino-5b65467f89-px8rm" Jul 06 19:16:53.282386 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:16:53.282349 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5jl6\" (UniqueName: \"kubernetes.io/projected/305566d2-d32b-4527-b87e-4e96c8fa0c3b-kube-api-access-s5jl6\") pod \"authorino-5b65467f89-px8rm\" (UID: \"305566d2-d32b-4527-b87e-4e96c8fa0c3b\") " pod="kuadrant-system/authorino-5b65467f89-px8rm" Jul 06 19:16:53.336289 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:16:53.336257 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b65467f89-px8rm" Jul 06 19:16:53.478145 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:16:53.478113 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b65467f89-px8rm"] Jul 06 19:16:53.481297 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:16:53.481270 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod305566d2_d32b_4527_b87e_4e96c8fa0c3b.slice/crio-4568f572697e7b8c7464945b5b58d6ffe756e68a474ecb476c77c0ce8753c1d8 WatchSource:0}: Error finding container 4568f572697e7b8c7464945b5b58d6ffe756e68a474ecb476c77c0ce8753c1d8: Status 404 returned error can't find the container with id 4568f572697e7b8c7464945b5b58d6ffe756e68a474ecb476c77c0ce8753c1d8 Jul 06 19:16:54.245631 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:16:54.245594 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b65467f89-px8rm" event={"ID":"305566d2-d32b-4527-b87e-4e96c8fa0c3b","Type":"ContainerStarted","Data":"0973c8e4162ff0d69d0f6bf156d855f7e7f1af53a4cb60f459fb03f624110402"} Jul 06 19:16:54.246053 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:16:54.245643 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b65467f89-px8rm" event={"ID":"305566d2-d32b-4527-b87e-4e96c8fa0c3b","Type":"ContainerStarted","Data":"4568f572697e7b8c7464945b5b58d6ffe756e68a474ecb476c77c0ce8753c1d8"} Jul 06 19:16:54.262586 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:16:54.262530 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5b65467f89-px8rm" podStartSLOduration=0.943150991 podStartE2EDuration="1.262510484s" totalImagesPullingTime="319.359493ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:16:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:16:54.262191462 +0000 UTC m=+689.453559125" watchObservedRunningTime="2026-07-06 19:16:54.262510484 +0000 UTC m=+689.453878150" Jul 06 19:17:22.780442 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:17:22.780352 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:17:33.774179 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:17:33.774147 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:17:42.785966 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:17:42.785925 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:17:52.476326 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:17:52.476272 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:18:03.771642 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:18:03.771603 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:19:10.882376 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:19:10.882343 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:19:19.879254 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:19:19.879215 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:19:25.285969 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:19:25.285937 2560 scope.go:117] "RemoveContainer" containerID="91665082190d7b5f1dbe8c64fcdffaa80250cba889ca169c1b1edac2bcc170d7" Jul 06 19:19:28.373880 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:19:28.373836 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:19:37.584684 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:19:37.584646 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:19:48.181204 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:19:48.181139 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:19:56.878145 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:19:56.878110 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:20:09.175603 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:20:09.175559 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:20:18.772969 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:20:18.772885 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:20:41.580729 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:20:41.580690 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:20:50.177090 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:20:50.177056 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:21:14.270826 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:21:14.270786 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:21:25.874153 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:21:25.874122 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:21:41.080534 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:21:41.080497 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:21:51.372517 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:21:51.372479 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:22:00.472944 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:22:00.472898 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:22:09.378734 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:22:09.378692 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:22:19.074558 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:22:19.074515 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:24:09.978504 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:24:09.978468 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:24:20.876807 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:24:20.876770 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:24:30.680243 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:24:30.680204 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:24:39.573450 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:24:39.573411 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:24:48.276510 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:24:48.276429 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:24:56.074357 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:24:56.074314 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:25:04.572424 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:25:04.572396 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:25:13.371647 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:25:13.371612 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:25:23.773533 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:25:23.773498 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:25:34.971999 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:25:34.971964 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:25:42.768467 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:25:42.768433 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:25:51.368331 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:25:51.368293 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:26:00.474248 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:26:00.474205 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:26:18.779715 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:26:18.779636 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:26:27.588071 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:26:27.588034 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:26:38.475154 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:26:38.475114 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:26:50.281932 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:26:50.281891 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:27:15.976550 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:27:15.976511 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:27:26.078809 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:27:26.078770 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:27:33.878928 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:27:33.878896 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:27:42.975758 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:27:42.975724 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:27:51.876374 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:27:51.876336 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:28:21.380477 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:28:21.380439 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:28:30.676277 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:28:30.676237 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:28:41.684128 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:28:41.684090 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:28:50.976466 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:28:50.976422 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:29:00.971829 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:29:00.971790 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:29:11.281869 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:29:11.281831 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:29:17.570979 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:29:17.570894 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:29:32.689898 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:29:32.689862 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:29:43.080167 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:29:43.080134 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:29:54.276505 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:29:54.276469 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:30:04.568969 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:30:04.568936 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:30:19.089593 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:30:19.089522 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:30:30.279486 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:30:30.279456 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:30:38.188810 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:30:38.188774 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:30:53.182850 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:30:53.182815 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:30:59.575492 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:30:59.575456 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:31:09.883242 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:31:09.883211 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:31:17.184148 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:31:17.184106 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:31:29.369018 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:31:29.368983 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:31:36.277513 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:31:36.277484 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:32:57.973162 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:32:57.973118 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:33:05.078747 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:33:05.078706 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:33:13.376714 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:33:13.376672 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:33:24.375921 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:33:24.375886 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:33:35.179593 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:33:35.179549 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:34:01.178124 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:34:01.178086 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:34:14.874702 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:34:14.874666 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:34:34.476927 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:34:34.476895 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:34:48.577804 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:34:48.577764 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:35:28.176722 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:35:28.176691 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:35:41.787993 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:35:41.787955 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:35:52.581838 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:35:52.581794 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:36:42.277807 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:36:42.277772 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:36:55.778303 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:36:55.778270 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:37:00.196185 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:00.196152 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d"] Jul 06 19:37:00.199359 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:00.199343 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" Jul 06 19:37:00.203519 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:00.203493 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c6176e\"/\"openshift-service-ca.crt\"" Jul 06 19:37:00.203669 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:00.203500 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c6176e\"/\"kube-root-ca.crt\"" Jul 06 19:37:00.203669 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:00.203564 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c6176e\"/\"default-dockercfg-78wv8\"" Jul 06 19:37:00.203669 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:00.203567 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c6176e\"/\"test-model-c6176e-kserve-self-signed-certs\"" Jul 06 19:37:00.210711 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:00.210689 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d"] Jul 06 19:37:00.296983 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:00.296950 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7db336cc-9388-48b1-a4c4-1908d42b4a7b-home\") pod \"test-model-c6176e-kserve-5f798979cc-z554d\" (UID: \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\") " pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" Jul 06 19:37:00.296983 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:00.296995 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7db336cc-9388-48b1-a4c4-1908d42b4a7b-dshm\") pod \"test-model-c6176e-kserve-5f798979cc-z554d\" (UID: \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\") " pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" Jul 06 19:37:00.297207 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:00.297022 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7db336cc-9388-48b1-a4c4-1908d42b4a7b-tls-certs\") pod \"test-model-c6176e-kserve-5f798979cc-z554d\" (UID: \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\") " pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" Jul 06 19:37:00.297207 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:00.297077 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7db336cc-9388-48b1-a4c4-1908d42b4a7b-tmp-dir\") pod \"test-model-c6176e-kserve-5f798979cc-z554d\" (UID: \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\") " pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" Jul 06 19:37:00.297207 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:00.297139 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5q94\" (UniqueName: \"kubernetes.io/projected/7db336cc-9388-48b1-a4c4-1908d42b4a7b-kube-api-access-l5q94\") pod \"test-model-c6176e-kserve-5f798979cc-z554d\" (UID: \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\") " pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" Jul 06 19:37:00.297207 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:00.297178 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7db336cc-9388-48b1-a4c4-1908d42b4a7b-model-cache\") pod \"test-model-c6176e-kserve-5f798979cc-z554d\" (UID: \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\") " pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" Jul 06 19:37:00.397778 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:00.397743 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5q94\" (UniqueName: \"kubernetes.io/projected/7db336cc-9388-48b1-a4c4-1908d42b4a7b-kube-api-access-l5q94\") pod \"test-model-c6176e-kserve-5f798979cc-z554d\" (UID: \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\") " pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" Jul 06 19:37:00.397963 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:00.397785 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7db336cc-9388-48b1-a4c4-1908d42b4a7b-model-cache\") pod \"test-model-c6176e-kserve-5f798979cc-z554d\" (UID: \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\") " pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" Jul 06 19:37:00.397963 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:00.397815 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7db336cc-9388-48b1-a4c4-1908d42b4a7b-home\") pod \"test-model-c6176e-kserve-5f798979cc-z554d\" (UID: \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\") " pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" Jul 06 19:37:00.397963 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:00.397839 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7db336cc-9388-48b1-a4c4-1908d42b4a7b-dshm\") pod \"test-model-c6176e-kserve-5f798979cc-z554d\" (UID: \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\") " pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" Jul 06 19:37:00.397963 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:00.397860 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7db336cc-9388-48b1-a4c4-1908d42b4a7b-tls-certs\") pod \"test-model-c6176e-kserve-5f798979cc-z554d\" (UID: \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\") " pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" Jul 06 19:37:00.397963 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:00.397892 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7db336cc-9388-48b1-a4c4-1908d42b4a7b-tmp-dir\") pod \"test-model-c6176e-kserve-5f798979cc-z554d\" (UID: \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\") " pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" Jul 06 19:37:00.398263 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:00.398240 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7db336cc-9388-48b1-a4c4-1908d42b4a7b-home\") pod \"test-model-c6176e-kserve-5f798979cc-z554d\" (UID: \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\") " pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" Jul 06 19:37:00.398263 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:00.398257 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7db336cc-9388-48b1-a4c4-1908d42b4a7b-tmp-dir\") pod \"test-model-c6176e-kserve-5f798979cc-z554d\" (UID: \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\") " pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" Jul 06 19:37:00.398400 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:00.398303 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7db336cc-9388-48b1-a4c4-1908d42b4a7b-model-cache\") pod \"test-model-c6176e-kserve-5f798979cc-z554d\" (UID: \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\") " pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" Jul 06 19:37:00.400093 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:00.400072 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7db336cc-9388-48b1-a4c4-1908d42b4a7b-dshm\") pod \"test-model-c6176e-kserve-5f798979cc-z554d\" (UID: \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\") " pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" Jul 06 19:37:00.400455 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:00.400436 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7db336cc-9388-48b1-a4c4-1908d42b4a7b-tls-certs\") pod \"test-model-c6176e-kserve-5f798979cc-z554d\" (UID: \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\") " pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" Jul 06 19:37:00.406327 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:00.406302 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5q94\" (UniqueName: \"kubernetes.io/projected/7db336cc-9388-48b1-a4c4-1908d42b4a7b-kube-api-access-l5q94\") pod \"test-model-c6176e-kserve-5f798979cc-z554d\" (UID: \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\") " pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" Jul 06 19:37:00.509356 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:00.509263 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" Jul 06 19:37:00.643464 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:00.643432 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d"] Jul 06 19:37:00.646670 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:37:00.646646 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7db336cc_9388_48b1_a4c4_1908d42b4a7b.slice/crio-a8d5aefa495e7d0f81896ef885727c2e21c12cace3ba7f8cfd196ba95cac6e90 WatchSource:0}: Error finding container a8d5aefa495e7d0f81896ef885727c2e21c12cace3ba7f8cfd196ba95cac6e90: Status 404 returned error can't find the container with id a8d5aefa495e7d0f81896ef885727c2e21c12cace3ba7f8cfd196ba95cac6e90 Jul 06 19:37:01.231351 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:01.231321 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" event={"ID":"7db336cc-9388-48b1-a4c4-1908d42b4a7b","Type":"ContainerStarted","Data":"fca8cb58877894816bd8fb4f7f8c1186444f247cefee58bd98e3d09654a4eb9d"} Jul 06 19:37:01.231834 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:01.231358 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" event={"ID":"7db336cc-9388-48b1-a4c4-1908d42b4a7b","Type":"ContainerStarted","Data":"a8d5aefa495e7d0f81896ef885727c2e21c12cace3ba7f8cfd196ba95cac6e90"} Jul 06 19:37:01.247302 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:01.247253 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" podStartSLOduration=1.247237451 podStartE2EDuration="1.247237451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:37:01.24668486 +0000 UTC m=+1896.438052534" watchObservedRunningTime="2026-07-06 19:37:01.247237451 +0000 UTC m=+1896.438605123" Jul 06 19:37:04.676987 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:04.676953 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:37:10.509980 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:10.509941 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" Jul 06 19:37:10.509980 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:10.509991 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" Jul 06 19:37:10.522231 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:10.522200 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" Jul 06 19:37:11.284322 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:11.284295 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" Jul 06 19:37:17.483665 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:17.483621 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:37:26.678013 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:26.677973 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:37:40.069412 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:40.069378 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:37:57.979128 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:37:57.979091 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:38:04.987885 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:04.987853 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq"] Jul 06 19:38:04.990983 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:04.990966 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" Jul 06 19:38:04.995694 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:04.995671 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-reuse-82e679\"/\"test-model-82e679-kserve-self-signed-certs\"" Jul 06 19:38:04.995694 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:04.995683 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-reuse-82e679\"/\"kube-root-ca.crt\"" Jul 06 19:38:04.995888 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:04.995707 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-reuse-82e679\"/\"default-dockercfg-6zc4r\"" Jul 06 19:38:04.995888 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:04.995723 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-reuse-82e679\"/\"openshift-service-ca.crt\"" Jul 06 19:38:05.002821 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:05.002798 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq"] Jul 06 19:38:05.146090 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:05.146035 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7434629a-c01f-4b97-a19f-5c2ae3a7006a-tmp-dir\") pod \"test-model-82e679-kserve-6b6f4dbdc-4djvq\" (UID: \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\") " pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" Jul 06 19:38:05.146282 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:05.146108 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7434629a-c01f-4b97-a19f-5c2ae3a7006a-tls-certs\") pod \"test-model-82e679-kserve-6b6f4dbdc-4djvq\" (UID: \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\") " pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" Jul 06 19:38:05.146282 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:05.146143 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwf76\" (UniqueName: \"kubernetes.io/projected/7434629a-c01f-4b97-a19f-5c2ae3a7006a-kube-api-access-gwf76\") pod \"test-model-82e679-kserve-6b6f4dbdc-4djvq\" (UID: \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\") " pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" Jul 06 19:38:05.146282 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:05.146169 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7434629a-c01f-4b97-a19f-5c2ae3a7006a-dshm\") pod \"test-model-82e679-kserve-6b6f4dbdc-4djvq\" (UID: \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\") " pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" Jul 06 19:38:05.146282 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:05.146215 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7434629a-c01f-4b97-a19f-5c2ae3a7006a-home\") pod \"test-model-82e679-kserve-6b6f4dbdc-4djvq\" (UID: \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\") " pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" Jul 06 19:38:05.146282 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:05.146229 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7434629a-c01f-4b97-a19f-5c2ae3a7006a-model-cache\") pod \"test-model-82e679-kserve-6b6f4dbdc-4djvq\" (UID: \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\") " pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" Jul 06 19:38:05.246871 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:05.246782 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7434629a-c01f-4b97-a19f-5c2ae3a7006a-tmp-dir\") pod \"test-model-82e679-kserve-6b6f4dbdc-4djvq\" (UID: \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\") " pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" Jul 06 19:38:05.246871 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:05.246851 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7434629a-c01f-4b97-a19f-5c2ae3a7006a-tls-certs\") pod \"test-model-82e679-kserve-6b6f4dbdc-4djvq\" (UID: \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\") " pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" Jul 06 19:38:05.247096 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:05.246904 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwf76\" (UniqueName: \"kubernetes.io/projected/7434629a-c01f-4b97-a19f-5c2ae3a7006a-kube-api-access-gwf76\") pod \"test-model-82e679-kserve-6b6f4dbdc-4djvq\" (UID: \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\") " pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" Jul 06 19:38:05.247096 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:05.246949 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7434629a-c01f-4b97-a19f-5c2ae3a7006a-dshm\") pod \"test-model-82e679-kserve-6b6f4dbdc-4djvq\" (UID: \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\") " pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" Jul 06 19:38:05.247096 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:05.246977 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7434629a-c01f-4b97-a19f-5c2ae3a7006a-home\") pod \"test-model-82e679-kserve-6b6f4dbdc-4djvq\" (UID: \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\") " pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" Jul 06 19:38:05.247096 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:05.247000 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7434629a-c01f-4b97-a19f-5c2ae3a7006a-model-cache\") pod \"test-model-82e679-kserve-6b6f4dbdc-4djvq\" (UID: \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\") " pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" Jul 06 19:38:05.247282 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:05.247234 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7434629a-c01f-4b97-a19f-5c2ae3a7006a-tmp-dir\") pod \"test-model-82e679-kserve-6b6f4dbdc-4djvq\" (UID: \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\") " pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" Jul 06 19:38:05.247382 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:05.247361 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7434629a-c01f-4b97-a19f-5c2ae3a7006a-home\") pod \"test-model-82e679-kserve-6b6f4dbdc-4djvq\" (UID: \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\") " pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" Jul 06 19:38:05.247438 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:05.247382 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7434629a-c01f-4b97-a19f-5c2ae3a7006a-model-cache\") pod \"test-model-82e679-kserve-6b6f4dbdc-4djvq\" (UID: \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\") " pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" Jul 06 19:38:05.249185 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:05.249159 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7434629a-c01f-4b97-a19f-5c2ae3a7006a-dshm\") pod \"test-model-82e679-kserve-6b6f4dbdc-4djvq\" (UID: \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\") " pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" Jul 06 19:38:05.249460 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:05.249443 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7434629a-c01f-4b97-a19f-5c2ae3a7006a-tls-certs\") pod \"test-model-82e679-kserve-6b6f4dbdc-4djvq\" (UID: \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\") " pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" Jul 06 19:38:05.256007 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:05.255983 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwf76\" (UniqueName: \"kubernetes.io/projected/7434629a-c01f-4b97-a19f-5c2ae3a7006a-kube-api-access-gwf76\") pod \"test-model-82e679-kserve-6b6f4dbdc-4djvq\" (UID: \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\") " pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" Jul 06 19:38:05.300248 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:05.300211 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" Jul 06 19:38:05.429958 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:05.429922 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq"] Jul 06 19:38:05.432881 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:38:05.432854 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7434629a_c01f_4b97_a19f_5c2ae3a7006a.slice/crio-e98ae277a3c0819d2e94f6c5bbbdc3322ab61bcff2356cbd7cd9e6aecbe04fe2 WatchSource:0}: Error finding container e98ae277a3c0819d2e94f6c5bbbdc3322ab61bcff2356cbd7cd9e6aecbe04fe2: Status 404 returned error can't find the container with id e98ae277a3c0819d2e94f6c5bbbdc3322ab61bcff2356cbd7cd9e6aecbe04fe2 Jul 06 19:38:05.500332 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:05.500251 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" event={"ID":"7434629a-c01f-4b97-a19f-5c2ae3a7006a","Type":"ContainerStarted","Data":"5d24968b2ba5b88a5cd334f017ee64fe8497d93d88d870af2c0614e606b1d462"} Jul 06 19:38:05.500332 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:05.500293 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" event={"ID":"7434629a-c01f-4b97-a19f-5c2ae3a7006a","Type":"ContainerStarted","Data":"e98ae277a3c0819d2e94f6c5bbbdc3322ab61bcff2356cbd7cd9e6aecbe04fe2"} Jul 06 19:38:05.518673 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:05.518610 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" podStartSLOduration=1.5185656669999998 podStartE2EDuration="1.518565667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:38:05.517694108 +0000 UTC m=+1960.709061780" watchObservedRunningTime="2026-07-06 19:38:05.518565667 +0000 UTC m=+1960.709933379" Jul 06 19:38:13.283291 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:13.283249 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:38:15.300754 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:15.300713 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" Jul 06 19:38:15.301200 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:15.300774 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" Jul 06 19:38:15.313220 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:15.313199 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" Jul 06 19:38:15.553463 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:15.553379 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" Jul 06 19:38:27.261242 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:27.261206 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq"] Jul 06 19:38:30.572146 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:30.572106 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:38:42.281089 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.281043 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" podUID="7434629a-c01f-4b97-a19f-5c2ae3a7006a" containerName="main" containerID="cri-o://5d24968b2ba5b88a5cd334f017ee64fe8497d93d88d870af2c0614e606b1d462" gracePeriod=45 Jul 06 19:38:42.527210 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.527183 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" Jul 06 19:38:42.649093 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.649001 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7434629a-c01f-4b97-a19f-5c2ae3a7006a-tls-certs\") pod \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\" (UID: \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\") " Jul 06 19:38:42.649093 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.649052 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7434629a-c01f-4b97-a19f-5c2ae3a7006a-home\") pod \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\" (UID: \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\") " Jul 06 19:38:42.649318 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.649103 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7434629a-c01f-4b97-a19f-5c2ae3a7006a-tmp-dir\") pod \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\" (UID: \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\") " Jul 06 19:38:42.649318 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.649142 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7434629a-c01f-4b97-a19f-5c2ae3a7006a-dshm\") pod \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\" (UID: \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\") " Jul 06 19:38:42.649318 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.649172 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwf76\" (UniqueName: \"kubernetes.io/projected/7434629a-c01f-4b97-a19f-5c2ae3a7006a-kube-api-access-gwf76\") pod \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\" (UID: \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\") " Jul 06 19:38:42.649318 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.649222 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7434629a-c01f-4b97-a19f-5c2ae3a7006a-model-cache\") pod \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\" (UID: \"7434629a-c01f-4b97-a19f-5c2ae3a7006a\") " Jul 06 19:38:42.649531 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.649354 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7434629a-c01f-4b97-a19f-5c2ae3a7006a-home" (OuterVolumeSpecName: "home") pod "7434629a-c01f-4b97-a19f-5c2ae3a7006a" (UID: "7434629a-c01f-4b97-a19f-5c2ae3a7006a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:38:42.649531 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.649373 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7434629a-c01f-4b97-a19f-5c2ae3a7006a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7434629a-c01f-4b97-a19f-5c2ae3a7006a" (UID: "7434629a-c01f-4b97-a19f-5c2ae3a7006a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:38:42.649667 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.649535 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7434629a-c01f-4b97-a19f-5c2ae3a7006a-model-cache" (OuterVolumeSpecName: "model-cache") pod "7434629a-c01f-4b97-a19f-5c2ae3a7006a" (UID: "7434629a-c01f-4b97-a19f-5c2ae3a7006a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:38:42.649667 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.649552 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7434629a-c01f-4b97-a19f-5c2ae3a7006a-home\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:38:42.649667 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.649617 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7434629a-c01f-4b97-a19f-5c2ae3a7006a-tmp-dir\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:38:42.651339 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.651313 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7434629a-c01f-4b97-a19f-5c2ae3a7006a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7434629a-c01f-4b97-a19f-5c2ae3a7006a" (UID: "7434629a-c01f-4b97-a19f-5c2ae3a7006a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:38:42.651454 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.651345 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7434629a-c01f-4b97-a19f-5c2ae3a7006a-dshm" (OuterVolumeSpecName: "dshm") pod "7434629a-c01f-4b97-a19f-5c2ae3a7006a" (UID: "7434629a-c01f-4b97-a19f-5c2ae3a7006a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:38:42.651454 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.651363 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7434629a-c01f-4b97-a19f-5c2ae3a7006a-kube-api-access-gwf76" (OuterVolumeSpecName: "kube-api-access-gwf76") pod "7434629a-c01f-4b97-a19f-5c2ae3a7006a" (UID: "7434629a-c01f-4b97-a19f-5c2ae3a7006a"). InnerVolumeSpecName "kube-api-access-gwf76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:38:42.656080 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.656056 2560 generic.go:356] "Generic (PLEG): container finished" podID="7434629a-c01f-4b97-a19f-5c2ae3a7006a" containerID="5d24968b2ba5b88a5cd334f017ee64fe8497d93d88d870af2c0614e606b1d462" exitCode=0 Jul 06 19:38:42.656319 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.656110 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" Jul 06 19:38:42.656319 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.656109 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" event={"ID":"7434629a-c01f-4b97-a19f-5c2ae3a7006a","Type":"ContainerDied","Data":"5d24968b2ba5b88a5cd334f017ee64fe8497d93d88d870af2c0614e606b1d462"} Jul 06 19:38:42.656319 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.656155 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq" event={"ID":"7434629a-c01f-4b97-a19f-5c2ae3a7006a","Type":"ContainerDied","Data":"e98ae277a3c0819d2e94f6c5bbbdc3322ab61bcff2356cbd7cd9e6aecbe04fe2"} Jul 06 19:38:42.656319 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.656176 2560 scope.go:117] "RemoveContainer" containerID="5d24968b2ba5b88a5cd334f017ee64fe8497d93d88d870af2c0614e606b1d462" Jul 06 19:38:42.667498 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.667472 2560 scope.go:117] "RemoveContainer" containerID="5d24968b2ba5b88a5cd334f017ee64fe8497d93d88d870af2c0614e606b1d462" Jul 06 19:38:42.667820 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:38:42.667801 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d24968b2ba5b88a5cd334f017ee64fe8497d93d88d870af2c0614e606b1d462\": container with ID starting with 5d24968b2ba5b88a5cd334f017ee64fe8497d93d88d870af2c0614e606b1d462 not found: ID does not exist" containerID="5d24968b2ba5b88a5cd334f017ee64fe8497d93d88d870af2c0614e606b1d462" Jul 06 19:38:42.667893 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.667828 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d24968b2ba5b88a5cd334f017ee64fe8497d93d88d870af2c0614e606b1d462"} err="failed to get container status \"5d24968b2ba5b88a5cd334f017ee64fe8497d93d88d870af2c0614e606b1d462\": rpc error: code = NotFound desc = could not find container \"5d24968b2ba5b88a5cd334f017ee64fe8497d93d88d870af2c0614e606b1d462\": container with ID starting with 5d24968b2ba5b88a5cd334f017ee64fe8497d93d88d870af2c0614e606b1d462 not found: ID does not exist" Jul 06 19:38:42.679458 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.679432 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq"] Jul 06 19:38:42.683323 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.683300 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-reuse-82e679/test-model-82e679-kserve-6b6f4dbdc-4djvq"] Jul 06 19:38:42.750565 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.750528 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7434629a-c01f-4b97-a19f-5c2ae3a7006a-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:38:42.750565 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.750561 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7434629a-c01f-4b97-a19f-5c2ae3a7006a-dshm\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:38:42.750768 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.750598 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwf76\" (UniqueName: \"kubernetes.io/projected/7434629a-c01f-4b97-a19f-5c2ae3a7006a-kube-api-access-gwf76\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:38:42.750768 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:42.750608 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7434629a-c01f-4b97-a19f-5c2ae3a7006a-model-cache\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:38:43.390480 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:38:43.390445 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7434629a-c01f-4b97-a19f-5c2ae3a7006a" path="/var/lib/kubelet/pods/7434629a-c01f-4b97-a19f-5c2ae3a7006a/volumes" Jul 06 19:39:08.587372 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:08.587336 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:39:11.899690 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:11.899659 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d"] Jul 06 19:39:24.679393 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:24.679361 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:39:26.919258 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:26.919184 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" podUID="7db336cc-9388-48b1-a4c4-1908d42b4a7b" containerName="main" containerID="cri-o://fca8cb58877894816bd8fb4f7f8c1186444f247cefee58bd98e3d09654a4eb9d" gracePeriod=45 Jul 06 19:39:27.164520 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.164494 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" Jul 06 19:39:27.214821 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.214787 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7db336cc-9388-48b1-a4c4-1908d42b4a7b-model-cache\") pod \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\" (UID: \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\") " Jul 06 19:39:27.214821 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.214825 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7db336cc-9388-48b1-a4c4-1908d42b4a7b-tls-certs\") pod \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\" (UID: \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\") " Jul 06 19:39:27.215013 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.214851 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5q94\" (UniqueName: \"kubernetes.io/projected/7db336cc-9388-48b1-a4c4-1908d42b4a7b-kube-api-access-l5q94\") pod \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\" (UID: \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\") " Jul 06 19:39:27.215013 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.214875 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7db336cc-9388-48b1-a4c4-1908d42b4a7b-home\") pod \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\" (UID: \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\") " Jul 06 19:39:27.215013 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.214891 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7db336cc-9388-48b1-a4c4-1908d42b4a7b-dshm\") pod \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\" (UID: \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\") " Jul 06 19:39:27.215013 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.214953 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7db336cc-9388-48b1-a4c4-1908d42b4a7b-tmp-dir\") pod \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\" (UID: \"7db336cc-9388-48b1-a4c4-1908d42b4a7b\") " Jul 06 19:39:27.215195 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.215123 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db336cc-9388-48b1-a4c4-1908d42b4a7b-model-cache" (OuterVolumeSpecName: "model-cache") pod "7db336cc-9388-48b1-a4c4-1908d42b4a7b" (UID: "7db336cc-9388-48b1-a4c4-1908d42b4a7b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:39:27.215195 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.215144 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db336cc-9388-48b1-a4c4-1908d42b4a7b-home" (OuterVolumeSpecName: "home") pod "7db336cc-9388-48b1-a4c4-1908d42b4a7b" (UID: "7db336cc-9388-48b1-a4c4-1908d42b4a7b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:39:27.215320 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.215292 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db336cc-9388-48b1-a4c4-1908d42b4a7b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7db336cc-9388-48b1-a4c4-1908d42b4a7b" (UID: "7db336cc-9388-48b1-a4c4-1908d42b4a7b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:39:27.217109 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.217089 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7db336cc-9388-48b1-a4c4-1908d42b4a7b-dshm" (OuterVolumeSpecName: "dshm") pod "7db336cc-9388-48b1-a4c4-1908d42b4a7b" (UID: "7db336cc-9388-48b1-a4c4-1908d42b4a7b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:39:27.217204 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.217184 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7db336cc-9388-48b1-a4c4-1908d42b4a7b-kube-api-access-l5q94" (OuterVolumeSpecName: "kube-api-access-l5q94") pod "7db336cc-9388-48b1-a4c4-1908d42b4a7b" (UID: "7db336cc-9388-48b1-a4c4-1908d42b4a7b"). InnerVolumeSpecName "kube-api-access-l5q94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:39:27.217258 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.217208 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7db336cc-9388-48b1-a4c4-1908d42b4a7b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7db336cc-9388-48b1-a4c4-1908d42b4a7b" (UID: "7db336cc-9388-48b1-a4c4-1908d42b4a7b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:39:27.315866 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.315831 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7db336cc-9388-48b1-a4c4-1908d42b4a7b-model-cache\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:39:27.315866 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.315861 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7db336cc-9388-48b1-a4c4-1908d42b4a7b-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:39:27.316068 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.315874 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5q94\" (UniqueName: \"kubernetes.io/projected/7db336cc-9388-48b1-a4c4-1908d42b4a7b-kube-api-access-l5q94\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:39:27.316068 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.315887 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7db336cc-9388-48b1-a4c4-1908d42b4a7b-home\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:39:27.316068 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.315897 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7db336cc-9388-48b1-a4c4-1908d42b4a7b-dshm\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:39:27.316068 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.315930 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7db336cc-9388-48b1-a4c4-1908d42b4a7b-tmp-dir\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:39:27.845077 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.845039 2560 generic.go:356] "Generic (PLEG): container finished" podID="7db336cc-9388-48b1-a4c4-1908d42b4a7b" containerID="fca8cb58877894816bd8fb4f7f8c1186444f247cefee58bd98e3d09654a4eb9d" exitCode=0 Jul 06 19:39:27.845254 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.845084 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" Jul 06 19:39:27.845254 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.845082 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" event={"ID":"7db336cc-9388-48b1-a4c4-1908d42b4a7b","Type":"ContainerDied","Data":"fca8cb58877894816bd8fb4f7f8c1186444f247cefee58bd98e3d09654a4eb9d"} Jul 06 19:39:27.845254 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.845121 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d" event={"ID":"7db336cc-9388-48b1-a4c4-1908d42b4a7b","Type":"ContainerDied","Data":"a8d5aefa495e7d0f81896ef885727c2e21c12cace3ba7f8cfd196ba95cac6e90"} Jul 06 19:39:27.845254 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.845150 2560 scope.go:117] "RemoveContainer" containerID="fca8cb58877894816bd8fb4f7f8c1186444f247cefee58bd98e3d09654a4eb9d" Jul 06 19:39:27.853529 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.853508 2560 scope.go:117] "RemoveContainer" containerID="fca8cb58877894816bd8fb4f7f8c1186444f247cefee58bd98e3d09654a4eb9d" Jul 06 19:39:27.853795 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:39:27.853774 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fca8cb58877894816bd8fb4f7f8c1186444f247cefee58bd98e3d09654a4eb9d\": container with ID starting with fca8cb58877894816bd8fb4f7f8c1186444f247cefee58bd98e3d09654a4eb9d not found: ID does not exist" containerID="fca8cb58877894816bd8fb4f7f8c1186444f247cefee58bd98e3d09654a4eb9d" Jul 06 19:39:27.853841 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.853804 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca8cb58877894816bd8fb4f7f8c1186444f247cefee58bd98e3d09654a4eb9d"} err="failed to get container status \"fca8cb58877894816bd8fb4f7f8c1186444f247cefee58bd98e3d09654a4eb9d\": rpc error: code = NotFound desc = could not find container \"fca8cb58877894816bd8fb4f7f8c1186444f247cefee58bd98e3d09654a4eb9d\": container with ID starting with fca8cb58877894816bd8fb4f7f8c1186444f247cefee58bd98e3d09654a4eb9d not found: ID does not exist" Jul 06 19:39:27.861474 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.861453 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d"] Jul 06 19:39:27.866250 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:27.866228 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c6176e/test-model-c6176e-kserve-5f798979cc-z554d"] Jul 06 19:39:29.390775 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:29.390744 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7db336cc-9388-48b1-a4c4-1908d42b4a7b" path="/var/lib/kubelet/pods/7db336cc-9388-48b1-a4c4-1908d42b4a7b/volumes" Jul 06 19:39:59.471465 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:39:59.471420 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:40:12.291009 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:40:12.290977 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:40:23.100008 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:40:23.099968 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:40:40.872763 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:40:40.872726 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:40:51.779484 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:40:51.779445 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:41:07.280019 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:07.279989 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:41:12.692361 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.692329 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r"] Jul 06 19:41:12.692864 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.692738 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7434629a-c01f-4b97-a19f-5c2ae3a7006a" containerName="main" Jul 06 19:41:12.692864 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.692756 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7434629a-c01f-4b97-a19f-5c2ae3a7006a" containerName="main" Jul 06 19:41:12.692864 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.692777 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7db336cc-9388-48b1-a4c4-1908d42b4a7b" containerName="main" Jul 06 19:41:12.692864 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.692785 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7db336cc-9388-48b1-a4c4-1908d42b4a7b" containerName="main" Jul 06 19:41:12.693063 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.692875 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7434629a-c01f-4b97-a19f-5c2ae3a7006a" containerName="main" Jul 06 19:41:12.693063 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.692888 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7db336cc-9388-48b1-a4c4-1908d42b4a7b" containerName="main" Jul 06 19:41:12.695844 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.695820 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" Jul 06 19:41:12.698922 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.698903 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-bd1298\"/\"openshift-service-ca.crt\"" Jul 06 19:41:12.700314 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.700294 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bd1298\"/\"auth-test-model-bd1298-kserve-self-signed-certs\"" Jul 06 19:41:12.700394 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.700294 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bd1298\"/\"default-dockercfg-7glcm\"" Jul 06 19:41:12.700394 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.700331 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-bd1298\"/\"kube-root-ca.crt\"" Jul 06 19:41:12.707242 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.707222 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r"] Jul 06 19:41:12.796783 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.796749 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-tmp-dir\") pod \"auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r\" (UID: \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\") " pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" Jul 06 19:41:12.796989 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.796791 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-model-cache\") pod \"auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r\" (UID: \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\") " pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" Jul 06 19:41:12.796989 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.796814 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-dshm\") pod \"auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r\" (UID: \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\") " pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" Jul 06 19:41:12.796989 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.796901 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-tls-certs\") pod \"auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r\" (UID: \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\") " pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" Jul 06 19:41:12.796989 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.796945 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-home\") pod \"auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r\" (UID: \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\") " pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" Jul 06 19:41:12.796989 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.796972 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpkt8\" (UniqueName: \"kubernetes.io/projected/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-kube-api-access-xpkt8\") pod \"auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r\" (UID: \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\") " pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" Jul 06 19:41:12.897331 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.897294 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-home\") pod \"auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r\" (UID: \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\") " pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" Jul 06 19:41:12.897331 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.897334 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpkt8\" (UniqueName: \"kubernetes.io/projected/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-kube-api-access-xpkt8\") pod \"auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r\" (UID: \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\") " pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" Jul 06 19:41:12.897599 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.897380 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-tmp-dir\") pod \"auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r\" (UID: \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\") " pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" Jul 06 19:41:12.897599 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.897422 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-model-cache\") pod \"auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r\" (UID: \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\") " pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" Jul 06 19:41:12.897599 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.897556 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-dshm\") pod \"auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r\" (UID: \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\") " pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" Jul 06 19:41:12.898041 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.897671 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-tls-certs\") pod \"auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r\" (UID: \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\") " pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" Jul 06 19:41:12.898041 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.897697 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-home\") pod \"auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r\" (UID: \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\") " pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" Jul 06 19:41:12.898041 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.897768 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-tmp-dir\") pod \"auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r\" (UID: \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\") " pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" Jul 06 19:41:12.898041 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.897808 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-model-cache\") pod \"auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r\" (UID: \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\") " pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" Jul 06 19:41:12.899674 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.899654 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-dshm\") pod \"auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r\" (UID: \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\") " pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" Jul 06 19:41:12.900021 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.900005 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-tls-certs\") pod \"auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r\" (UID: \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\") " pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" Jul 06 19:41:12.905645 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:12.905627 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpkt8\" (UniqueName: \"kubernetes.io/projected/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-kube-api-access-xpkt8\") pod \"auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r\" (UID: \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\") " pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" Jul 06 19:41:13.005861 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:13.005821 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" Jul 06 19:41:13.134101 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:13.134079 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r"] Jul 06 19:41:13.136606 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:41:13.136549 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb721bb21_1092_47d9_bfe9_12b3fd8ccd59.slice/crio-fb4d06b68d85f7965365b07382c3a17b21ead8b3ceecef9fa76913c65a3c065b WatchSource:0}: Error finding container fb4d06b68d85f7965365b07382c3a17b21ead8b3ceecef9fa76913c65a3c065b: Status 404 returned error can't find the container with id fb4d06b68d85f7965365b07382c3a17b21ead8b3ceecef9fa76913c65a3c065b Jul 06 19:41:13.282888 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:13.282800 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" event={"ID":"b721bb21-1092-47d9-bfe9-12b3fd8ccd59","Type":"ContainerStarted","Data":"85bd8fc49e17a9d58e9fde2f2f6a25cb489f0f2bbabcefa929c5dfdcc29c8275"} Jul 06 19:41:13.282888 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:13.282850 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" event={"ID":"b721bb21-1092-47d9-bfe9-12b3fd8ccd59","Type":"ContainerStarted","Data":"fb4d06b68d85f7965365b07382c3a17b21ead8b3ceecef9fa76913c65a3c065b"} Jul 06 19:41:13.301334 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:13.301289 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" podStartSLOduration=1.301274081 podStartE2EDuration="1.301274081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:41:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:41:13.300549923 +0000 UTC m=+2148.491917610" watchObservedRunningTime="2026-07-06 19:41:13.301274081 +0000 UTC m=+2148.492641754" Jul 06 19:41:16.478337 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:16.478250 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:41:23.006373 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:23.006341 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" Jul 06 19:41:23.006894 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:23.006608 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" Jul 06 19:41:23.018851 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:23.018829 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" Jul 06 19:41:23.334174 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:23.334093 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" Jul 06 19:41:38.599357 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:38.599324 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:41:51.490224 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:41:51.490189 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:42:06.684216 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:42:06.684182 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:42:20.282387 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:42:20.282353 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:42:36.178446 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:42:36.178407 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:42:50.790442 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:42:50.790364 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:43:06.678867 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:06.678822 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:43:18.129491 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:18.129457 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r"] Jul 06 19:43:23.485716 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:23.485677 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:43:33.149341 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.149294 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" podUID="b721bb21-1092-47d9-bfe9-12b3fd8ccd59" containerName="main" containerID="cri-o://85bd8fc49e17a9d58e9fde2f2f6a25cb489f0f2bbabcefa929c5dfdcc29c8275" gracePeriod=45 Jul 06 19:43:33.388203 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.388178 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" Jul 06 19:43:33.465380 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.465357 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-tmp-dir\") pod \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\" (UID: \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\") " Jul 06 19:43:33.465523 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.465386 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-dshm\") pod \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\" (UID: \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\") " Jul 06 19:43:33.465523 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.465405 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-model-cache\") pod \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\" (UID: \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\") " Jul 06 19:43:33.465523 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.465469 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpkt8\" (UniqueName: \"kubernetes.io/projected/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-kube-api-access-xpkt8\") pod \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\" (UID: \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\") " Jul 06 19:43:33.465523 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.465491 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-tls-certs\") pod \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\" (UID: \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\") " Jul 06 19:43:33.465523 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.465511 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-home\") pod \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\" (UID: \"b721bb21-1092-47d9-bfe9-12b3fd8ccd59\") " Jul 06 19:43:33.465827 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.465709 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b721bb21-1092-47d9-bfe9-12b3fd8ccd59" (UID: "b721bb21-1092-47d9-bfe9-12b3fd8ccd59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:43:33.465827 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.465790 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-model-cache" (OuterVolumeSpecName: "model-cache") pod "b721bb21-1092-47d9-bfe9-12b3fd8ccd59" (UID: "b721bb21-1092-47d9-bfe9-12b3fd8ccd59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:43:33.465931 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.465848 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-home" (OuterVolumeSpecName: "home") pod "b721bb21-1092-47d9-bfe9-12b3fd8ccd59" (UID: "b721bb21-1092-47d9-bfe9-12b3fd8ccd59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:43:33.466063 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.466043 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-tmp-dir\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:43:33.466124 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.466067 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-model-cache\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:43:33.466124 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.466082 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-home\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:43:33.467744 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.467718 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b721bb21-1092-47d9-bfe9-12b3fd8ccd59" (UID: "b721bb21-1092-47d9-bfe9-12b3fd8ccd59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:43:33.467859 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.467773 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-dshm" (OuterVolumeSpecName: "dshm") pod "b721bb21-1092-47d9-bfe9-12b3fd8ccd59" (UID: "b721bb21-1092-47d9-bfe9-12b3fd8ccd59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:43:33.467859 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.467786 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-kube-api-access-xpkt8" (OuterVolumeSpecName: "kube-api-access-xpkt8") pod "b721bb21-1092-47d9-bfe9-12b3fd8ccd59" (UID: "b721bb21-1092-47d9-bfe9-12b3fd8ccd59"). InnerVolumeSpecName "kube-api-access-xpkt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:43:33.566912 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.566883 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-dshm\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:43:33.566912 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.566907 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpkt8\" (UniqueName: \"kubernetes.io/projected/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-kube-api-access-xpkt8\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:43:33.566912 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.566918 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b721bb21-1092-47d9-bfe9-12b3fd8ccd59-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:43:33.880461 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.880368 2560 generic.go:356] "Generic (PLEG): container finished" podID="b721bb21-1092-47d9-bfe9-12b3fd8ccd59" containerID="85bd8fc49e17a9d58e9fde2f2f6a25cb489f0f2bbabcefa929c5dfdcc29c8275" exitCode=0 Jul 06 19:43:33.880461 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.880419 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" Jul 06 19:43:33.880461 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.880437 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" event={"ID":"b721bb21-1092-47d9-bfe9-12b3fd8ccd59","Type":"ContainerDied","Data":"85bd8fc49e17a9d58e9fde2f2f6a25cb489f0f2bbabcefa929c5dfdcc29c8275"} Jul 06 19:43:33.880775 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.880473 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" event={"ID":"b721bb21-1092-47d9-bfe9-12b3fd8ccd59","Type":"ContainerDied","Data":"fb4d06b68d85f7965365b07382c3a17b21ead8b3ceecef9fa76913c65a3c065b"} Jul 06 19:43:33.880775 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.880489 2560 scope.go:117] "RemoveContainer" containerID="85bd8fc49e17a9d58e9fde2f2f6a25cb489f0f2bbabcefa929c5dfdcc29c8275" Jul 06 19:43:33.893275 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.893253 2560 scope.go:117] "RemoveContainer" containerID="85bd8fc49e17a9d58e9fde2f2f6a25cb489f0f2bbabcefa929c5dfdcc29c8275" Jul 06 19:43:33.893586 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:43:33.893554 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85bd8fc49e17a9d58e9fde2f2f6a25cb489f0f2bbabcefa929c5dfdcc29c8275\": container with ID starting with 85bd8fc49e17a9d58e9fde2f2f6a25cb489f0f2bbabcefa929c5dfdcc29c8275 not found: ID does not exist" containerID="85bd8fc49e17a9d58e9fde2f2f6a25cb489f0f2bbabcefa929c5dfdcc29c8275" Jul 06 19:43:33.893654 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.893635 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85bd8fc49e17a9d58e9fde2f2f6a25cb489f0f2bbabcefa929c5dfdcc29c8275"} err="failed to get container status \"85bd8fc49e17a9d58e9fde2f2f6a25cb489f0f2bbabcefa929c5dfdcc29c8275\": rpc error: code = NotFound desc = could not find container \"85bd8fc49e17a9d58e9fde2f2f6a25cb489f0f2bbabcefa929c5dfdcc29c8275\": container with ID starting with 85bd8fc49e17a9d58e9fde2f2f6a25cb489f0f2bbabcefa929c5dfdcc29c8275 not found: ID does not exist" Jul 06 19:43:33.910489 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.910464 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r"] Jul 06 19:43:33.913174 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:33.913154 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r"] Jul 06 19:43:34.901400 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:34.901372 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:43:35.389795 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:35.389766 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b721bb21-1092-47d9-bfe9-12b3fd8ccd59" path="/var/lib/kubelet/pods/b721bb21-1092-47d9-bfe9-12b3fd8ccd59/volumes" Jul 06 19:43:38.324774 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:43:38.324732 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-bd1298/auth-test-model-bd1298-kserve-7fc5fd5d8c-5g59r" podUID="b721bb21-1092-47d9-bfe9-12b3fd8ccd59" containerName="main" probeResult="failure" output="Get \"https://10.133.0.31:8000/ready\": context deadline exceeded" Jul 06 19:44:05.099466 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.099435 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx"] Jul 06 19:44:05.099926 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.099740 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b721bb21-1092-47d9-bfe9-12b3fd8ccd59" containerName="main" Jul 06 19:44:05.099926 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.099753 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b721bb21-1092-47d9-bfe9-12b3fd8ccd59" containerName="main" Jul 06 19:44:05.099926 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.099803 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b721bb21-1092-47d9-bfe9-12b3fd8ccd59" containerName="main" Jul 06 19:44:05.102440 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.102425 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" Jul 06 19:44:05.109728 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.109705 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f2694d\"/\"sub-test-model-f2694d-kserve-self-signed-certs\"" Jul 06 19:44:05.109840 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.109760 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f2694d\"/\"default-dockercfg-jjm5n\"" Jul 06 19:44:05.109893 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.109852 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f2694d\"/\"kube-root-ca.crt\"" Jul 06 19:44:05.110049 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.110033 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f2694d\"/\"openshift-service-ca.crt\"" Jul 06 19:44:05.122770 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.122745 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-home\") pod \"sub-test-model-f2694d-kserve-f78998949-2hrpx\" (UID: \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\") " pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" Jul 06 19:44:05.122900 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.122776 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-tls-certs\") pod \"sub-test-model-f2694d-kserve-f78998949-2hrpx\" (UID: \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\") " pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" Jul 06 19:44:05.122900 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.122804 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-model-cache\") pod \"sub-test-model-f2694d-kserve-f78998949-2hrpx\" (UID: \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\") " pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" Jul 06 19:44:05.122900 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.122819 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-tmp-dir\") pod \"sub-test-model-f2694d-kserve-f78998949-2hrpx\" (UID: \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\") " pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" Jul 06 19:44:05.122900 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.122887 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-dshm\") pod \"sub-test-model-f2694d-kserve-f78998949-2hrpx\" (UID: \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\") " pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" Jul 06 19:44:05.123113 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.122929 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dml6d\" (UniqueName: \"kubernetes.io/projected/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-kube-api-access-dml6d\") pod \"sub-test-model-f2694d-kserve-f78998949-2hrpx\" (UID: \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\") " pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" Jul 06 19:44:05.137322 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.137297 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx"] Jul 06 19:44:05.223698 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.223665 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-dshm\") pod \"sub-test-model-f2694d-kserve-f78998949-2hrpx\" (UID: \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\") " pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" Jul 06 19:44:05.223698 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.223700 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dml6d\" (UniqueName: \"kubernetes.io/projected/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-kube-api-access-dml6d\") pod \"sub-test-model-f2694d-kserve-f78998949-2hrpx\" (UID: \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\") " pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" Jul 06 19:44:05.223933 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.223745 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-home\") pod \"sub-test-model-f2694d-kserve-f78998949-2hrpx\" (UID: \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\") " pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" Jul 06 19:44:05.223933 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.223761 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-tls-certs\") pod \"sub-test-model-f2694d-kserve-f78998949-2hrpx\" (UID: \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\") " pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" Jul 06 19:44:05.223933 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.223787 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-model-cache\") pod \"sub-test-model-f2694d-kserve-f78998949-2hrpx\" (UID: \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\") " pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" Jul 06 19:44:05.223933 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.223870 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-tmp-dir\") pod \"sub-test-model-f2694d-kserve-f78998949-2hrpx\" (UID: \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\") " pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" Jul 06 19:44:05.224238 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.224217 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-home\") pod \"sub-test-model-f2694d-kserve-f78998949-2hrpx\" (UID: \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\") " pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" Jul 06 19:44:05.224238 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.224231 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-tmp-dir\") pod \"sub-test-model-f2694d-kserve-f78998949-2hrpx\" (UID: \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\") " pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" Jul 06 19:44:05.224333 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.224310 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-model-cache\") pod \"sub-test-model-f2694d-kserve-f78998949-2hrpx\" (UID: \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\") " pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" Jul 06 19:44:05.225981 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.225961 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-dshm\") pod \"sub-test-model-f2694d-kserve-f78998949-2hrpx\" (UID: \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\") " pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" Jul 06 19:44:05.226264 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.226248 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-tls-certs\") pod \"sub-test-model-f2694d-kserve-f78998949-2hrpx\" (UID: \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\") " pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" Jul 06 19:44:05.238262 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.238238 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dml6d\" (UniqueName: \"kubernetes.io/projected/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-kube-api-access-dml6d\") pod \"sub-test-model-f2694d-kserve-f78998949-2hrpx\" (UID: \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\") " pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" Jul 06 19:44:05.411796 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.411711 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" Jul 06 19:44:05.541631 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:05.541609 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx"] Jul 06 19:44:05.543987 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:44:05.543964 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb80b2de7_e2ea_4e00_aeaa_6e0cdfbb2dbd.slice/crio-abac7942ef38276b32fd27e5cbbaf0140eaef96a8e89633d704bdb43fb2f63d0 WatchSource:0}: Error finding container abac7942ef38276b32fd27e5cbbaf0140eaef96a8e89633d704bdb43fb2f63d0: Status 404 returned error can't find the container with id abac7942ef38276b32fd27e5cbbaf0140eaef96a8e89633d704bdb43fb2f63d0 Jul 06 19:44:06.014665 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:06.014627 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" event={"ID":"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd","Type":"ContainerStarted","Data":"57c39ca1f9223ef86411f3046ce0ae7557dd20637edc6693668bfbceac5e122c"} Jul 06 19:44:06.014843 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:06.014673 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" event={"ID":"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd","Type":"ContainerStarted","Data":"abac7942ef38276b32fd27e5cbbaf0140eaef96a8e89633d704bdb43fb2f63d0"} Jul 06 19:44:06.032912 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:06.032862 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" podStartSLOduration=1.032847728 podStartE2EDuration="1.032847728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:44:06.030632163 +0000 UTC m=+2321.221999835" watchObservedRunningTime="2026-07-06 19:44:06.032847728 +0000 UTC m=+2321.224215401" Jul 06 19:44:09.177854 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:09.177814 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:44:15.412824 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:15.412788 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" Jul 06 19:44:15.413268 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:15.412834 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" Jul 06 19:44:15.425260 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:15.425232 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" Jul 06 19:44:16.067442 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:16.067415 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" Jul 06 19:44:23.095553 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:23.095518 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:44:41.081776 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:41.081739 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:44:47.370647 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:47.370617 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx"] Jul 06 19:44:57.875868 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:44:57.875833 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:45:02.391538 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:02.391477 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" podUID="b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd" containerName="main" containerID="cri-o://57c39ca1f9223ef86411f3046ce0ae7557dd20637edc6693668bfbceac5e122c" gracePeriod=45 Jul 06 19:45:02.637013 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:02.636987 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" Jul 06 19:45:02.707343 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:02.707321 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-tls-certs\") pod \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\" (UID: \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\") " Jul 06 19:45:02.707522 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:02.707362 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-dshm\") pod \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\" (UID: \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\") " Jul 06 19:45:02.707522 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:02.707393 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-model-cache\") pod \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\" (UID: \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\") " Jul 06 19:45:02.707522 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:02.707427 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-home\") pod \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\" (UID: \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\") " Jul 06 19:45:02.707522 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:02.707467 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-tmp-dir\") pod \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\" (UID: \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\") " Jul 06 19:45:02.707522 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:02.707488 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dml6d\" (UniqueName: \"kubernetes.io/projected/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-kube-api-access-dml6d\") pod \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\" (UID: \"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd\") " Jul 06 19:45:02.707902 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:02.707739 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-model-cache" (OuterVolumeSpecName: "model-cache") pod "b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd" (UID: "b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:45:02.707902 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:02.707756 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-home" (OuterVolumeSpecName: "home") pod "b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd" (UID: "b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:45:02.707902 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:02.707765 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd" (UID: "b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:45:02.709556 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:02.709532 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-dshm" (OuterVolumeSpecName: "dshm") pod "b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd" (UID: "b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:45:02.709679 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:02.709614 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-kube-api-access-dml6d" (OuterVolumeSpecName: "kube-api-access-dml6d") pod "b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd" (UID: "b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd"). InnerVolumeSpecName "kube-api-access-dml6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:45:02.709726 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:02.709699 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd" (UID: "b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:45:02.808326 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:02.808294 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-home\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:45:02.808326 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:02.808319 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-tmp-dir\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:45:02.808326 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:02.808328 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dml6d\" (UniqueName: \"kubernetes.io/projected/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-kube-api-access-dml6d\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:45:02.808543 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:02.808340 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:45:02.808543 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:02.808348 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-dshm\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:45:02.808543 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:02.808356 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd-model-cache\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:45:03.256009 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:03.255971 2560 generic.go:356] "Generic (PLEG): container finished" podID="b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd" containerID="57c39ca1f9223ef86411f3046ce0ae7557dd20637edc6693668bfbceac5e122c" exitCode=0 Jul 06 19:45:03.256176 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:03.256014 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" Jul 06 19:45:03.256176 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:03.256029 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" event={"ID":"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd","Type":"ContainerDied","Data":"57c39ca1f9223ef86411f3046ce0ae7557dd20637edc6693668bfbceac5e122c"} Jul 06 19:45:03.256176 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:03.256068 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx" event={"ID":"b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd","Type":"ContainerDied","Data":"abac7942ef38276b32fd27e5cbbaf0140eaef96a8e89633d704bdb43fb2f63d0"} Jul 06 19:45:03.256176 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:03.256089 2560 scope.go:117] "RemoveContainer" containerID="57c39ca1f9223ef86411f3046ce0ae7557dd20637edc6693668bfbceac5e122c" Jul 06 19:45:03.264340 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:03.264320 2560 scope.go:117] "RemoveContainer" containerID="57c39ca1f9223ef86411f3046ce0ae7557dd20637edc6693668bfbceac5e122c" Jul 06 19:45:03.264609 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:45:03.264563 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57c39ca1f9223ef86411f3046ce0ae7557dd20637edc6693668bfbceac5e122c\": container with ID starting with 57c39ca1f9223ef86411f3046ce0ae7557dd20637edc6693668bfbceac5e122c not found: ID does not exist" containerID="57c39ca1f9223ef86411f3046ce0ae7557dd20637edc6693668bfbceac5e122c" Jul 06 19:45:03.264675 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:03.264617 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57c39ca1f9223ef86411f3046ce0ae7557dd20637edc6693668bfbceac5e122c"} err="failed to get container status \"57c39ca1f9223ef86411f3046ce0ae7557dd20637edc6693668bfbceac5e122c\": rpc error: code = NotFound desc = could not find container \"57c39ca1f9223ef86411f3046ce0ae7557dd20637edc6693668bfbceac5e122c\": container with ID starting with 57c39ca1f9223ef86411f3046ce0ae7557dd20637edc6693668bfbceac5e122c not found: ID does not exist" Jul 06 19:45:03.276767 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:03.276735 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx"] Jul 06 19:45:03.280862 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:03.280841 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f2694d/sub-test-model-f2694d-kserve-f78998949-2hrpx"] Jul 06 19:45:03.390053 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:03.390024 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd" path="/var/lib/kubelet/pods/b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd/volumes" Jul 06 19:45:07.183390 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.183356 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2"] Jul 06 19:45:07.183766 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.183679 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd" containerName="main" Jul 06 19:45:07.183766 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.183691 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd" containerName="main" Jul 06 19:45:07.183766 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.183753 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b80b2de7-e2ea-4e00-aeaa-6e0cdfbb2dbd" containerName="main" Jul 06 19:45:07.187835 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.187815 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" Jul 06 19:45:07.190839 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.190819 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-bd1298\"/\"kube-root-ca.crt\"" Jul 06 19:45:07.192061 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.192039 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bd1298\"/\"rate-test-model-bd1298-kserve-self-signed-certs\"" Jul 06 19:45:07.192121 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.192074 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bd1298\"/\"default-dockercfg-7glcm\"" Jul 06 19:45:07.192168 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.192079 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-bd1298\"/\"openshift-service-ca.crt\"" Jul 06 19:45:07.198181 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.198156 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2"] Jul 06 19:45:07.242886 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.242855 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-model-cache\") pod \"rate-test-model-bd1298-kserve-8544c6cd79-tsgp2\" (UID: \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\") " pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" Jul 06 19:45:07.243079 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.242903 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ssvx\" (UniqueName: \"kubernetes.io/projected/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-kube-api-access-9ssvx\") pod \"rate-test-model-bd1298-kserve-8544c6cd79-tsgp2\" (UID: \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\") " pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" Jul 06 19:45:07.243079 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.242967 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-tmp-dir\") pod \"rate-test-model-bd1298-kserve-8544c6cd79-tsgp2\" (UID: \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\") " pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" Jul 06 19:45:07.243079 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.243018 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-tls-certs\") pod \"rate-test-model-bd1298-kserve-8544c6cd79-tsgp2\" (UID: \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\") " pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" Jul 06 19:45:07.243079 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.243065 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-home\") pod \"rate-test-model-bd1298-kserve-8544c6cd79-tsgp2\" (UID: \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\") " pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" Jul 06 19:45:07.243270 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.243096 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-dshm\") pod \"rate-test-model-bd1298-kserve-8544c6cd79-tsgp2\" (UID: \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\") " pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" Jul 06 19:45:07.344071 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.344025 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-tmp-dir\") pod \"rate-test-model-bd1298-kserve-8544c6cd79-tsgp2\" (UID: \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\") " pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" Jul 06 19:45:07.344275 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.344093 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-tls-certs\") pod \"rate-test-model-bd1298-kserve-8544c6cd79-tsgp2\" (UID: \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\") " pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" Jul 06 19:45:07.344275 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.344132 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-home\") pod \"rate-test-model-bd1298-kserve-8544c6cd79-tsgp2\" (UID: \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\") " pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" Jul 06 19:45:07.344275 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.344162 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-dshm\") pod \"rate-test-model-bd1298-kserve-8544c6cd79-tsgp2\" (UID: \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\") " pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" Jul 06 19:45:07.344275 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.344206 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-model-cache\") pod \"rate-test-model-bd1298-kserve-8544c6cd79-tsgp2\" (UID: \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\") " pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" Jul 06 19:45:07.344275 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.344242 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ssvx\" (UniqueName: \"kubernetes.io/projected/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-kube-api-access-9ssvx\") pod \"rate-test-model-bd1298-kserve-8544c6cd79-tsgp2\" (UID: \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\") " pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" Jul 06 19:45:07.344523 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.344427 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-tmp-dir\") pod \"rate-test-model-bd1298-kserve-8544c6cd79-tsgp2\" (UID: \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\") " pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" Jul 06 19:45:07.344523 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.344485 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-home\") pod \"rate-test-model-bd1298-kserve-8544c6cd79-tsgp2\" (UID: \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\") " pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" Jul 06 19:45:07.344656 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.344557 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-model-cache\") pod \"rate-test-model-bd1298-kserve-8544c6cd79-tsgp2\" (UID: \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\") " pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" Jul 06 19:45:07.346499 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.346467 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-dshm\") pod \"rate-test-model-bd1298-kserve-8544c6cd79-tsgp2\" (UID: \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\") " pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" Jul 06 19:45:07.346702 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.346685 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-tls-certs\") pod \"rate-test-model-bd1298-kserve-8544c6cd79-tsgp2\" (UID: \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\") " pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" Jul 06 19:45:07.352037 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.352014 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ssvx\" (UniqueName: \"kubernetes.io/projected/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-kube-api-access-9ssvx\") pod \"rate-test-model-bd1298-kserve-8544c6cd79-tsgp2\" (UID: \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\") " pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" Jul 06 19:45:07.496959 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.496923 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" Jul 06 19:45:07.625231 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:07.625210 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2"] Jul 06 19:45:07.626957 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:45:07.626925 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e1ca23d_2aaf_4c6a_917f_fbc0848d4d82.slice/crio-7b24fecdc95914a78f8adba25c2e1edb41de940117c3d20e9e4de74adc9ef892 WatchSource:0}: Error finding container 7b24fecdc95914a78f8adba25c2e1edb41de940117c3d20e9e4de74adc9ef892: Status 404 returned error can't find the container with id 7b24fecdc95914a78f8adba25c2e1edb41de940117c3d20e9e4de74adc9ef892 Jul 06 19:45:08.280725 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:08.280691 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" event={"ID":"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82","Type":"ContainerStarted","Data":"e0099705a8348f9ffd7b5b745ed8ed10fcbd00e934371cdece3b4f4dbb5e28a0"} Jul 06 19:45:08.280725 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:08.280729 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" event={"ID":"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82","Type":"ContainerStarted","Data":"7b24fecdc95914a78f8adba25c2e1edb41de940117c3d20e9e4de74adc9ef892"} Jul 06 19:45:08.297709 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:08.297660 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" podStartSLOduration=1.297642116 podStartE2EDuration="1.297642116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 19:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:45:08.297020926 +0000 UTC m=+2383.488388604" watchObservedRunningTime="2026-07-06 19:45:08.297642116 +0000 UTC m=+2383.489009789" Jul 06 19:45:11.070101 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:11.070068 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:45:17.498020 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:17.497985 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" Jul 06 19:45:17.498636 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:17.498035 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" Jul 06 19:45:17.510196 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:17.510173 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" Jul 06 19:45:18.331042 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:18.331011 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" Jul 06 19:45:50.471412 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:45:50.471335 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:46:08.278044 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:08.278009 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:46:22.191462 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:22.191421 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:46:30.008091 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:30.008056 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2"] Jul 06 19:46:34.470709 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:34.470676 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:46:45.027473 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.027413 2560 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" podUID="0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82" containerName="main" containerID="cri-o://e0099705a8348f9ffd7b5b745ed8ed10fcbd00e934371cdece3b4f4dbb5e28a0" gracePeriod=45 Jul 06 19:46:45.287300 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.287241 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" Jul 06 19:46:45.456242 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.456207 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ssvx\" (UniqueName: \"kubernetes.io/projected/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-kube-api-access-9ssvx\") pod \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\" (UID: \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\") " Jul 06 19:46:45.456419 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.456275 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-tmp-dir\") pod \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\" (UID: \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\") " Jul 06 19:46:45.456419 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.456297 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-tls-certs\") pod \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\" (UID: \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\") " Jul 06 19:46:45.456419 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.456382 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-model-cache\") pod \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\" (UID: \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\") " Jul 06 19:46:45.456419 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.456405 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-home\") pod \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\" (UID: \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\") " Jul 06 19:46:45.456666 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.456444 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-dshm\") pod \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\" (UID: \"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82\") " Jul 06 19:46:45.456666 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.456604 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82" (UID: "0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:46:45.456772 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.456677 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-home" (OuterVolumeSpecName: "home") pod "0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82" (UID: "0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:46:45.456772 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.456698 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82" (UID: "0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:46:45.456772 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.456729 2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-tmp-dir\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:46:45.458469 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.458441 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-dshm" (OuterVolumeSpecName: "dshm") pod "0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82" (UID: "0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 19:46:45.458602 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.458445 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-kube-api-access-9ssvx" (OuterVolumeSpecName: "kube-api-access-9ssvx") pod "0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82" (UID: "0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82"). InnerVolumeSpecName "kube-api-access-9ssvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 19:46:45.458602 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.458519 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82" (UID: "0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 19:46:45.557304 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.557220 2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-dshm\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:46:45.557304 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.557249 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ssvx\" (UniqueName: \"kubernetes.io/projected/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-kube-api-access-9ssvx\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:46:45.557304 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.557260 2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-tls-certs\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:46:45.557304 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.557270 2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-model-cache\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:46:45.557304 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.557277 2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82-home\") on node \"ip-10-0-129-80.ec2.internal\" DevicePath \"\"" Jul 06 19:46:45.688664 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.688633 2560 generic.go:356] "Generic (PLEG): container finished" podID="0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82" containerID="e0099705a8348f9ffd7b5b745ed8ed10fcbd00e934371cdece3b4f4dbb5e28a0" exitCode=0 Jul 06 19:46:45.688827 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.688680 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" Jul 06 19:46:45.688827 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.688681 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" event={"ID":"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82","Type":"ContainerDied","Data":"e0099705a8348f9ffd7b5b745ed8ed10fcbd00e934371cdece3b4f4dbb5e28a0"} Jul 06 19:46:45.688827 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.688799 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2" event={"ID":"0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82","Type":"ContainerDied","Data":"7b24fecdc95914a78f8adba25c2e1edb41de940117c3d20e9e4de74adc9ef892"} Jul 06 19:46:45.688827 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.688820 2560 scope.go:117] "RemoveContainer" containerID="e0099705a8348f9ffd7b5b745ed8ed10fcbd00e934371cdece3b4f4dbb5e28a0" Jul 06 19:46:45.698972 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.698950 2560 scope.go:117] "RemoveContainer" containerID="e0099705a8348f9ffd7b5b745ed8ed10fcbd00e934371cdece3b4f4dbb5e28a0" Jul 06 19:46:45.700016 ip-10-0-129-80 kubenswrapper[2560]: E0706 19:46:45.699990 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0099705a8348f9ffd7b5b745ed8ed10fcbd00e934371cdece3b4f4dbb5e28a0\": container with ID starting with e0099705a8348f9ffd7b5b745ed8ed10fcbd00e934371cdece3b4f4dbb5e28a0 not found: ID does not exist" containerID="e0099705a8348f9ffd7b5b745ed8ed10fcbd00e934371cdece3b4f4dbb5e28a0" Jul 06 19:46:45.700113 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.700033 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0099705a8348f9ffd7b5b745ed8ed10fcbd00e934371cdece3b4f4dbb5e28a0"} err="failed to get container status \"e0099705a8348f9ffd7b5b745ed8ed10fcbd00e934371cdece3b4f4dbb5e28a0\": rpc error: code = NotFound desc = could not find container \"e0099705a8348f9ffd7b5b745ed8ed10fcbd00e934371cdece3b4f4dbb5e28a0\": container with ID starting with e0099705a8348f9ffd7b5b745ed8ed10fcbd00e934371cdece3b4f4dbb5e28a0 not found: ID does not exist" Jul 06 19:46:45.718292 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.718272 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2"] Jul 06 19:46:45.722711 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:45.722689 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-bd1298/rate-test-model-bd1298-kserve-8544c6cd79-tsgp2"] Jul 06 19:46:46.170622 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:46.170589 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:46:47.390873 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:47.390841 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82" path="/var/lib/kubelet/pods/0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82/volumes" Jul 06 19:46:56.215253 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:56.215221 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b65467f89-px8rm_305566d2-d32b-4527-b87e-4e96c8fa0c3b/authorino/0.log" Jul 06 19:46:57.273217 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:46:57.273186 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-2mmvv"] Jul 06 19:47:02.136317 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:02.136289 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn_fca2463e-c1ba-42e0-85ab-5b321c61940c/copy-manifests/0.log" Jul 06 19:47:02.148060 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:02.148031 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn_fca2463e-c1ba-42e0-85ab-5b321c61940c/manager/0.log" Jul 06 19:47:03.713556 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:03.713523 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b65467f89-px8rm_305566d2-d32b-4527-b87e-4e96c8fa0c3b/authorino/0.log" Jul 06 19:47:04.075421 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:04.075341 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-b429r_4b3029aa-1e23-4bb9-9f21-cbb38517cb0c/kuadrant-console-plugin/0.log" Jul 06 19:47:04.448340 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:04.448307 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-2mmvv_3445ab7e-f3f4-4c28-8b1d-5c3771c211d1/limitador/0.log" Jul 06 19:47:05.179177 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:05.179150 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d5b9884b5-78zsz_0cf4ca25-f9f2-412e-bf20-9e918f48a804/kube-auth-proxy/0.log" Jul 06 19:47:06.632475 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:06.632439 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-6l6gs_b361b590-f458-4a0f-ac61-178547d6892c/main/0.log" Jul 06 19:47:06.749519 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:06.749489 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-5wt67_c99200dd-114d-4f40-8946-608b63e10695/main/0.log" Jul 06 19:47:14.186872 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:14.186844 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dl7lj_de165982-d4dc-4449-a40d-1fc3bf2754cc/global-pull-secret-syncer/0.log" Jul 06 19:47:14.330645 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:14.330610 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-z9v9z_58a69bd8-7312-4728-9ac1-efc5384484bd/konnectivity-agent/0.log" Jul 06 19:47:14.348727 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:14.348702 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-80.ec2.internal_831a3fedbcc35899a7ac03738589a136/haproxy/0.log" Jul 06 19:47:20.034478 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:20.034400 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b65467f89-px8rm_305566d2-d32b-4527-b87e-4e96c8fa0c3b/authorino/0.log" Jul 06 19:47:20.159972 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:20.159944 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-b429r_4b3029aa-1e23-4bb9-9f21-cbb38517cb0c/kuadrant-console-plugin/0.log" Jul 06 19:47:20.433464 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:20.433386 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-2mmvv_3445ab7e-f3f4-4c28-8b1d-5c3771c211d1/limitador/0.log" Jul 06 19:47:22.588194 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:22.588163 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d2tsw_a3897544-f579-477e-b95c-a9a0415a9cea/node-exporter/0.log" Jul 06 19:47:22.605548 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:22.605522 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d2tsw_a3897544-f579-477e-b95c-a9a0415a9cea/kube-rbac-proxy/0.log" Jul 06 19:47:22.622591 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:22.622546 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d2tsw_a3897544-f579-477e-b95c-a9a0415a9cea/init-textfile/0.log" Jul 06 19:47:23.886296 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:23.886260 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-976df/perf-node-gather-daemonset-ld7jt"] Jul 06 19:47:23.886690 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:23.886549 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82" containerName="main" Jul 06 19:47:23.886690 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:23.886559 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82" containerName="main" Jul 06 19:47:23.886690 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:23.886626 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e1ca23d-2aaf-4c6a-917f-fbc0848d4d82" containerName="main" Jul 06 19:47:23.889655 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:23.889635 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-976df/perf-node-gather-daemonset-ld7jt" Jul 06 19:47:23.892500 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:23.892480 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-976df\"/\"kube-root-ca.crt\"" Jul 06 19:47:23.893823 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:23.893803 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-976df\"/\"default-dockercfg-dhsqh\"" Jul 06 19:47:23.893937 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:23.893919 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-976df\"/\"openshift-service-ca.crt\"" Jul 06 19:47:23.898841 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:23.898819 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-976df/perf-node-gather-daemonset-ld7jt"] Jul 06 19:47:23.947986 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:23.947955 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bf6b684-ea63-4598-b53d-06be9cda1a2a-sys\") pod \"perf-node-gather-daemonset-ld7jt\" (UID: \"4bf6b684-ea63-4598-b53d-06be9cda1a2a\") " pod="openshift-must-gather-976df/perf-node-gather-daemonset-ld7jt" Jul 06 19:47:23.948150 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:23.947996 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bf6b684-ea63-4598-b53d-06be9cda1a2a-lib-modules\") pod \"perf-node-gather-daemonset-ld7jt\" (UID: \"4bf6b684-ea63-4598-b53d-06be9cda1a2a\") " pod="openshift-must-gather-976df/perf-node-gather-daemonset-ld7jt" Jul 06 19:47:23.948150 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:23.948018 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4bf6b684-ea63-4598-b53d-06be9cda1a2a-podres\") pod \"perf-node-gather-daemonset-ld7jt\" (UID: \"4bf6b684-ea63-4598-b53d-06be9cda1a2a\") " pod="openshift-must-gather-976df/perf-node-gather-daemonset-ld7jt" Jul 06 19:47:23.948150 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:23.948094 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4bf6b684-ea63-4598-b53d-06be9cda1a2a-proc\") pod \"perf-node-gather-daemonset-ld7jt\" (UID: \"4bf6b684-ea63-4598-b53d-06be9cda1a2a\") " pod="openshift-must-gather-976df/perf-node-gather-daemonset-ld7jt" Jul 06 19:47:23.948254 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:23.948151 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r957\" (UniqueName: \"kubernetes.io/projected/4bf6b684-ea63-4598-b53d-06be9cda1a2a-kube-api-access-5r957\") pod \"perf-node-gather-daemonset-ld7jt\" (UID: \"4bf6b684-ea63-4598-b53d-06be9cda1a2a\") " pod="openshift-must-gather-976df/perf-node-gather-daemonset-ld7jt" Jul 06 19:47:24.049120 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:24.049083 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bf6b684-ea63-4598-b53d-06be9cda1a2a-sys\") pod \"perf-node-gather-daemonset-ld7jt\" (UID: \"4bf6b684-ea63-4598-b53d-06be9cda1a2a\") " pod="openshift-must-gather-976df/perf-node-gather-daemonset-ld7jt" Jul 06 19:47:24.049292 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:24.049127 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bf6b684-ea63-4598-b53d-06be9cda1a2a-lib-modules\") pod \"perf-node-gather-daemonset-ld7jt\" (UID: \"4bf6b684-ea63-4598-b53d-06be9cda1a2a\") " pod="openshift-must-gather-976df/perf-node-gather-daemonset-ld7jt" Jul 06 19:47:24.049292 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:24.049160 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4bf6b684-ea63-4598-b53d-06be9cda1a2a-podres\") pod \"perf-node-gather-daemonset-ld7jt\" (UID: \"4bf6b684-ea63-4598-b53d-06be9cda1a2a\") " pod="openshift-must-gather-976df/perf-node-gather-daemonset-ld7jt" Jul 06 19:47:24.049292 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:24.049192 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4bf6b684-ea63-4598-b53d-06be9cda1a2a-proc\") pod \"perf-node-gather-daemonset-ld7jt\" (UID: \"4bf6b684-ea63-4598-b53d-06be9cda1a2a\") " pod="openshift-must-gather-976df/perf-node-gather-daemonset-ld7jt" Jul 06 19:47:24.049292 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:24.049212 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4bf6b684-ea63-4598-b53d-06be9cda1a2a-sys\") pod \"perf-node-gather-daemonset-ld7jt\" (UID: \"4bf6b684-ea63-4598-b53d-06be9cda1a2a\") " pod="openshift-must-gather-976df/perf-node-gather-daemonset-ld7jt" Jul 06 19:47:24.049292 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:24.049251 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r957\" (UniqueName: \"kubernetes.io/projected/4bf6b684-ea63-4598-b53d-06be9cda1a2a-kube-api-access-5r957\") pod \"perf-node-gather-daemonset-ld7jt\" (UID: \"4bf6b684-ea63-4598-b53d-06be9cda1a2a\") " pod="openshift-must-gather-976df/perf-node-gather-daemonset-ld7jt" Jul 06 19:47:24.049292 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:24.049284 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4bf6b684-ea63-4598-b53d-06be9cda1a2a-proc\") pod \"perf-node-gather-daemonset-ld7jt\" (UID: \"4bf6b684-ea63-4598-b53d-06be9cda1a2a\") " pod="openshift-must-gather-976df/perf-node-gather-daemonset-ld7jt" Jul 06 19:47:24.049484 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:24.049291 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4bf6b684-ea63-4598-b53d-06be9cda1a2a-lib-modules\") pod \"perf-node-gather-daemonset-ld7jt\" (UID: \"4bf6b684-ea63-4598-b53d-06be9cda1a2a\") " pod="openshift-must-gather-976df/perf-node-gather-daemonset-ld7jt" Jul 06 19:47:24.049484 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:24.049311 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4bf6b684-ea63-4598-b53d-06be9cda1a2a-podres\") pod \"perf-node-gather-daemonset-ld7jt\" (UID: \"4bf6b684-ea63-4598-b53d-06be9cda1a2a\") " pod="openshift-must-gather-976df/perf-node-gather-daemonset-ld7jt" Jul 06 19:47:24.057843 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:24.057814 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r957\" (UniqueName: \"kubernetes.io/projected/4bf6b684-ea63-4598-b53d-06be9cda1a2a-kube-api-access-5r957\") pod \"perf-node-gather-daemonset-ld7jt\" (UID: \"4bf6b684-ea63-4598-b53d-06be9cda1a2a\") " pod="openshift-must-gather-976df/perf-node-gather-daemonset-ld7jt" Jul 06 19:47:24.201212 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:24.201186 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-976df/perf-node-gather-daemonset-ld7jt" Jul 06 19:47:24.330162 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:24.330134 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-976df/perf-node-gather-daemonset-ld7jt"] Jul 06 19:47:24.332135 ip-10-0-129-80 kubenswrapper[2560]: W0706 19:47:24.332108 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4bf6b684_ea63_4598_b53d_06be9cda1a2a.slice/crio-feef1a0001f4c0de41bfddbb51530779dddd9c8016c4658c9a712f2d3840f2f0 WatchSource:0}: Error finding container feef1a0001f4c0de41bfddbb51530779dddd9c8016c4658c9a712f2d3840f2f0: Status 404 returned error can't find the container with id feef1a0001f4c0de41bfddbb51530779dddd9c8016c4658c9a712f2d3840f2f0 Jul 06 19:47:24.860790 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:24.860749 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-976df/perf-node-gather-daemonset-ld7jt" event={"ID":"4bf6b684-ea63-4598-b53d-06be9cda1a2a","Type":"ContainerStarted","Data":"60b13d4c6cd327c161b31fb733c2190a4781245f9002c40cabcc317e98ae1a4f"} Jul 06 19:47:24.860961 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:24.860842 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-976df/perf-node-gather-daemonset-ld7jt" event={"ID":"4bf6b684-ea63-4598-b53d-06be9cda1a2a","Type":"ContainerStarted","Data":"feef1a0001f4c0de41bfddbb51530779dddd9c8016c4658c9a712f2d3840f2f0"} Jul 06 19:47:24.860961 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:24.860870 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-976df/perf-node-gather-daemonset-ld7jt" Jul 06 19:47:24.879102 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:24.879053 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-976df/perf-node-gather-daemonset-ld7jt" podStartSLOduration=1.87903975 podStartE2EDuration="1.87903975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 19:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 19:47:24.878289869 +0000 UTC m=+2520.069657541" watchObservedRunningTime="2026-07-06 19:47:24.87903975 +0000 UTC m=+2520.070407401" Jul 06 19:47:25.306106 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:25.306067 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-68f967bf9b-5cn9r_fc451369-ba32-4e0f-a332-b6d2c0702f4e/console/0.log" Jul 06 19:47:25.336383 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:25.336343 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-m8s99_4f2e60c7-e7d9-4d33-88aa-1d9572b42567/download-server/0.log" Jul 06 19:47:26.831683 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:26.831653 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-72dgz_5c23cd74-f515-4362-8f23-724a60a71d92/dns/0.log" Jul 06 19:47:26.848702 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:26.848661 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-72dgz_5c23cd74-f515-4362-8f23-724a60a71d92/kube-rbac-proxy/0.log" Jul 06 19:47:26.996336 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:26.996305 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m5pz7_84c43c19-1748-4d6b-8c86-b9171445c9b8/dns-node-resolver/0.log" Jul 06 19:47:27.556569 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:27.556525 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gghgg_fe072be2-5f94-440e-b0e4-83e1e4899fc8/node-ca/0.log" Jul 06 19:47:28.716658 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:28.716630 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d5b9884b5-78zsz_0cf4ca25-f9f2-412e-bf20-9e918f48a804/kube-auth-proxy/0.log" Jul 06 19:47:29.591870 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:29.591839 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kr28m_a9fe8873-82b2-4f13-82e3-d78b5f2d77b5/serve-healthcheck-canary/0.log" Jul 06 19:47:30.081002 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:30.080974 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-24gdx_d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6/kube-rbac-proxy/0.log" Jul 06 19:47:30.098126 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:30.098102 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-24gdx_d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6/exporter/0.log" Jul 06 19:47:30.119485 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:30.119462 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-24gdx_d88ae87e-56ea-4b82-b7cf-6d7bbb0612a6/extractor/0.log" Jul 06 19:47:30.877132 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:30.877099 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-976df/perf-node-gather-daemonset-ld7jt" Jul 06 19:47:33.036112 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:33.036079 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn_fca2463e-c1ba-42e0-85ab-5b321c61940c/manager/0.log" Jul 06 19:47:33.053112 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:33.053088 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d4cdfbd4d-5lzvn_fca2463e-c1ba-42e0-85ab-5b321c61940c/copy-manifests/0.log" Jul 06 19:47:34.620471 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:34.620438 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7995d775b8-ddtnd_cfbd124b-5f35-45c8-abc6-4c321941360c/manager/0.log" Jul 06 19:47:34.875416 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:34.875335 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d5b9884b5-78zsz_0cf4ca25-f9f2-412e-bf20-9e918f48a804/kube-auth-proxy/0.log" Jul 06 19:47:36.392771 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:36.392742 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d5b9884b5-78zsz_0cf4ca25-f9f2-412e-bf20-9e918f48a804/kube-auth-proxy/0.log" Jul 06 19:47:40.985591 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:40.985550 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vj7fc_b6e4a6e5-96c2-4496-af5a-e162d61d9c20/kube-multus-additional-cni-plugins/0.log" Jul 06 19:47:41.009870 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:41.009847 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vj7fc_b6e4a6e5-96c2-4496-af5a-e162d61d9c20/egress-router-binary-copy/0.log" Jul 06 19:47:41.029926 ip-10-0-129-80 kubenswrapper[2560]: I0706 19:47:41.029902 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vj7fc_b6e4a6e5-96c2-4496-af5a-e162d61d9c20/cni-plugins/0.log"