Jul 22 12:40:51.492196 ip-10-0-129-34 systemd[1]: Starting Kubernetes Kubelet... Jul 22 12:40:51.863642 ip-10-0-129-34 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 12:40:51.863642 ip-10-0-129-34 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 22 12:40:51.863642 ip-10-0-129-34 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 12:40:51.863642 ip-10-0-129-34 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 22 12:40:51.863642 ip-10-0-129-34 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 22 12:40:51.863642 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.813421 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 22 12:40:51.863642 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820529 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 12:40:51.863642 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820545 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 12:40:51.863642 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820549 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 12:40:51.863642 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820552 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 12:40:51.863642 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820556 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 12:40:51.863642 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820558 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 12:40:51.863642 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820561 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 12:40:51.863642 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820564 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 12:40:51.865694 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820566 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 12:40:51.865694 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820572 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 12:40:51.865694 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820576 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 12:40:51.865694 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820579 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 12:40:51.865694 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820582 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 12:40:51.865694 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820584 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 12:40:51.865694 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820587 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 12:40:51.865694 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820590 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 12:40:51.865694 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820593 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 12:40:51.865694 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820595 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 12:40:51.865694 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820598 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 12:40:51.865694 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820600 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 12:40:51.865694 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820603 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 12:40:51.865694 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820606 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 12:40:51.865694 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820608 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 12:40:51.865694 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820611 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 12:40:51.865694 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820613 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 12:40:51.865694 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820616 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 12:40:51.865694 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820619 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 12:40:51.867946 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820622 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 12:40:51.867946 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820624 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 12:40:51.867946 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820627 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 12:40:51.867946 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820630 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 12:40:51.867946 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820633 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 12:40:51.867946 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820635 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 12:40:51.867946 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820638 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 12:40:51.867946 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820641 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 12:40:51.867946 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820643 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 12:40:51.867946 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820646 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 12:40:51.867946 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820648 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 12:40:51.867946 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820651 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 12:40:51.867946 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820653 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 12:40:51.867946 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820656 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 12:40:51.867946 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820658 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 12:40:51.867946 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820661 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 12:40:51.867946 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820663 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 12:40:51.867946 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820666 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 12:40:51.867946 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820668 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 12:40:51.867946 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820671 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 12:40:51.884374 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820673 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 12:40:51.884374 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820676 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 12:40:51.884374 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820679 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 12:40:51.884374 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820682 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 12:40:51.884374 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820686 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 12:40:51.884374 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820688 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 12:40:51.884374 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820691 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 12:40:51.884374 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820694 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 12:40:51.884374 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820696 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 12:40:51.884374 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820699 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 12:40:51.884374 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820701 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 12:40:51.884374 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820704 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 12:40:51.884374 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820707 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 12:40:51.884374 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820709 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 12:40:51.884374 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820712 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 12:40:51.884374 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820715 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 12:40:51.884374 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820717 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 12:40:51.884374 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820721 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 12:40:51.884374 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820725 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 12:40:51.882800 ip-10-0-129-34 systemd[1]: Started Kubernetes Kubelet. Jul 22 12:40:51.887621 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820727 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 12:40:51.887621 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820730 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 12:40:51.887621 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820732 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 12:40:51.887621 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820735 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 12:40:51.887621 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820739 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 12:40:51.887621 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820742 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 12:40:51.887621 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820744 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 12:40:51.887621 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820747 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 22 12:40:51.887621 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820750 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 12:40:51.887621 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820753 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 12:40:51.887621 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820755 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 12:40:51.887621 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820758 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 12:40:51.887621 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820761 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 12:40:51.887621 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820765 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 12:40:51.887621 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820767 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 12:40:51.887621 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820770 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 12:40:51.887621 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820772 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 12:40:51.887621 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820775 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 12:40:51.887621 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820778 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 12:40:51.887621 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820781 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 12:40:51.899512 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820785 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 12:40:51.899512 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820788 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 12:40:51.899512 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820791 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 12:40:51.899512 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820795 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 12:40:51.899512 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820798 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 12:40:51.899512 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820801 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 12:40:51.899512 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820804 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 12:40:51.899512 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820806 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 12:40:51.899512 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820809 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 12:40:51.899512 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820812 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 12:40:51.899512 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820815 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 12:40:51.899512 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820817 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 12:40:51.899512 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.820820 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 12:40:51.899512 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821548 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 12:40:51.899512 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821555 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 12:40:51.899512 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821559 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 12:40:51.899512 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821563 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 12:40:51.899512 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821566 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 12:40:51.899512 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821569 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 12:40:51.899512 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821571 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 12:40:51.904586 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821574 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 12:40:51.904586 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821577 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 12:40:51.904586 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821579 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 12:40:51.904586 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821581 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 12:40:51.904586 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821584 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 12:40:51.904586 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821591 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 12:40:51.904586 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821594 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 12:40:51.904586 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821598 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 12:40:51.904586 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821601 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 12:40:51.904586 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821604 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 12:40:51.904586 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821607 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 12:40:51.904586 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821610 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 12:40:51.904586 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821612 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 12:40:51.904586 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821615 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 12:40:51.904586 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821618 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 12:40:51.904586 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821620 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 12:40:51.904586 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821623 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 12:40:51.904586 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821626 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 12:40:51.904586 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821628 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 12:40:51.908472 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821631 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 12:40:51.908472 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821634 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 12:40:51.908472 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821636 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 12:40:51.908472 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821639 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 12:40:51.908472 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821641 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 12:40:51.908472 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821644 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 12:40:51.908472 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821646 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 22 12:40:51.908472 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821649 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 12:40:51.908472 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821652 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 12:40:51.908472 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821654 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 12:40:51.908472 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821657 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 12:40:51.908472 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821659 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 12:40:51.908472 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821665 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 12:40:51.908472 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821667 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 12:40:51.908472 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821670 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 12:40:51.908472 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821672 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 12:40:51.908472 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821676 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 12:40:51.908472 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821679 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 12:40:51.908472 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821687 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 12:40:51.914129 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821689 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 12:40:51.914129 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821692 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 12:40:51.914129 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821695 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 12:40:51.914129 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821697 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 12:40:51.914129 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821701 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 12:40:51.914129 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821704 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 12:40:51.914129 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821706 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 12:40:51.914129 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821709 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 12:40:51.914129 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821712 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 12:40:51.914129 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821714 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 12:40:51.914129 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821717 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 12:40:51.914129 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821719 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 12:40:51.914129 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821722 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 12:40:51.914129 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821724 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 12:40:51.914129 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821727 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 12:40:51.914129 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821729 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 12:40:51.914129 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821732 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 12:40:51.914129 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821735 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 12:40:51.914129 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821737 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 12:40:51.914129 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821740 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 12:40:51.917603 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821742 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 12:40:51.917603 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821744 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 12:40:51.917603 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821747 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 12:40:51.917603 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821751 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 12:40:51.917603 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821754 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 12:40:51.917603 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821757 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 12:40:51.917603 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821760 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 12:40:51.917603 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821763 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 12:40:51.917603 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821765 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 12:40:51.917603 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821768 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 12:40:51.917603 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821771 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 12:40:51.917603 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821776 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 12:40:51.917603 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821778 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 12:40:51.917603 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821781 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 12:40:51.917603 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821783 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 12:40:51.917603 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821787 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 12:40:51.917603 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821789 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 12:40:51.917603 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821792 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 12:40:51.917603 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821795 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 12:40:51.917603 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821798 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 12:40:51.923568 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821800 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 12:40:51.923568 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821803 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 12:40:51.923568 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821806 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 12:40:51.923568 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821808 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 12:40:51.923568 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821811 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 12:40:51.923568 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821813 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 12:40:51.923568 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821816 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 12:40:51.923568 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821818 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 12:40:51.923568 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821821 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 12:40:51.923568 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821823 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 12:40:51.923568 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821825 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 12:40:51.923568 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821828 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 12:40:51.923568 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821830 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 12:40:51.923568 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.821833 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 12:40:51.923568 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822219 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 12:40:51.923568 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822224 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 12:40:51.923568 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822227 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 12:40:51.923568 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822230 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 12:40:51.923568 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822233 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 12:40:51.923568 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822235 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 12:40:52.073167 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822238 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 12:40:52.073167 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822241 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 12:40:52.073167 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822243 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 12:40:52.073167 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822249 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 12:40:52.073167 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822252 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 12:40:52.073167 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822254 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 12:40:52.073167 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822257 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 12:40:52.073167 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822259 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 12:40:52.073167 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822262 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 12:40:52.073167 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822264 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 12:40:52.073167 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822267 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 12:40:52.073167 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822270 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 12:40:52.073167 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822273 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 12:40:52.073167 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822276 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 12:40:52.073167 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822278 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 12:40:52.073167 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822281 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 12:40:52.073167 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822283 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 12:40:52.073167 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822286 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 12:40:52.073167 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822288 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 12:40:52.073167 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822291 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 12:40:52.093676 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822293 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 12:40:52.093676 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822296 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 12:40:52.093676 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822300 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 12:40:52.093676 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822304 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 12:40:52.093676 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822307 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 12:40:52.093676 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822310 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 12:40:52.093676 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822312 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 12:40:52.093676 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822315 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 12:40:52.093676 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822318 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 12:40:52.093676 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822320 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 12:40:52.093676 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822323 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 12:40:52.093676 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822325 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 12:40:52.093676 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822328 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 12:40:52.093676 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822330 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 12:40:52.093676 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822332 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 12:40:52.093676 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822338 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 12:40:52.093676 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822340 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 12:40:52.093676 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822344 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 12:40:52.093676 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822347 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 12:40:52.096464 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822350 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 12:40:52.096464 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822352 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 12:40:52.096464 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822355 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 12:40:52.096464 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822358 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 12:40:52.096464 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822361 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 12:40:52.096464 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822363 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 12:40:52.096464 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822366 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 12:40:52.096464 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822368 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 12:40:52.096464 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822371 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 12:40:52.096464 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822373 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 12:40:52.096464 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822376 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 12:40:52.096464 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822379 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 12:40:52.096464 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822381 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 12:40:52.096464 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822384 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 12:40:52.096464 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822386 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 12:40:52.096464 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822388 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 12:40:52.096464 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822391 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 12:40:52.096464 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822393 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 12:40:52.096464 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822396 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 12:40:52.096464 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822398 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 22 12:40:52.098588 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822401 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 12:40:52.098588 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822403 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 12:40:52.098588 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822406 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 12:40:52.098588 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822408 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 12:40:52.098588 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822410 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 12:40:52.098588 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822413 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 12:40:52.098588 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822416 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 12:40:52.098588 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822418 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 12:40:52.098588 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822423 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 12:40:52.098588 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822426 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 12:40:52.098588 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822431 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 12:40:52.098588 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822434 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 12:40:52.098588 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822437 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 12:40:52.098588 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822440 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 12:40:52.098588 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822443 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 12:40:52.098588 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822446 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 12:40:52.098588 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822448 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 12:40:52.098588 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822452 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 12:40:52.098588 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822454 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 12:40:52.101643 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822456 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 12:40:52.101643 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822459 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 12:40:52.101643 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822461 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 12:40:52.101643 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822464 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 12:40:52.101643 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822466 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 12:40:52.101643 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822469 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 12:40:52.101643 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822471 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 12:40:52.101643 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822474 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 12:40:52.101643 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822477 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 12:40:52.101643 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822479 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 12:40:52.101643 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822482 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 12:40:52.101643 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822484 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 12:40:52.101643 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822487 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 12:40:52.101643 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822489 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 12:40:52.101643 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.822492 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 12:40:52.101643 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822557 2579 flags.go:64] FLAG: --address="0.0.0.0" Jul 22 12:40:52.101643 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822564 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 22 12:40:52.101643 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822570 2579 flags.go:64] FLAG: --anonymous-auth="true" Jul 22 12:40:52.101643 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822574 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 22 12:40:52.101643 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822578 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 22 12:40:52.101643 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822582 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 22 12:40:52.105121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822588 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 22 12:40:52.105121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822592 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 22 12:40:52.105121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822595 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 22 12:40:52.105121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822598 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 22 12:40:52.105121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822601 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 22 12:40:52.105121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822605 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 22 12:40:52.105121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822608 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 22 12:40:52.105121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822611 2579 flags.go:64] FLAG: --cgroup-root="" Jul 22 12:40:52.105121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822614 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 22 12:40:52.105121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822616 2579 flags.go:64] FLAG: --client-ca-file="" Jul 22 12:40:52.105121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822621 2579 flags.go:64] FLAG: --cloud-provider="external" Jul 22 12:40:52.105121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822624 2579 flags.go:64] FLAG: --cluster-dns="[]" Jul 22 12:40:52.105121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822628 2579 flags.go:64] FLAG: --cluster-domain="" Jul 22 12:40:52.105121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822631 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 22 12:40:52.105121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822634 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 22 12:40:52.105121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822637 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 22 12:40:52.105121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822640 2579 flags.go:64] FLAG: --container-log-max-files="5" Jul 22 12:40:52.105121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822644 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 22 12:40:52.105121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822647 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 22 12:40:52.105121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822650 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 22 12:40:52.105121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822653 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 22 12:40:52.105121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822656 2579 flags.go:64] FLAG: --contention-profiling="false" Jul 22 12:40:52.105121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822659 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 22 12:40:52.105121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822662 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 22 12:40:52.107864 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822665 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 22 12:40:52.107864 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822668 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 22 12:40:52.107864 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822671 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 22 12:40:52.107864 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822674 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 22 12:40:52.107864 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822677 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 22 12:40:52.107864 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822680 2579 flags.go:64] FLAG: --enable-load-reader="false" Jul 22 12:40:52.107864 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822683 2579 flags.go:64] FLAG: --enable-server="true" Jul 22 12:40:52.107864 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822686 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 22 12:40:52.107864 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822693 2579 flags.go:64] FLAG: --event-burst="100" Jul 22 12:40:52.107864 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822696 2579 flags.go:64] FLAG: --event-qps="50" Jul 22 12:40:52.107864 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822699 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 22 12:40:52.107864 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822702 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 22 12:40:52.107864 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822704 2579 flags.go:64] FLAG: --eviction-hard="" Jul 22 12:40:52.107864 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822708 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 22 12:40:52.107864 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822711 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 22 12:40:52.107864 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822714 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 22 12:40:52.107864 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822717 2579 flags.go:64] FLAG: --eviction-soft="" Jul 22 12:40:52.107864 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822720 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 22 12:40:52.107864 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822722 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 22 12:40:52.107864 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822726 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 22 12:40:52.107864 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822728 2579 flags.go:64] FLAG: --experimental-mounter-path="" Jul 22 12:40:52.107864 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822731 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 22 12:40:52.107864 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822734 2579 flags.go:64] FLAG: --fail-swap-on="true" Jul 22 12:40:52.107864 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822737 2579 flags.go:64] FLAG: --feature-gates="" Jul 22 12:40:52.107864 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822740 2579 flags.go:64] FLAG: --file-check-frequency="20s" Jul 22 12:40:52.109939 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822743 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 22 12:40:52.109939 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822746 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 22 12:40:52.109939 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822749 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 22 12:40:52.109939 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822752 2579 flags.go:64] FLAG: --healthz-port="10248" Jul 22 12:40:52.109939 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822755 2579 flags.go:64] FLAG: --help="false" Jul 22 12:40:52.109939 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822758 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-129-34.ec2.internal" Jul 22 12:40:52.109939 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822761 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 22 12:40:52.109939 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822763 2579 flags.go:64] FLAG: --http-check-frequency="20s" Jul 22 12:40:52.109939 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822766 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 22 12:40:52.109939 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822770 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 22 12:40:52.109939 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822773 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 22 12:40:52.109939 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822776 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 22 12:40:52.109939 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822779 2579 flags.go:64] FLAG: --image-service-endpoint="" Jul 22 12:40:52.109939 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822781 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 22 12:40:52.109939 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822784 2579 flags.go:64] FLAG: --kube-api-burst="100" Jul 22 12:40:52.109939 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822790 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 22 12:40:52.109939 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822794 2579 flags.go:64] FLAG: --kube-api-qps="50" Jul 22 12:40:52.109939 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822796 2579 flags.go:64] FLAG: --kube-reserved="" Jul 22 12:40:52.109939 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822799 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 22 12:40:52.109939 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822802 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 22 12:40:52.109939 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822805 2579 flags.go:64] FLAG: --kubelet-cgroups="" Jul 22 12:40:52.109939 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822808 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 22 12:40:52.109939 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822811 2579 flags.go:64] FLAG: --lock-file="" Jul 22 12:40:52.109939 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822813 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 22 12:40:52.131060 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822816 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 22 12:40:52.131060 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822819 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 22 12:40:52.131060 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822824 2579 flags.go:64] FLAG: --log-json-split-stream="false" Jul 22 12:40:52.131060 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822828 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 22 12:40:52.131060 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822830 2579 flags.go:64] FLAG: --log-text-split-stream="false" Jul 22 12:40:52.131060 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822833 2579 flags.go:64] FLAG: --logging-format="text" Jul 22 12:40:52.131060 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822836 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 22 12:40:52.131060 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822839 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 22 12:40:52.131060 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822842 2579 flags.go:64] FLAG: --manifest-url="" Jul 22 12:40:52.131060 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822845 2579 flags.go:64] FLAG: --manifest-url-header="" Jul 22 12:40:52.131060 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822849 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 22 12:40:52.131060 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822852 2579 flags.go:64] FLAG: --max-open-files="1000000" Jul 22 12:40:52.131060 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822856 2579 flags.go:64] FLAG: --max-pods="110" Jul 22 12:40:52.131060 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822859 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 22 12:40:52.131060 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822862 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 22 12:40:52.131060 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822864 2579 flags.go:64] FLAG: --memory-manager-policy="None" Jul 22 12:40:52.131060 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822867 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 22 12:40:52.131060 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822870 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 22 12:40:52.131060 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822873 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 22 12:40:52.131060 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822876 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 22 12:40:52.131060 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822882 2579 flags.go:64] FLAG: --node-status-max-images="50" Jul 22 12:40:52.131060 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822885 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 22 12:40:52.131060 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822888 2579 flags.go:64] FLAG: --oom-score-adj="-999" Jul 22 12:40:52.131060 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822893 2579 flags.go:64] FLAG: --pod-cidr="" Jul 22 12:40:52.135681 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822896 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 22 12:40:52.135681 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822902 2579 flags.go:64] FLAG: --pod-manifest-path="" Jul 22 12:40:52.135681 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822905 2579 flags.go:64] FLAG: --pod-max-pids="-1" Jul 22 12:40:52.135681 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822908 2579 flags.go:64] FLAG: --pods-per-core="0" Jul 22 12:40:52.135681 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822911 2579 flags.go:64] FLAG: --port="10250" Jul 22 12:40:52.135681 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822914 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 22 12:40:52.135681 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822917 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f25c83f3c7daedd6" Jul 22 12:40:52.135681 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822933 2579 flags.go:64] FLAG: --qos-reserved="" Jul 22 12:40:52.135681 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822937 2579 flags.go:64] FLAG: --read-only-port="10255" Jul 22 12:40:52.135681 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822940 2579 flags.go:64] FLAG: --register-node="true" Jul 22 12:40:52.135681 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822942 2579 flags.go:64] FLAG: --register-with-taints="" Jul 22 12:40:52.135681 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822946 2579 flags.go:64] FLAG: --registry-burst="10" Jul 22 12:40:52.135681 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822949 2579 flags.go:64] FLAG: --registry-qps="5" Jul 22 12:40:52.135681 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822953 2579 flags.go:64] FLAG: --reserved-cpus="" Jul 22 12:40:52.135681 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822956 2579 flags.go:64] FLAG: --reserved-memory="" Jul 22 12:40:52.135681 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822962 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 22 12:40:52.135681 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822965 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 22 12:40:52.135681 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822968 2579 flags.go:64] FLAG: --rotate-certificates="false" Jul 22 12:40:52.135681 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822971 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 22 12:40:52.135681 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822974 2579 flags.go:64] FLAG: --runonce="false" Jul 22 12:40:52.135681 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822977 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 22 12:40:52.135681 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822980 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 22 12:40:52.135681 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822983 2579 flags.go:64] FLAG: --seccomp-default="false" Jul 22 12:40:52.135681 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822986 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 22 12:40:52.135681 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822988 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 22 12:40:52.161776 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822991 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 22 12:40:52.161776 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822994 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 22 12:40:52.161776 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.822997 2579 flags.go:64] FLAG: --storage-driver-password="root" Jul 22 12:40:52.161776 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.823000 2579 flags.go:64] FLAG: --storage-driver-secure="false" Jul 22 12:40:52.161776 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.823003 2579 flags.go:64] FLAG: --storage-driver-table="stats" Jul 22 12:40:52.161776 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.823006 2579 flags.go:64] FLAG: --storage-driver-user="root" Jul 22 12:40:52.161776 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.823010 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 22 12:40:52.161776 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.823015 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 22 12:40:52.161776 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.823018 2579 flags.go:64] FLAG: --system-cgroups="" Jul 22 12:40:52.161776 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.823021 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 22 12:40:52.161776 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.823026 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 22 12:40:52.161776 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.823029 2579 flags.go:64] FLAG: --tls-cert-file="" Jul 22 12:40:52.161776 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.823032 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 22 12:40:52.161776 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.823036 2579 flags.go:64] FLAG: --tls-min-version="" Jul 22 12:40:52.161776 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.823039 2579 flags.go:64] FLAG: --tls-private-key-file="" Jul 22 12:40:52.161776 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.823042 2579 flags.go:64] FLAG: --topology-manager-policy="none" Jul 22 12:40:52.161776 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.823044 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 22 12:40:52.161776 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.823047 2579 flags.go:64] FLAG: --topology-manager-scope="container" Jul 22 12:40:52.161776 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.823050 2579 flags.go:64] FLAG: --v="2" Jul 22 12:40:52.161776 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.823054 2579 flags.go:64] FLAG: --version="false" Jul 22 12:40:52.161776 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.823059 2579 flags.go:64] FLAG: --vmodule="" Jul 22 12:40:52.161776 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.823062 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 22 12:40:52.161776 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.823066 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 22 12:40:52.161776 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823175 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 12:40:52.161776 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823179 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 12:40:52.162663 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823182 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 12:40:52.162663 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823185 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 12:40:52.162663 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823188 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 12:40:52.162663 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823190 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 12:40:52.162663 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823193 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 12:40:52.162663 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823195 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 12:40:52.162663 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823198 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 12:40:52.162663 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823200 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 12:40:52.162663 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823203 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 12:40:52.162663 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823205 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 12:40:52.162663 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823208 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 12:40:52.162663 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823210 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 12:40:52.162663 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823213 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 12:40:52.162663 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823217 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 12:40:52.162663 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823220 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 12:40:52.162663 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823223 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 12:40:52.162663 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823226 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 12:40:52.162663 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823228 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 12:40:52.162663 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823231 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 12:40:52.162663 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823233 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 12:40:52.163462 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823236 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 12:40:52.163462 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823239 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 12:40:52.163462 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823241 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 12:40:52.163462 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823244 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 12:40:52.163462 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823247 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 12:40:52.163462 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823249 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 12:40:52.163462 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823253 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 12:40:52.163462 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823255 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 12:40:52.163462 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823258 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 12:40:52.163462 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823261 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 12:40:52.163462 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823263 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 12:40:52.163462 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823266 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 12:40:52.163462 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823270 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 12:40:52.163462 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823273 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 12:40:52.163462 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823276 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 12:40:52.163462 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823279 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 12:40:52.163462 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823281 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 12:40:52.163462 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823284 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 12:40:52.163462 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823287 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 12:40:52.164288 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823289 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 12:40:52.164288 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823292 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 12:40:52.164288 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823295 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 12:40:52.164288 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823297 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 12:40:52.164288 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823300 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 12:40:52.164288 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823302 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 12:40:52.164288 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823306 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 12:40:52.164288 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823310 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 12:40:52.164288 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823313 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 12:40:52.164288 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823315 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 12:40:52.164288 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823317 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 12:40:52.164288 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823320 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 12:40:52.164288 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823323 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 12:40:52.164288 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823325 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 12:40:52.164288 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823328 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 12:40:52.164288 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823330 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 12:40:52.164288 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823333 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 12:40:52.164288 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823335 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 12:40:52.164288 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823338 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 12:40:52.164288 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823341 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 12:40:52.165123 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823344 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 12:40:52.165123 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823346 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 12:40:52.165123 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823349 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 12:40:52.165123 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823352 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 12:40:52.165123 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823355 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 12:40:52.165123 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823357 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 12:40:52.165123 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823360 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 12:40:52.165123 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823362 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 12:40:52.165123 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823364 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 12:40:52.165123 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823367 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 12:40:52.165123 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823371 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 12:40:52.165123 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823375 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 12:40:52.165123 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823379 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 12:40:52.165123 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823382 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 12:40:52.165123 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823385 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 12:40:52.165123 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823387 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 22 12:40:52.165123 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823390 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 12:40:52.165123 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823393 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 12:40:52.165123 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823397 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 12:40:52.165123 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823401 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 12:40:52.165942 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823403 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 12:40:52.165942 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823406 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 12:40:52.165942 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823408 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 12:40:52.165942 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823411 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 12:40:52.165942 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823414 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 12:40:52.165942 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823417 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 12:40:52.165942 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823419 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 12:40:52.165942 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823422 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 12:40:52.165942 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823424 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 12:40:52.165942 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823427 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 12:40:52.165942 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823429 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 12:40:52.165942 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823432 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 12:40:52.165942 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823434 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 12:40:52.165942 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823437 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 12:40:52.165942 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823439 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 12:40:52.165942 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823443 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 12:40:52.165942 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823445 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 12:40:52.165942 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.823448 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 12:40:52.166744 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.823939 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 12:40:52.166744 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.830270 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 22 12:40:52.166744 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.830285 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 22 12:40:52.166744 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.830297 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 22 12:40:52.166744 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.830303 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 22 12:40:52.166744 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830359 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 12:40:52.166744 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830365 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 12:40:52.166744 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830368 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 12:40:52.166744 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830371 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 12:40:52.166744 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830374 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 12:40:52.166744 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830377 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 12:40:52.166744 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830379 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 12:40:52.166744 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830382 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 12:40:52.166744 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830385 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 12:40:52.166744 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830388 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 12:40:52.167562 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830391 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 12:40:52.167562 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830394 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 12:40:52.167562 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830397 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 12:40:52.167562 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830399 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 12:40:52.167562 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830402 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 12:40:52.167562 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830405 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 12:40:52.167562 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830407 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 12:40:52.167562 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830410 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 12:40:52.167562 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830413 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 12:40:52.167562 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830415 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 12:40:52.167562 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830420 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 12:40:52.167562 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830423 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 12:40:52.167562 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830425 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 12:40:52.167562 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830430 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 12:40:52.167562 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830434 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 12:40:52.167562 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830436 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 12:40:52.167562 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830439 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 12:40:52.167562 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830441 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 12:40:52.167562 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830444 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 12:40:52.167562 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830446 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 12:40:52.168375 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830449 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 12:40:52.168375 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830452 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 12:40:52.168375 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830454 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 12:40:52.168375 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830457 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 12:40:52.168375 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830459 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 12:40:52.168375 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830462 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 12:40:52.168375 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830464 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 12:40:52.168375 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830467 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 12:40:52.168375 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830470 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 12:40:52.168375 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830472 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 12:40:52.168375 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830475 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 12:40:52.168375 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830477 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 12:40:52.168375 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830480 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 12:40:52.168375 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830482 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 12:40:52.168375 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830485 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 12:40:52.168375 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830487 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 12:40:52.168375 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830490 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 12:40:52.168375 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830492 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 12:40:52.168375 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830495 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 12:40:52.168375 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830497 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 12:40:52.169204 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830499 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 12:40:52.169204 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830502 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 12:40:52.169204 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830506 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 12:40:52.169204 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830509 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 12:40:52.169204 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830512 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 12:40:52.169204 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830515 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 12:40:52.169204 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830518 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 22 12:40:52.169204 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830521 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 12:40:52.169204 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830524 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 12:40:52.169204 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830528 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 12:40:52.169204 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830532 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 12:40:52.169204 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830535 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 12:40:52.169204 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830537 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 12:40:52.169204 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830540 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 12:40:52.169204 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830542 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 12:40:52.169204 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830545 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 12:40:52.169204 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830547 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 12:40:52.169204 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830550 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 12:40:52.169204 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830553 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 12:40:52.170033 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830555 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 12:40:52.170033 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830558 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 12:40:52.170033 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830561 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 12:40:52.170033 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830563 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 12:40:52.170033 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830566 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 12:40:52.170033 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830569 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 12:40:52.170033 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830571 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 12:40:52.170033 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830574 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 12:40:52.170033 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830576 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 12:40:52.170033 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830579 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 12:40:52.170033 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830581 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 12:40:52.170033 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830584 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 12:40:52.170033 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830587 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 12:40:52.170033 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830589 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 12:40:52.170033 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830592 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 12:40:52.170033 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830596 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 12:40:52.170033 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830598 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 12:40:52.170033 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830601 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 12:40:52.170033 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830604 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 12:40:52.170033 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830608 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 12:40:52.170849 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830612 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 12:40:52.170849 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830615 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 12:40:52.170849 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830619 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 12:40:52.170849 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830622 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 12:40:52.170849 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830626 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 12:40:52.170849 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830629 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 12:40:52.170849 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830632 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 12:40:52.170849 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830635 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 12:40:52.170849 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830638 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 12:40:52.170849 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830641 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 12:40:52.170849 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.830646 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 12:40:52.170849 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830756 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 22 12:40:52.170849 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830761 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 22 12:40:52.170849 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830764 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 22 12:40:52.170849 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830767 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 22 12:40:52.171668 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830770 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 22 12:40:52.171668 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830773 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 22 12:40:52.171668 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830775 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 22 12:40:52.171668 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830778 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 22 12:40:52.171668 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830781 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 22 12:40:52.171668 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830783 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 22 12:40:52.171668 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830786 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 22 12:40:52.171668 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830788 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 22 12:40:52.171668 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830791 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 22 12:40:52.171668 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830793 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 22 12:40:52.171668 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830795 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 22 12:40:52.171668 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830798 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 22 12:40:52.171668 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830800 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 22 12:40:52.171668 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830804 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 22 12:40:52.171668 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830807 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 22 12:40:52.171668 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830811 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 22 12:40:52.171668 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830814 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 22 12:40:52.171668 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830816 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 22 12:40:52.171668 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830820 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 22 12:40:52.172490 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830824 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 22 12:40:52.172490 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830827 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 22 12:40:52.172490 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830830 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 22 12:40:52.172490 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830832 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 22 12:40:52.172490 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830835 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 22 12:40:52.172490 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830837 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 22 12:40:52.172490 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830840 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 22 12:40:52.172490 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830842 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 22 12:40:52.172490 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830845 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 22 12:40:52.172490 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830849 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 22 12:40:52.172490 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830852 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 22 12:40:52.172490 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830855 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 22 12:40:52.172490 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830857 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 22 12:40:52.172490 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830860 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 22 12:40:52.172490 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830863 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 22 12:40:52.172490 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830866 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 22 12:40:52.172490 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830869 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 22 12:40:52.172490 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830872 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 22 12:40:52.172490 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830874 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 22 12:40:52.172490 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830877 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 22 12:40:52.173313 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830879 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 22 12:40:52.173313 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830881 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 22 12:40:52.173313 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830884 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 22 12:40:52.173313 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830886 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 22 12:40:52.173313 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830889 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 22 12:40:52.173313 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830892 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 22 12:40:52.173313 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830897 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 22 12:40:52.173313 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830900 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 22 12:40:52.173313 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830904 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 22 12:40:52.173313 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830907 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 22 12:40:52.173313 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830910 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 22 12:40:52.173313 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830912 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 22 12:40:52.173313 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830915 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 22 12:40:52.173313 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830918 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 22 12:40:52.173313 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830935 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 22 12:40:52.173313 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830939 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 22 12:40:52.173313 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830942 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 22 12:40:52.173313 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830945 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 22 12:40:52.173313 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830947 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 22 12:40:52.173313 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830950 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 22 12:40:52.174135 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830952 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 22 12:40:52.174135 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830955 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 22 12:40:52.174135 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830958 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 22 12:40:52.174135 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830960 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 22 12:40:52.174135 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830963 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 22 12:40:52.174135 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830966 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 22 12:40:52.174135 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830969 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 22 12:40:52.174135 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830971 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 22 12:40:52.174135 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830974 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 22 12:40:52.174135 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830976 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 22 12:40:52.174135 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830978 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 22 12:40:52.174135 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830981 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 22 12:40:52.174135 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830984 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 22 12:40:52.174135 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830986 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 22 12:40:52.174135 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830988 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 22 12:40:52.174135 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830991 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 22 12:40:52.174135 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830993 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 22 12:40:52.174135 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.830996 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 22 12:40:52.174135 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.831001 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 22 12:40:52.174135 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.831003 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 22 12:40:52.174957 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.831007 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 22 12:40:52.174957 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.831010 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 22 12:40:52.174957 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.831013 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 22 12:40:52.174957 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.831016 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 22 12:40:52.174957 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.831018 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 22 12:40:52.174957 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.831021 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 22 12:40:52.174957 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.831027 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 22 12:40:52.174957 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.831030 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 22 12:40:52.174957 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.831033 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 22 12:40:52.174957 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.831035 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 22 12:40:52.174957 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.831037 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 22 12:40:52.174957 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.831040 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 22 12:40:52.174957 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.831042 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 22 12:40:52.174957 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.831045 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 22 12:40:52.174957 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.831047 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 22 12:40:52.174957 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:51.831050 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 22 12:40:52.175760 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.831055 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 22 12:40:52.175760 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.831149 2579 server.go:962] "Client rotation is on, will bootstrap in background" Jul 22 12:40:52.175760 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.833101 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 22 12:40:52.175760 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.834176 2579 server.go:1019] "Starting client certificate rotation" Jul 22 12:40:52.175760 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.834276 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 12:40:52.175760 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.834311 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 22 12:40:52.175760 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.836072 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 12:40:52.175760 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.838464 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 22 12:40:52.175760 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.849340 2579 log.go:25] "Validated CRI v1 runtime API" Jul 22 12:40:52.175760 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.851961 2579 log.go:25] "Validated CRI v1 image API" Jul 22 12:40:52.175760 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.852720 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 22 12:40:52.175760 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.856147 2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 eb9d5d2a-aa4d-4039-a76c-51c705b08797:/dev/nvme0n1p3 f14a05c3-2c9b-4749-9f6d-50d182db2c6b:/dev/nvme0n1p4] Jul 22 12:40:52.176180 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.856163 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 22 12:40:52.176583 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.861544 2579 manager.go:216] Machine: {Timestamp:2026-07-22 12:40:51.860491731 +0000 UTC m=+0.261722645 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101478 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20015a781d2888a78d4c9ef8f813b8 SystemUUID:ec20015a-781d-2888-a78d-4c9ef8f813b8 BootID:d5aa6c28-b18c-432f-bfaa-ce99eea6e823 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1b:b6:e0:a4:2b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1b:b6:e0:a4:2b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:de:25:03:a9:9e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 22 12:40:52.176583 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.861647 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 22 12:40:52.176583 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.861725 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 22 12:40:52.176583 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.863474 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 12:40:52.176583 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.864156 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 22 12:40:52.176583 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.864182 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-34.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 22 12:40:52.176583 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.864760 2579 topology_manager.go:138] "Creating topology manager with none policy" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.864770 2579 container_manager_linux.go:309] "Creating device plugin manager" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.864778 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.864792 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.866026 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.869852 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.869974 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.870002 2579 kubelet.go:486] "Attempting to sync node with API server" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.870012 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.870022 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.870031 2579 kubelet.go:398] "Adding apiserver pod source" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.870040 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.870430 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.870484 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.872376 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.874197 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.874223 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.875255 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.875267 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.875274 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.875280 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.875289 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.875295 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.875306 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.875313 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.876912 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.876954 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.876984 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.876994 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.878103 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.878126 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:51.880727 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-34.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.880749 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-34.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:51.880766 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.882187 2579 server.go:1268] "Started kubelet" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.882427 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.882519 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.882567 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.883899 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 22 12:40:52.181121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.884586 2579 server.go:310] "Adding debug handlers to kubelet server" Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:51.887591 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-34.ec2.internal.18c49d11be847ca1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-34.ec2.internal,UID:ip-10-0-129-34.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-34.ec2.internal,},FirstTimestamp:2026-07-22 12:40:51.881917601 +0000 UTC m=+0.283148512,LastTimestamp:2026-07-22 12:40:51.881917601 +0000 UTC m=+0.283148512,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-34.ec2.internal,}" Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:51.888512 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.890459 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pvrsn" Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.891107 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.891114 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:51.891166 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-34.ec2.internal\" not found" Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:51.893231 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-34.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.893317 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.893362 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.893374 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.893481 2579 reconstruct.go:97] "Volume reconstruction finished" Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.893492 2579 reconciler.go:29] "Reconciler: start to sync state" Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:51.893509 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.893563 2579 factory.go:153] Registering CRI-O factory Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.893588 2579 factory.go:223] Registration of the crio container factory successfully Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.893640 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.893649 2579 factory.go:55] Registering systemd factory Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.893657 2579 factory.go:223] Registration of the systemd container factory successfully Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.893678 2579 factory.go:103] Registering Raw factory Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.893692 2579 manager.go:1196] Started watching for new ooms in manager Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.894097 2579 manager.go:349] Starting recovery of all containers Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.897403 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pvrsn" Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.903311 2579 manager.go:354] Recovery completed Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.910563 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.912349 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-34.ec2.internal" event="NodeHasSufficientMemory" Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.912380 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-34.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.912394 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-34.ec2.internal" event="NodeHasSufficientPID" Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.912960 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.912970 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.912988 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.912986 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 12:40:52.279963 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.913681 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-34.ec2.internal" event="NodeHasSufficientMemory" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.913711 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-34.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.913723 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-34.ec2.internal" event="NodeHasSufficientPID" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.917345 2579 policy_none.go:49] "None policy: Start" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.917361 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.917375 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.919374 2579 policy_none.go:47] "Start" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.966461 2579 manager.go:337] "Starting Device Plugin manager" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:51.966495 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.966505 2579 server.go:87] "Starting device plugin registration server" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.966723 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.966733 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.966848 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.966947 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:51.966957 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:51.967413 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:51.967470 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-34.ec2.internal\" not found" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:52.059683 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:52.060865 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:52.060901 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:52.060937 2579 kubelet.go:2448] "Starting kubelet main sync loop" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:52.060979 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:52.063456 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:52.067519 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:52.068327 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-34.ec2.internal" event="NodeHasSufficientMemory" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:52.068357 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-34.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:52.068367 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-34.ec2.internal" event="NodeHasSufficientPID" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:52.068388 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-34.ec2.internal" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:52.076641 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-34.ec2.internal" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:52.076659 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-34.ec2.internal\": node \"ip-10-0-129-34.ec2.internal\" not found" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:52.092529 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-34.ec2.internal\" not found" Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:52.161619 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-34.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-34.ec2.internal"] Jul 22 12:40:52.291153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:52.161692 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:53.906506 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:53.906570 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-21 12:35:52 +0000 UTC" deadline="2028-04-02 03:38:27.396143774 +0000 UTC" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:53.906604 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14870h57m33.489543951s" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:53.906624 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-34.ec2.internal" event="NodeHasSufficientMemory" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:53.906653 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-34.ec2.internal\" not found" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:53.906655 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-34.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:53.906691 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-34.ec2.internal" event="NodeHasSufficientPID" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:53.909369 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:53.909477 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-34.ec2.internal" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:53.909507 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:53.910150 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-34.ec2.internal" event="NodeHasSufficientMemory" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:53.910168 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-34.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:53.910181 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-34.ec2.internal" event="NodeHasSufficientPID" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:53.910197 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-34.ec2.internal" event="NodeHasSufficientMemory" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:53.910211 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-34.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:53.910220 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-34.ec2.internal" event="NodeHasSufficientPID" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:53.912672 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-34.ec2.internal" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:53.912696 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:53.913462 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-34.ec2.internal" event="NodeHasSufficientMemory" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:53.913488 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-34.ec2.internal" event="NodeHasNoDiskPressure" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:53.913497 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-34.ec2.internal" event="NodeHasSufficientPID" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:53.918596 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:53.922955 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:53.944756 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-34.ec2.internal\" not found" node="ip-10-0-129-34.ec2.internal" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:53.949702 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-34.ec2.internal\" not found" node="ip-10-0-129-34.ec2.internal" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:54.007604 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-34.ec2.internal\" not found" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:54.007679 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8eac028a224ef233879714f64598e3db-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-34.ec2.internal\" (UID: \"8eac028a224ef233879714f64598e3db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-34.ec2.internal" Jul 22 12:40:54.076057 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:54.007703 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8eac028a224ef233879714f64598e3db-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-34.ec2.internal\" (UID: \"8eac028a224ef233879714f64598e3db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-34.ec2.internal" Jul 22 12:40:54.077618 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:54.007724 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ea81ad4ab5ce353c7d7f37c0cd1009b2-config\") pod \"kube-apiserver-proxy-ip-10-0-129-34.ec2.internal\" (UID: \"ea81ad4ab5ce353c7d7f37c0cd1009b2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-34.ec2.internal" Jul 22 12:40:54.110326 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:54.108615 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-34.ec2.internal\" not found" Jul 22 12:40:54.110326 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:54.108636 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8eac028a224ef233879714f64598e3db-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-34.ec2.internal\" (UID: \"8eac028a224ef233879714f64598e3db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-34.ec2.internal" Jul 22 12:40:54.110326 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:54.108685 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8eac028a224ef233879714f64598e3db-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-34.ec2.internal\" (UID: \"8eac028a224ef233879714f64598e3db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-34.ec2.internal" Jul 22 12:40:54.110326 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:54.108705 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8eac028a224ef233879714f64598e3db-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-34.ec2.internal\" (UID: \"8eac028a224ef233879714f64598e3db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-34.ec2.internal" Jul 22 12:40:54.110326 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:54.108712 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ea81ad4ab5ce353c7d7f37c0cd1009b2-config\") pod \"kube-apiserver-proxy-ip-10-0-129-34.ec2.internal\" (UID: \"ea81ad4ab5ce353c7d7f37c0cd1009b2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-34.ec2.internal" Jul 22 12:40:54.110326 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:54.108754 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ea81ad4ab5ce353c7d7f37c0cd1009b2-config\") pod \"kube-apiserver-proxy-ip-10-0-129-34.ec2.internal\" (UID: \"ea81ad4ab5ce353c7d7f37c0cd1009b2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-34.ec2.internal" Jul 22 12:40:54.110326 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:54.108781 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8eac028a224ef233879714f64598e3db-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-34.ec2.internal\" (UID: \"8eac028a224ef233879714f64598e3db\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-34.ec2.internal" Jul 22 12:40:54.208939 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:54.208884 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-34.ec2.internal\" not found" Jul 22 12:40:54.249112 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:54.249073 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-34.ec2.internal" Jul 22 12:40:54.252449 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:54.252432 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-34.ec2.internal" Jul 22 12:40:54.309434 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:54.309401 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-34.ec2.internal\" not found" Jul 22 12:40:54.410349 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:54.410279 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-34.ec2.internal\" not found" Jul 22 12:40:54.510685 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:54.510654 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-34.ec2.internal\" not found" Jul 22 12:40:54.611829 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:54.611791 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-34.ec2.internal\" not found" Jul 22 12:40:54.712752 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:54.712678 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-34.ec2.internal\" not found" Jul 22 12:40:54.813719 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:54.813685 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-34.ec2.internal\" not found" Jul 22 12:40:54.910114 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:54.910086 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 22 12:40:54.991610 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:54.991534 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-34.ec2.internal" Jul 22 12:40:55.007868 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.007844 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 12:40:55.008675 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.008661 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-34.ec2.internal" Jul 22 12:40:55.022839 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.022813 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 22 12:40:55.043253 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:55.043222 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eac028a224ef233879714f64598e3db.slice/crio-c18517eda4b76e0282666041b87c5c7052b835bdf8695e423c88577b76890aa3 WatchSource:0}: Error finding container c18517eda4b76e0282666041b87c5c7052b835bdf8695e423c88577b76890aa3: Status 404 returned error can't find the container with id c18517eda4b76e0282666041b87c5c7052b835bdf8695e423c88577b76890aa3 Jul 22 12:40:55.047226 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.047213 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 12:40:55.202660 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:55.202631 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea81ad4ab5ce353c7d7f37c0cd1009b2.slice/crio-8367a60d63df022717a67cdfb002c0c0df8c5479755298ea541942b276c7a29b WatchSource:0}: Error finding container 8367a60d63df022717a67cdfb002c0c0df8c5479755298ea541942b276c7a29b: Status 404 returned error can't find the container with id 8367a60d63df022717a67cdfb002c0c0df8c5479755298ea541942b276c7a29b Jul 22 12:40:55.907431 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.907399 2579 apiserver.go:52] "Watching apiserver" Jul 22 12:40:55.907620 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.907462 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 22 12:40:55.914846 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.914744 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-34.ec2.internal" event={"ID":"ea81ad4ab5ce353c7d7f37c0cd1009b2","Type":"ContainerStarted","Data":"8367a60d63df022717a67cdfb002c0c0df8c5479755298ea541942b276c7a29b"} Jul 22 12:40:55.916334 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.916301 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-34.ec2.internal" event={"ID":"8eac028a224ef233879714f64598e3db","Type":"ContainerStarted","Data":"c18517eda4b76e0282666041b87c5c7052b835bdf8695e423c88577b76890aa3"} Jul 22 12:40:55.916873 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.916850 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 22 12:40:55.917454 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.917428 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-wgxgn","openshift-multus/multus-pgrh7","openshift-multus/network-metrics-daemon-jch9g","openshift-ovn-kubernetes/ovnkube-node-8ddj6","kube-system/konnectivity-agent-ns7kz","openshift-dns/node-resolver-fgc67","openshift-image-registry/node-ca-w6nhq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-34.ec2.internal","openshift-multus/multus-additional-cni-plugins-sfx4m","openshift-network-diagnostics/network-check-target-6ghfv","openshift-network-operator/iptables-alerter-5c5j7","kube-system/global-pull-secret-syncer-7gsw5","kube-system/kube-apiserver-proxy-ip-10-0-129-34.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w"] Jul 22 12:40:55.920309 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.920278 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w6nhq" Jul 22 12:40:55.921731 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.921544 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 22 12:40:55.923121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.922707 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:40:55.923121 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:55.922773 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ghfv" podUID="9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed" Jul 22 12:40:55.923121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.922883 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wlg9p\"" Jul 22 12:40:55.923341 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.923206 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 22 12:40:55.923395 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.923385 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 22 12:40:55.923804 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.923635 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 22 12:40:55.925791 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.925505 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:40:55.925791 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:55.925574 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jch9g" podUID="3c54132f-3e4d-427b-b38d-cd0daabb33d9" Jul 22 12:40:55.928133 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.928103 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:55.931104 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.930660 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 22 12:40:55.931228 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.931176 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 22 12:40:55.931305 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.931288 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ns7kz" Jul 22 12:40:55.931574 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.931388 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 22 12:40:55.931574 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.931459 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-v8zsk\"" Jul 22 12:40:55.932434 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.932414 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 22 12:40:55.932664 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.932647 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 22 12:40:55.932791 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.932739 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 22 12:40:55.933537 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.933290 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" Jul 22 12:40:55.934401 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.934376 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 22 12:40:55.934711 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.934692 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pbtqc\"" Jul 22 12:40:55.935048 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.935029 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 22 12:40:55.936417 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.935837 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 22 12:40:55.936417 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.936321 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bkr29\"" Jul 22 12:40:55.936610 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.936590 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 22 12:40:55.936694 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.936676 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sfx4m" Jul 22 12:40:55.936870 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.936855 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 22 12:40:55.939072 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.938822 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:55.939654 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.939630 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 22 12:40:55.940094 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.940075 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 22 12:40:55.940300 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.940281 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j962k\"" Jul 22 12:40:55.940379 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.940360 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 22 12:40:55.940631 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.940593 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 22 12:40:55.940712 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.940681 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 22 12:40:55.941419 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.941365 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 22 12:40:55.941508 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.941450 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 22 12:40:55.941686 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.941666 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cfklb\"" Jul 22 12:40:55.942074 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.942058 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pgrh7" Jul 22 12:40:55.945152 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.944827 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5c5j7" Jul 22 12:40:55.945152 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.944900 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6f6vp\"" Jul 22 12:40:55.945152 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.945138 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 22 12:40:55.948332 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.947429 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 22 12:40:55.948332 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.947925 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 22 12:40:55.948332 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.948054 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 22 12:40:55.948332 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.948113 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:40:55.948332 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:55.948186 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gsw5" podUID="28d7c937-a848-4d26-861f-29f9d2b0baf7" Jul 22 12:40:55.948332 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.948198 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9gsbd\"" Jul 22 12:40:55.948994 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.948966 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ltz9b" Jul 22 12:40:55.950216 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.950198 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fgc67" Jul 22 12:40:55.953286 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.953264 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 22 12:40:55.953380 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.953321 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 22 12:40:55.953593 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.953575 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s5qgh\"" Jul 22 12:40:55.958159 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.958140 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ltz9b" Jul 22 12:40:55.994919 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:55.994899 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 22 12:40:56.023539 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.023511 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28d7c937-a848-4d26-861f-29f9d2b0baf7-original-pull-secret\") pod \"global-pull-secret-syncer-7gsw5\" (UID: \"28d7c937-a848-4d26-861f-29f9d2b0baf7\") " pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:40:56.023674 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.023547 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fee5353-19d0-4530-baff-6831516314d3-cni-binary-copy\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.023674 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.023567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-multus-socket-dir-parent\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.023674 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.023584 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/55113322-3d54-4862-a5da-17d438ad52a7-device-dir\") pod \"aws-ebs-csi-driver-node-82v8w\" (UID: \"55113322-3d54-4862-a5da-17d438ad52a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" Jul 22 12:40:56.023674 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.023605 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-sys\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.023674 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.023624 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-host-cni-netd\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.023674 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.023660 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.024077 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.023690 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b28e0ca4-8512-45da-91ed-2cde1491b2e2-ovnkube-config\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.024077 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.023708 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3dc0c773-526a-4eee-8155-77cf64929958-konnectivity-ca\") pod \"konnectivity-agent-ns7kz\" (UID: \"3dc0c773-526a-4eee-8155-77cf64929958\") " pod="kube-system/konnectivity-agent-ns7kz" Jul 22 12:40:56.024077 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.023744 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-etc-kubernetes\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.024077 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.023789 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24f69e77-f691-4a49-9cbf-8000995cd2a0-cni-binary-copy\") pod \"multus-additional-cni-plugins-sfx4m\" (UID: \"24f69e77-f691-4a49-9cbf-8000995cd2a0\") " pod="openshift-multus/multus-additional-cni-plugins-sfx4m" Jul 22 12:40:56.024077 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.023824 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jks8\" (UniqueName: \"kubernetes.io/projected/24f69e77-f691-4a49-9cbf-8000995cd2a0-kube-api-access-4jks8\") pod \"multus-additional-cni-plugins-sfx4m\" (UID: \"24f69e77-f691-4a49-9cbf-8000995cd2a0\") " pod="openshift-multus/multus-additional-cni-plugins-sfx4m" Jul 22 12:40:56.024077 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.023856 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-lib-modules\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.024077 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.023907 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c504a9e7-e48d-496d-8a1d-3b307361714c-serviceca\") pod \"node-ca-w6nhq\" (UID: \"c504a9e7-e48d-496d-8a1d-3b307361714c\") " pod="openshift-image-registry/node-ca-w6nhq" Jul 22 12:40:56.024077 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.023933 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-host-kubelet\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.024077 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.023959 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-multus-cni-dir\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.024077 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.023985 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/24f69e77-f691-4a49-9cbf-8000995cd2a0-os-release\") pod \"multus-additional-cni-plugins-sfx4m\" (UID: \"24f69e77-f691-4a49-9cbf-8000995cd2a0\") " pod="openshift-multus/multus-additional-cni-plugins-sfx4m" Jul 22 12:40:56.024077 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024023 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-host-run-k8s-cni-cncf-io\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.024077 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024048 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfhmz\" (UniqueName: \"kubernetes.io/projected/d5897b36-633d-4ccc-b69f-7b50e656545c-kube-api-access-bfhmz\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.024679 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024088 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq4zw\" (UniqueName: \"kubernetes.io/projected/c504a9e7-e48d-496d-8a1d-3b307361714c-kube-api-access-lq4zw\") pod \"node-ca-w6nhq\" (UID: \"c504a9e7-e48d-496d-8a1d-3b307361714c\") " pod="openshift-image-registry/node-ca-w6nhq" Jul 22 12:40:56.024679 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024111 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-host-var-lib-kubelet\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.024679 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024141 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-host\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.024679 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024167 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3dc0c773-526a-4eee-8155-77cf64929958-agent-certs\") pod \"konnectivity-agent-ns7kz\" (UID: \"3dc0c773-526a-4eee-8155-77cf64929958\") " pod="kube-system/konnectivity-agent-ns7kz" Jul 22 12:40:56.024679 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024193 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/55113322-3d54-4862-a5da-17d438ad52a7-registration-dir\") pod \"aws-ebs-csi-driver-node-82v8w\" (UID: \"55113322-3d54-4862-a5da-17d438ad52a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" Jul 22 12:40:56.024679 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024216 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-etc-sysctl-conf\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.024679 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024240 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-var-lib-openvswitch\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.024679 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024272 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-cnibin\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.024679 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024323 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxmtx\" (UniqueName: \"kubernetes.io/projected/9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed-kube-api-access-hxmtx\") pod \"network-check-target-6ghfv\" (UID: \"9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed\") " pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:40:56.024679 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024364 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/55113322-3d54-4862-a5da-17d438ad52a7-sys-fs\") pod \"aws-ebs-csi-driver-node-82v8w\" (UID: \"55113322-3d54-4862-a5da-17d438ad52a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" Jul 22 12:40:56.024679 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024391 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-etc-kubernetes\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.024679 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024414 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-run\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.024679 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024435 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b80aabc-b7ef-4649-bb58-4e0a39906c4f-tmp-dir\") pod \"node-resolver-fgc67\" (UID: \"8b80aabc-b7ef-4649-bb58-4e0a39906c4f\") " pod="openshift-dns/node-resolver-fgc67" Jul 22 12:40:56.024679 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024455 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-systemd-units\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.025397 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024476 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2fee5353-19d0-4530-baff-6831516314d3-multus-daemon-config\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.025397 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024507 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/55113322-3d54-4862-a5da-17d438ad52a7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-82v8w\" (UID: \"55113322-3d54-4862-a5da-17d438ad52a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" Jul 22 12:40:56.025397 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024529 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/24f69e77-f691-4a49-9cbf-8000995cd2a0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sfx4m\" (UID: \"24f69e77-f691-4a49-9cbf-8000995cd2a0\") " pod="openshift-multus/multus-additional-cni-plugins-sfx4m" Jul 22 12:40:56.025397 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024552 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b80aabc-b7ef-4649-bb58-4e0a39906c4f-hosts-file\") pod \"node-resolver-fgc67\" (UID: \"8b80aabc-b7ef-4649-bb58-4e0a39906c4f\") " pod="openshift-dns/node-resolver-fgc67" Jul 22 12:40:56.025397 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024571 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-host-slash\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.025397 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024606 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-run-systemd\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.025397 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024637 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-run-openvswitch\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.025397 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024663 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6hrz\" (UniqueName: \"kubernetes.io/projected/b28e0ca4-8512-45da-91ed-2cde1491b2e2-kube-api-access-t6hrz\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.025397 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024687 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f3b3f37-8156-495b-b73b-011f8decc3fa-host-slash\") pod \"iptables-alerter-5c5j7\" (UID: \"6f3b3f37-8156-495b-b73b-011f8decc3fa\") " pod="openshift-network-operator/iptables-alerter-5c5j7" Jul 22 12:40:56.025397 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024710 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-log-socket\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.025397 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024733 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b28e0ca4-8512-45da-91ed-2cde1491b2e2-ovnkube-script-lib\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.025397 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024756 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-hostroot\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.025397 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024780 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6f3b3f37-8156-495b-b73b-011f8decc3fa-iptables-alerter-script\") pod \"iptables-alerter-5c5j7\" (UID: \"6f3b3f37-8156-495b-b73b-011f8decc3fa\") " pod="openshift-network-operator/iptables-alerter-5c5j7" Jul 22 12:40:56.025928 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024841 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktmv9\" (UniqueName: \"kubernetes.io/projected/6f3b3f37-8156-495b-b73b-011f8decc3fa-kube-api-access-ktmv9\") pod \"iptables-alerter-5c5j7\" (UID: \"6f3b3f37-8156-495b-b73b-011f8decc3fa\") " pod="openshift-network-operator/iptables-alerter-5c5j7" Jul 22 12:40:56.025928 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024866 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/28d7c937-a848-4d26-861f-29f9d2b0baf7-dbus\") pod \"global-pull-secret-syncer-7gsw5\" (UID: \"28d7c937-a848-4d26-861f-29f9d2b0baf7\") " pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:40:56.025928 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024897 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/24f69e77-f691-4a49-9cbf-8000995cd2a0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sfx4m\" (UID: \"24f69e77-f691-4a49-9cbf-8000995cd2a0\") " pod="openshift-multus/multus-additional-cni-plugins-sfx4m" Jul 22 12:40:56.025928 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024923 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-host-run-netns\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.025928 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024970 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-host-cni-bin\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.025928 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.024993 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-host-var-lib-cni-bin\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.025928 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025016 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-host-run-netns\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.025928 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025040 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24f69e77-f691-4a49-9cbf-8000995cd2a0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sfx4m\" (UID: \"24f69e77-f691-4a49-9cbf-8000995cd2a0\") " pod="openshift-multus/multus-additional-cni-plugins-sfx4m" Jul 22 12:40:56.025928 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025069 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-etc-modprobe-d\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.025928 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025092 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d5897b36-633d-4ccc-b69f-7b50e656545c-etc-tuned\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.025928 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025113 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-system-cni-dir\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.025928 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025135 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-multus-conf-dir\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.025928 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025163 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24f69e77-f691-4a49-9cbf-8000995cd2a0-system-cni-dir\") pod \"multus-additional-cni-plugins-sfx4m\" (UID: \"24f69e77-f691-4a49-9cbf-8000995cd2a0\") " pod="openshift-multus/multus-additional-cni-plugins-sfx4m" Jul 22 12:40:56.025928 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025188 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-var-lib-kubelet\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.026661 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025243 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-etc-openvswitch\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.026661 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025265 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-node-log\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.026661 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025290 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b28e0ca4-8512-45da-91ed-2cde1491b2e2-ovn-node-metrics-cert\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.026661 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025313 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/28d7c937-a848-4d26-861f-29f9d2b0baf7-kubelet-config\") pod \"global-pull-secret-syncer-7gsw5\" (UID: \"28d7c937-a848-4d26-861f-29f9d2b0baf7\") " pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:40:56.026661 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025343 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-host-var-lib-cni-multus\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.026661 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025366 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfxnd\" (UniqueName: \"kubernetes.io/projected/2fee5353-19d0-4530-baff-6831516314d3-kube-api-access-dfxnd\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.026661 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025390 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j8hv\" (UniqueName: \"kubernetes.io/projected/55113322-3d54-4862-a5da-17d438ad52a7-kube-api-access-6j8hv\") pod \"aws-ebs-csi-driver-node-82v8w\" (UID: \"55113322-3d54-4862-a5da-17d438ad52a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" Jul 22 12:40:56.026661 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025415 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-etc-sysconfig\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.026661 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025446 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-run-ovn\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.026661 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025466 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-os-release\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.026661 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025488 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/55113322-3d54-4862-a5da-17d438ad52a7-socket-dir\") pod \"aws-ebs-csi-driver-node-82v8w\" (UID: \"55113322-3d54-4862-a5da-17d438ad52a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" Jul 22 12:40:56.026661 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025511 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-etc-sysctl-d\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.026661 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025532 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5897b36-633d-4ccc-b69f-7b50e656545c-tmp\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.026661 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025554 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qxpp\" (UniqueName: \"kubernetes.io/projected/8b80aabc-b7ef-4649-bb58-4e0a39906c4f-kube-api-access-7qxpp\") pod \"node-resolver-fgc67\" (UID: \"8b80aabc-b7ef-4649-bb58-4e0a39906c4f\") " pod="openshift-dns/node-resolver-fgc67" Jul 22 12:40:56.027328 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025591 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-host-run-ovn-kubernetes\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.027328 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025615 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b28e0ca4-8512-45da-91ed-2cde1491b2e2-env-overrides\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.027328 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025638 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c54132f-3e4d-427b-b38d-cd0daabb33d9-metrics-certs\") pod \"network-metrics-daemon-jch9g\" (UID: \"3c54132f-3e4d-427b-b38d-cd0daabb33d9\") " pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:40:56.027328 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025659 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/55113322-3d54-4862-a5da-17d438ad52a7-etc-selinux\") pod \"aws-ebs-csi-driver-node-82v8w\" (UID: \"55113322-3d54-4862-a5da-17d438ad52a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" Jul 22 12:40:56.027328 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025681 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-host-run-multus-certs\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.027328 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025703 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9qrs\" (UniqueName: \"kubernetes.io/projected/3c54132f-3e4d-427b-b38d-cd0daabb33d9-kube-api-access-x9qrs\") pod \"network-metrics-daemon-jch9g\" (UID: \"3c54132f-3e4d-427b-b38d-cd0daabb33d9\") " pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:40:56.027328 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025725 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24f69e77-f691-4a49-9cbf-8000995cd2a0-cnibin\") pod \"multus-additional-cni-plugins-sfx4m\" (UID: \"24f69e77-f691-4a49-9cbf-8000995cd2a0\") " pod="openshift-multus/multus-additional-cni-plugins-sfx4m" Jul 22 12:40:56.027328 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025745 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-etc-systemd\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.027328 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.025769 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c504a9e7-e48d-496d-8a1d-3b307361714c-host\") pod \"node-ca-w6nhq\" (UID: \"c504a9e7-e48d-496d-8a1d-3b307361714c\") " pod="openshift-image-registry/node-ca-w6nhq" Jul 22 12:40:56.126779 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.126709 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b28e0ca4-8512-45da-91ed-2cde1491b2e2-env-overrides\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.126997 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.126796 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c54132f-3e4d-427b-b38d-cd0daabb33d9-metrics-certs\") pod \"network-metrics-daemon-jch9g\" (UID: \"3c54132f-3e4d-427b-b38d-cd0daabb33d9\") " pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:40:56.126997 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.126823 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/55113322-3d54-4862-a5da-17d438ad52a7-etc-selinux\") pod \"aws-ebs-csi-driver-node-82v8w\" (UID: \"55113322-3d54-4862-a5da-17d438ad52a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" Jul 22 12:40:56.126997 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.126849 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-host-run-multus-certs\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.126997 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.126882 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9qrs\" (UniqueName: \"kubernetes.io/projected/3c54132f-3e4d-427b-b38d-cd0daabb33d9-kube-api-access-x9qrs\") pod \"network-metrics-daemon-jch9g\" (UID: \"3c54132f-3e4d-427b-b38d-cd0daabb33d9\") " pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:40:56.126997 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.126909 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24f69e77-f691-4a49-9cbf-8000995cd2a0-cnibin\") pod \"multus-additional-cni-plugins-sfx4m\" (UID: \"24f69e77-f691-4a49-9cbf-8000995cd2a0\") " pod="openshift-multus/multus-additional-cni-plugins-sfx4m" Jul 22 12:40:56.126997 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.126934 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-etc-systemd\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.126997 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.126975 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c504a9e7-e48d-496d-8a1d-3b307361714c-host\") pod \"node-ca-w6nhq\" (UID: \"c504a9e7-e48d-496d-8a1d-3b307361714c\") " pod="openshift-image-registry/node-ca-w6nhq" Jul 22 12:40:56.126997 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.126998 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28d7c937-a848-4d26-861f-29f9d2b0baf7-original-pull-secret\") pod \"global-pull-secret-syncer-7gsw5\" (UID: \"28d7c937-a848-4d26-861f-29f9d2b0baf7\") " pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:40:56.127431 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127023 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fee5353-19d0-4530-baff-6831516314d3-cni-binary-copy\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.127431 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:56.127041 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:40:56.127431 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127097 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c504a9e7-e48d-496d-8a1d-3b307361714c-host\") pod \"node-ca-w6nhq\" (UID: \"c504a9e7-e48d-496d-8a1d-3b307361714c\") " pod="openshift-image-registry/node-ca-w6nhq" Jul 22 12:40:56.127431 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127154 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-host-run-multus-certs\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.127431 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127098 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-etc-systemd\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.127431 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:56.127159 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c54132f-3e4d-427b-b38d-cd0daabb33d9-metrics-certs podName:3c54132f-3e4d-427b-b38d-cd0daabb33d9 nodeName:}" failed. No retries permitted until 2026-07-22 12:40:56.627114942 +0000 UTC m=+5.028345859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c54132f-3e4d-427b-b38d-cd0daabb33d9-metrics-certs") pod "network-metrics-daemon-jch9g" (UID: "3c54132f-3e4d-427b-b38d-cd0daabb33d9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:40:56.127431 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:56.127201 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 12:40:56.127431 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127222 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24f69e77-f691-4a49-9cbf-8000995cd2a0-cnibin\") pod \"multus-additional-cni-plugins-sfx4m\" (UID: \"24f69e77-f691-4a49-9cbf-8000995cd2a0\") " pod="openshift-multus/multus-additional-cni-plugins-sfx4m" Jul 22 12:40:56.127431 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127045 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/55113322-3d54-4862-a5da-17d438ad52a7-etc-selinux\") pod \"aws-ebs-csi-driver-node-82v8w\" (UID: \"55113322-3d54-4862-a5da-17d438ad52a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" Jul 22 12:40:56.127431 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:56.127260 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28d7c937-a848-4d26-861f-29f9d2b0baf7-original-pull-secret podName:28d7c937-a848-4d26-861f-29f9d2b0baf7 nodeName:}" failed. No retries permitted until 2026-07-22 12:40:56.627241344 +0000 UTC m=+5.028472244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28d7c937-a848-4d26-861f-29f9d2b0baf7-original-pull-secret") pod "global-pull-secret-syncer-7gsw5" (UID: "28d7c937-a848-4d26-861f-29f9d2b0baf7") : object "kube-system"/"original-pull-secret" not registered Jul 22 12:40:56.127431 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127047 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-multus-socket-dir-parent\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.127431 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127378 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-multus-socket-dir-parent\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.127431 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127392 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/55113322-3d54-4862-a5da-17d438ad52a7-device-dir\") pod \"aws-ebs-csi-driver-node-82v8w\" (UID: \"55113322-3d54-4862-a5da-17d438ad52a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" Jul 22 12:40:56.127431 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127396 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b28e0ca4-8512-45da-91ed-2cde1491b2e2-env-overrides\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.128214 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127432 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-sys\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.128214 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127458 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-host-cni-netd\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.128214 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127489 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.128214 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127493 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/55113322-3d54-4862-a5da-17d438ad52a7-device-dir\") pod \"aws-ebs-csi-driver-node-82v8w\" (UID: \"55113322-3d54-4862-a5da-17d438ad52a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" Jul 22 12:40:56.128214 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127517 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b28e0ca4-8512-45da-91ed-2cde1491b2e2-ovnkube-config\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.128214 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127537 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-host-cni-netd\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.128214 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127543 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3dc0c773-526a-4eee-8155-77cf64929958-konnectivity-ca\") pod \"konnectivity-agent-ns7kz\" (UID: \"3dc0c773-526a-4eee-8155-77cf64929958\") " pod="kube-system/konnectivity-agent-ns7kz" Jul 22 12:40:56.128214 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127571 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-etc-kubernetes\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.128214 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127586 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-sys\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.128214 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127598 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24f69e77-f691-4a49-9cbf-8000995cd2a0-cni-binary-copy\") pod \"multus-additional-cni-plugins-sfx4m\" (UID: \"24f69e77-f691-4a49-9cbf-8000995cd2a0\") " pod="openshift-multus/multus-additional-cni-plugins-sfx4m" Jul 22 12:40:56.128214 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127660 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-etc-kubernetes\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.128214 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127694 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jks8\" (UniqueName: \"kubernetes.io/projected/24f69e77-f691-4a49-9cbf-8000995cd2a0-kube-api-access-4jks8\") pod \"multus-additional-cni-plugins-sfx4m\" (UID: \"24f69e77-f691-4a49-9cbf-8000995cd2a0\") " pod="openshift-multus/multus-additional-cni-plugins-sfx4m" Jul 22 12:40:56.128214 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127724 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-lib-modules\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.128214 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127775 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c504a9e7-e48d-496d-8a1d-3b307361714c-serviceca\") pod \"node-ca-w6nhq\" (UID: \"c504a9e7-e48d-496d-8a1d-3b307361714c\") " pod="openshift-image-registry/node-ca-w6nhq" Jul 22 12:40:56.128214 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127810 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2fee5353-19d0-4530-baff-6831516314d3-cni-binary-copy\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.128819 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127801 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-host-kubelet\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.128819 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127871 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-multus-cni-dir\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.128819 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.127951 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-lib-modules\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.128819 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128038 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/24f69e77-f691-4a49-9cbf-8000995cd2a0-os-release\") pod \"multus-additional-cni-plugins-sfx4m\" (UID: \"24f69e77-f691-4a49-9cbf-8000995cd2a0\") " pod="openshift-multus/multus-additional-cni-plugins-sfx4m" Jul 22 12:40:56.128819 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128067 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-host-run-k8s-cni-cncf-io\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.128819 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128097 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfhmz\" (UniqueName: \"kubernetes.io/projected/d5897b36-633d-4ccc-b69f-7b50e656545c-kube-api-access-bfhmz\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.128819 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128118 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq4zw\" (UniqueName: \"kubernetes.io/projected/c504a9e7-e48d-496d-8a1d-3b307361714c-kube-api-access-lq4zw\") pod \"node-ca-w6nhq\" (UID: \"c504a9e7-e48d-496d-8a1d-3b307361714c\") " pod="openshift-image-registry/node-ca-w6nhq" Jul 22 12:40:56.128819 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128142 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-host-var-lib-kubelet\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.128819 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128154 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b28e0ca4-8512-45da-91ed-2cde1491b2e2-ovnkube-config\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.128819 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128166 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-host\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.128819 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128215 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3dc0c773-526a-4eee-8155-77cf64929958-agent-certs\") pod \"konnectivity-agent-ns7kz\" (UID: \"3dc0c773-526a-4eee-8155-77cf64929958\") " pod="kube-system/konnectivity-agent-ns7kz" Jul 22 12:40:56.128819 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128219 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3dc0c773-526a-4eee-8155-77cf64929958-konnectivity-ca\") pod \"konnectivity-agent-ns7kz\" (UID: \"3dc0c773-526a-4eee-8155-77cf64929958\") " pod="kube-system/konnectivity-agent-ns7kz" Jul 22 12:40:56.128819 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128248 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/55113322-3d54-4862-a5da-17d438ad52a7-registration-dir\") pod \"aws-ebs-csi-driver-node-82v8w\" (UID: \"55113322-3d54-4862-a5da-17d438ad52a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" Jul 22 12:40:56.128819 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128251 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-host-kubelet\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.128819 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128305 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-host-run-k8s-cni-cncf-io\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.129574 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128315 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-host\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.129574 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128317 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/24f69e77-f691-4a49-9cbf-8000995cd2a0-os-release\") pod \"multus-additional-cni-plugins-sfx4m\" (UID: \"24f69e77-f691-4a49-9cbf-8000995cd2a0\") " pod="openshift-multus/multus-additional-cni-plugins-sfx4m" Jul 22 12:40:56.129574 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128214 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.129574 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128454 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-multus-cni-dir\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.129574 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128670 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 22 12:40:56.129574 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128741 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c504a9e7-e48d-496d-8a1d-3b307361714c-serviceca\") pod \"node-ca-w6nhq\" (UID: \"c504a9e7-e48d-496d-8a1d-3b307361714c\") " pod="openshift-image-registry/node-ca-w6nhq" Jul 22 12:40:56.129574 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128774 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24f69e77-f691-4a49-9cbf-8000995cd2a0-cni-binary-copy\") pod \"multus-additional-cni-plugins-sfx4m\" (UID: \"24f69e77-f691-4a49-9cbf-8000995cd2a0\") " pod="openshift-multus/multus-additional-cni-plugins-sfx4m" Jul 22 12:40:56.129574 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128781 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-etc-sysctl-conf\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.129574 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-var-lib-openvswitch\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.129574 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128840 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/55113322-3d54-4862-a5da-17d438ad52a7-registration-dir\") pod \"aws-ebs-csi-driver-node-82v8w\" (UID: \"55113322-3d54-4862-a5da-17d438ad52a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" Jul 22 12:40:56.129574 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128840 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-host-var-lib-kubelet\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.129574 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128855 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-cnibin\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.129574 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128893 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-var-lib-openvswitch\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.129574 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxmtx\" (UniqueName: \"kubernetes.io/projected/9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed-kube-api-access-hxmtx\") pod \"network-check-target-6ghfv\" (UID: \"9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed\") " pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:40:56.129574 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128925 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/55113322-3d54-4862-a5da-17d438ad52a7-sys-fs\") pod \"aws-ebs-csi-driver-node-82v8w\" (UID: \"55113322-3d54-4862-a5da-17d438ad52a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" Jul 22 12:40:56.130424 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128952 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-etc-kubernetes\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.130424 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128979 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-run\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.130424 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128994 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-cnibin\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.130424 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129025 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b80aabc-b7ef-4649-bb58-4e0a39906c4f-tmp-dir\") pod \"node-resolver-fgc67\" (UID: \"8b80aabc-b7ef-4649-bb58-4e0a39906c4f\") " pod="openshift-dns/node-resolver-fgc67" Jul 22 12:40:56.130424 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129051 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-systemd-units\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.130424 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129066 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/55113322-3d54-4862-a5da-17d438ad52a7-sys-fs\") pod \"aws-ebs-csi-driver-node-82v8w\" (UID: \"55113322-3d54-4862-a5da-17d438ad52a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" Jul 22 12:40:56.130424 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129061 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-etc-kubernetes\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.130424 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129077 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2fee5353-19d0-4530-baff-6831516314d3-multus-daemon-config\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.130424 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129110 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-run\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.130424 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.128953 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-etc-sysctl-conf\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.130424 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129134 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-systemd-units\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.130424 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/55113322-3d54-4862-a5da-17d438ad52a7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-82v8w\" (UID: \"55113322-3d54-4862-a5da-17d438ad52a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" Jul 22 12:40:56.130424 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129197 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/24f69e77-f691-4a49-9cbf-8000995cd2a0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sfx4m\" (UID: \"24f69e77-f691-4a49-9cbf-8000995cd2a0\") " pod="openshift-multus/multus-additional-cni-plugins-sfx4m" Jul 22 12:40:56.130424 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129224 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b80aabc-b7ef-4649-bb58-4e0a39906c4f-hosts-file\") pod \"node-resolver-fgc67\" (UID: \"8b80aabc-b7ef-4649-bb58-4e0a39906c4f\") " pod="openshift-dns/node-resolver-fgc67" Jul 22 12:40:56.130424 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129249 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-host-slash\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.131211 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129258 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/55113322-3d54-4862-a5da-17d438ad52a7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-82v8w\" (UID: \"55113322-3d54-4862-a5da-17d438ad52a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" Jul 22 12:40:56.131211 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129273 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-run-systemd\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.131211 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129298 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-run-openvswitch\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.131211 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129312 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8b80aabc-b7ef-4649-bb58-4e0a39906c4f-hosts-file\") pod \"node-resolver-fgc67\" (UID: \"8b80aabc-b7ef-4649-bb58-4e0a39906c4f\") " pod="openshift-dns/node-resolver-fgc67" Jul 22 12:40:56.131211 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129324 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6hrz\" (UniqueName: \"kubernetes.io/projected/b28e0ca4-8512-45da-91ed-2cde1491b2e2-kube-api-access-t6hrz\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.131211 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129348 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f3b3f37-8156-495b-b73b-011f8decc3fa-host-slash\") pod \"iptables-alerter-5c5j7\" (UID: \"6f3b3f37-8156-495b-b73b-011f8decc3fa\") " pod="openshift-network-operator/iptables-alerter-5c5j7" Jul 22 12:40:56.131211 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129358 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b80aabc-b7ef-4649-bb58-4e0a39906c4f-tmp-dir\") pod \"node-resolver-fgc67\" (UID: \"8b80aabc-b7ef-4649-bb58-4e0a39906c4f\") " pod="openshift-dns/node-resolver-fgc67" Jul 22 12:40:56.131211 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129372 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-log-socket\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.131211 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129400 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b28e0ca4-8512-45da-91ed-2cde1491b2e2-ovnkube-script-lib\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.131211 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129424 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-hostroot\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.131211 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129450 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6f3b3f37-8156-495b-b73b-011f8decc3fa-iptables-alerter-script\") pod \"iptables-alerter-5c5j7\" (UID: \"6f3b3f37-8156-495b-b73b-011f8decc3fa\") " pod="openshift-network-operator/iptables-alerter-5c5j7" Jul 22 12:40:56.131211 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129477 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktmv9\" (UniqueName: \"kubernetes.io/projected/6f3b3f37-8156-495b-b73b-011f8decc3fa-kube-api-access-ktmv9\") pod \"iptables-alerter-5c5j7\" (UID: \"6f3b3f37-8156-495b-b73b-011f8decc3fa\") " pod="openshift-network-operator/iptables-alerter-5c5j7" Jul 22 12:40:56.131211 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129499 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-host-slash\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.131211 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129505 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/28d7c937-a848-4d26-861f-29f9d2b0baf7-dbus\") pod \"global-pull-secret-syncer-7gsw5\" (UID: \"28d7c937-a848-4d26-861f-29f9d2b0baf7\") " pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:40:56.132300 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129543 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/24f69e77-f691-4a49-9cbf-8000995cd2a0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sfx4m\" (UID: \"24f69e77-f691-4a49-9cbf-8000995cd2a0\") " pod="openshift-multus/multus-additional-cni-plugins-sfx4m" Jul 22 12:40:56.132300 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129565 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/28d7c937-a848-4d26-861f-29f9d2b0baf7-dbus\") pod \"global-pull-secret-syncer-7gsw5\" (UID: \"28d7c937-a848-4d26-861f-29f9d2b0baf7\") " pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:40:56.132300 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129571 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2fee5353-19d0-4530-baff-6831516314d3-multus-daemon-config\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.132300 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129588 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-host-run-netns\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.132300 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129614 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-host-cni-bin\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.132300 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129629 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-run-systemd\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.132300 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129639 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-host-var-lib-cni-bin\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.132300 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129681 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-host-var-lib-cni-bin\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.132300 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129676 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-hostroot\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.132300 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129702 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-host-run-netns\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.132300 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129730 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24f69e77-f691-4a49-9cbf-8000995cd2a0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sfx4m\" (UID: \"24f69e77-f691-4a49-9cbf-8000995cd2a0\") " pod="openshift-multus/multus-additional-cni-plugins-sfx4m" Jul 22 12:40:56.132300 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-etc-modprobe-d\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.132300 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129452 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-run-openvswitch\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.132300 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129798 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/24f69e77-f691-4a49-9cbf-8000995cd2a0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sfx4m\" (UID: \"24f69e77-f691-4a49-9cbf-8000995cd2a0\") " pod="openshift-multus/multus-additional-cni-plugins-sfx4m" Jul 22 12:40:56.132300 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129799 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d5897b36-633d-4ccc-b69f-7b50e656545c-etc-tuned\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.133202 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129855 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-system-cni-dir\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.133202 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129859 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-host-run-netns\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.133202 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129882 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-multus-conf-dir\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.133202 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129906 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f3b3f37-8156-495b-b73b-011f8decc3fa-host-slash\") pod \"iptables-alerter-5c5j7\" (UID: \"6f3b3f37-8156-495b-b73b-011f8decc3fa\") " pod="openshift-network-operator/iptables-alerter-5c5j7" Jul 22 12:40:56.133202 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129909 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24f69e77-f691-4a49-9cbf-8000995cd2a0-system-cni-dir\") pod \"multus-additional-cni-plugins-sfx4m\" (UID: \"24f69e77-f691-4a49-9cbf-8000995cd2a0\") " pod="openshift-multus/multus-additional-cni-plugins-sfx4m" Jul 22 12:40:56.133202 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129953 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-log-socket\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.133202 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.129960 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-var-lib-kubelet\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.133202 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130013 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-etc-openvswitch\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.133202 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130047 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-node-log\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.133202 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130093 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b28e0ca4-8512-45da-91ed-2cde1491b2e2-ovn-node-metrics-cert\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.133202 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130120 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/28d7c937-a848-4d26-861f-29f9d2b0baf7-kubelet-config\") pod \"global-pull-secret-syncer-7gsw5\" (UID: \"28d7c937-a848-4d26-861f-29f9d2b0baf7\") " pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:40:56.133202 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130137 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24f69e77-f691-4a49-9cbf-8000995cd2a0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sfx4m\" (UID: \"24f69e77-f691-4a49-9cbf-8000995cd2a0\") " pod="openshift-multus/multus-additional-cni-plugins-sfx4m" Jul 22 12:40:56.133202 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130159 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-host-var-lib-cni-multus\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.133202 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130184 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/24f69e77-f691-4a49-9cbf-8000995cd2a0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sfx4m\" (UID: \"24f69e77-f691-4a49-9cbf-8000995cd2a0\") " pod="openshift-multus/multus-additional-cni-plugins-sfx4m" Jul 22 12:40:56.134250 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130212 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-host-var-lib-cni-multus\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.134250 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130234 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-host-run-netns\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.134250 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130257 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-etc-modprobe-d\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.134250 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130273 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-system-cni-dir\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.134250 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130276 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-host-cni-bin\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.134250 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130295 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfxnd\" (UniqueName: \"kubernetes.io/projected/2fee5353-19d0-4530-baff-6831516314d3-kube-api-access-dfxnd\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.134250 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130321 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-multus-conf-dir\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.134250 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130326 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j8hv\" (UniqueName: \"kubernetes.io/projected/55113322-3d54-4862-a5da-17d438ad52a7-kube-api-access-6j8hv\") pod \"aws-ebs-csi-driver-node-82v8w\" (UID: \"55113322-3d54-4862-a5da-17d438ad52a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" Jul 22 12:40:56.134250 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130356 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-etc-sysconfig\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.134250 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130359 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24f69e77-f691-4a49-9cbf-8000995cd2a0-system-cni-dir\") pod \"multus-additional-cni-plugins-sfx4m\" (UID: \"24f69e77-f691-4a49-9cbf-8000995cd2a0\") " pod="openshift-multus/multus-additional-cni-plugins-sfx4m" Jul 22 12:40:56.134250 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130367 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-node-log\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.134250 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130383 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-run-ovn\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.134250 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130409 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-os-release\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.134250 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130439 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/55113322-3d54-4862-a5da-17d438ad52a7-socket-dir\") pod \"aws-ebs-csi-driver-node-82v8w\" (UID: \"55113322-3d54-4862-a5da-17d438ad52a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" Jul 22 12:40:56.134250 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130452 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b28e0ca4-8512-45da-91ed-2cde1491b2e2-ovnkube-script-lib\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.135084 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130464 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-etc-sysctl-d\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.135084 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130488 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5897b36-633d-4ccc-b69f-7b50e656545c-tmp\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.135084 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130516 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qxpp\" (UniqueName: \"kubernetes.io/projected/8b80aabc-b7ef-4649-bb58-4e0a39906c4f-kube-api-access-7qxpp\") pod \"node-resolver-fgc67\" (UID: \"8b80aabc-b7ef-4649-bb58-4e0a39906c4f\") " pod="openshift-dns/node-resolver-fgc67" Jul 22 12:40:56.135084 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130544 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-host-run-ovn-kubernetes\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.135084 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130618 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-host-run-ovn-kubernetes\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.135084 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130666 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-run-ovn\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.135084 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130731 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2fee5353-19d0-4530-baff-6831516314d3-os-release\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.135084 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130412 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-var-lib-kubelet\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.135084 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130860 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/55113322-3d54-4862-a5da-17d438ad52a7-socket-dir\") pod \"aws-ebs-csi-driver-node-82v8w\" (UID: \"55113322-3d54-4862-a5da-17d438ad52a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" Jul 22 12:40:56.135084 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130966 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-etc-sysctl-d\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.135084 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130982 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6f3b3f37-8156-495b-b73b-011f8decc3fa-iptables-alerter-script\") pod \"iptables-alerter-5c5j7\" (UID: \"6f3b3f37-8156-495b-b73b-011f8decc3fa\") " pod="openshift-network-operator/iptables-alerter-5c5j7" Jul 22 12:40:56.135084 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130519 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d5897b36-633d-4ccc-b69f-7b50e656545c-etc-sysconfig\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.135084 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.130324 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b28e0ca4-8512-45da-91ed-2cde1491b2e2-etc-openvswitch\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.135084 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.131300 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/28d7c937-a848-4d26-861f-29f9d2b0baf7-kubelet-config\") pod \"global-pull-secret-syncer-7gsw5\" (UID: \"28d7c937-a848-4d26-861f-29f9d2b0baf7\") " pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:40:56.135084 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.133473 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d5897b36-633d-4ccc-b69f-7b50e656545c-etc-tuned\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.135761 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.133573 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b28e0ca4-8512-45da-91ed-2cde1491b2e2-ovn-node-metrics-cert\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.135761 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.133637 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5897b36-633d-4ccc-b69f-7b50e656545c-tmp\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.135761 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.134461 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3dc0c773-526a-4eee-8155-77cf64929958-agent-certs\") pod \"konnectivity-agent-ns7kz\" (UID: \"3dc0c773-526a-4eee-8155-77cf64929958\") " pod="kube-system/konnectivity-agent-ns7kz" Jul 22 12:40:56.138008 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:56.137869 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 12:40:56.138008 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:56.137891 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 12:40:56.138008 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:56.137906 2579 projected.go:196] Error preparing data for projected volume kube-api-access-hxmtx for pod openshift-network-diagnostics/network-check-target-6ghfv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:40:56.138008 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:56.137974 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed-kube-api-access-hxmtx podName:9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed nodeName:}" failed. No retries permitted until 2026-07-22 12:40:56.637941903 +0000 UTC m=+5.039172821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hxmtx" (UniqueName: "kubernetes.io/projected/9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed-kube-api-access-hxmtx") pod "network-check-target-6ghfv" (UID: "9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:40:56.140769 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.140712 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9qrs\" (UniqueName: \"kubernetes.io/projected/3c54132f-3e4d-427b-b38d-cd0daabb33d9-kube-api-access-x9qrs\") pod \"network-metrics-daemon-jch9g\" (UID: \"3c54132f-3e4d-427b-b38d-cd0daabb33d9\") " pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:40:56.141109 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.141087 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jks8\" (UniqueName: \"kubernetes.io/projected/24f69e77-f691-4a49-9cbf-8000995cd2a0-kube-api-access-4jks8\") pod \"multus-additional-cni-plugins-sfx4m\" (UID: \"24f69e77-f691-4a49-9cbf-8000995cd2a0\") " pod="openshift-multus/multus-additional-cni-plugins-sfx4m" Jul 22 12:40:56.143229 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.143205 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq4zw\" (UniqueName: \"kubernetes.io/projected/c504a9e7-e48d-496d-8a1d-3b307361714c-kube-api-access-lq4zw\") pod \"node-ca-w6nhq\" (UID: \"c504a9e7-e48d-496d-8a1d-3b307361714c\") " pod="openshift-image-registry/node-ca-w6nhq" Jul 22 12:40:56.147212 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.146731 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktmv9\" (UniqueName: \"kubernetes.io/projected/6f3b3f37-8156-495b-b73b-011f8decc3fa-kube-api-access-ktmv9\") pod \"iptables-alerter-5c5j7\" (UID: \"6f3b3f37-8156-495b-b73b-011f8decc3fa\") " pod="openshift-network-operator/iptables-alerter-5c5j7" Jul 22 12:40:56.149437 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.149353 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfhmz\" (UniqueName: \"kubernetes.io/projected/d5897b36-633d-4ccc-b69f-7b50e656545c-kube-api-access-bfhmz\") pod \"tuned-wgxgn\" (UID: \"d5897b36-633d-4ccc-b69f-7b50e656545c\") " pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.149990 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.149930 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfxnd\" (UniqueName: \"kubernetes.io/projected/2fee5353-19d0-4530-baff-6831516314d3-kube-api-access-dfxnd\") pod \"multus-pgrh7\" (UID: \"2fee5353-19d0-4530-baff-6831516314d3\") " pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.149990 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.149967 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qxpp\" (UniqueName: \"kubernetes.io/projected/8b80aabc-b7ef-4649-bb58-4e0a39906c4f-kube-api-access-7qxpp\") pod \"node-resolver-fgc67\" (UID: \"8b80aabc-b7ef-4649-bb58-4e0a39906c4f\") " pod="openshift-dns/node-resolver-fgc67" Jul 22 12:40:56.151723 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.151676 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6hrz\" (UniqueName: \"kubernetes.io/projected/b28e0ca4-8512-45da-91ed-2cde1491b2e2-kube-api-access-t6hrz\") pod \"ovnkube-node-8ddj6\" (UID: \"b28e0ca4-8512-45da-91ed-2cde1491b2e2\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.154088 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.154044 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j8hv\" (UniqueName: \"kubernetes.io/projected/55113322-3d54-4862-a5da-17d438ad52a7-kube-api-access-6j8hv\") pod \"aws-ebs-csi-driver-node-82v8w\" (UID: \"55113322-3d54-4862-a5da-17d438ad52a7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" Jul 22 12:40:56.234350 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.234281 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w6nhq" Jul 22 12:40:56.247402 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.247376 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:40:56.261329 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.261308 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ns7kz" Jul 22 12:40:56.267253 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.267234 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" Jul 22 12:40:56.274124 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.274101 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sfx4m" Jul 22 12:40:56.282117 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.282097 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" Jul 22 12:40:56.290424 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.290406 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pgrh7" Jul 22 12:40:56.300257 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.300238 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5c5j7" Jul 22 12:40:56.308028 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.308009 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fgc67" Jul 22 12:40:56.634593 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.634489 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c54132f-3e4d-427b-b38d-cd0daabb33d9-metrics-certs\") pod \"network-metrics-daemon-jch9g\" (UID: \"3c54132f-3e4d-427b-b38d-cd0daabb33d9\") " pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:40:56.634593 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.634544 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28d7c937-a848-4d26-861f-29f9d2b0baf7-original-pull-secret\") pod \"global-pull-secret-syncer-7gsw5\" (UID: \"28d7c937-a848-4d26-861f-29f9d2b0baf7\") " pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:40:56.634823 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:56.634654 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 12:40:56.634823 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:56.634665 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:40:56.634823 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:56.634732 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28d7c937-a848-4d26-861f-29f9d2b0baf7-original-pull-secret podName:28d7c937-a848-4d26-861f-29f9d2b0baf7 nodeName:}" failed. No retries permitted until 2026-07-22 12:40:57.63471002 +0000 UTC m=+6.035940945 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28d7c937-a848-4d26-861f-29f9d2b0baf7-original-pull-secret") pod "global-pull-secret-syncer-7gsw5" (UID: "28d7c937-a848-4d26-861f-29f9d2b0baf7") : object "kube-system"/"original-pull-secret" not registered Jul 22 12:40:56.634823 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:56.634752 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c54132f-3e4d-427b-b38d-cd0daabb33d9-metrics-certs podName:3c54132f-3e4d-427b-b38d-cd0daabb33d9 nodeName:}" failed. No retries permitted until 2026-07-22 12:40:57.63474336 +0000 UTC m=+6.035974260 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c54132f-3e4d-427b-b38d-cd0daabb33d9-metrics-certs") pod "network-metrics-daemon-jch9g" (UID: "3c54132f-3e4d-427b-b38d-cd0daabb33d9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:40:56.735592 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.735552 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxmtx\" (UniqueName: \"kubernetes.io/projected/9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed-kube-api-access-hxmtx\") pod \"network-check-target-6ghfv\" (UID: \"9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed\") " pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:40:56.735785 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:56.735747 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 12:40:56.735785 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:56.735776 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 12:40:56.735902 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:56.735790 2579 projected.go:196] Error preparing data for projected volume kube-api-access-hxmtx for pod openshift-network-diagnostics/network-check-target-6ghfv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:40:56.735902 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:56.735864 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed-kube-api-access-hxmtx podName:9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed nodeName:}" failed. No retries permitted until 2026-07-22 12:40:57.735841593 +0000 UTC m=+6.137072511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxmtx" (UniqueName: "kubernetes.io/projected/9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed-kube-api-access-hxmtx") pod "network-check-target-6ghfv" (UID: "9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:40:56.962207 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.962129 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 12:35:56 +0000 UTC" deadline="2028-04-13 06:11:25.692828362 +0000 UTC" Jul 22 12:40:56.962207 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:56.962160 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15137h30m28.730672018s" Jul 22 12:40:57.061209 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:57.061173 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:40:57.061209 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:57.061211 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:40:57.061444 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:57.061319 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ghfv" podUID="9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed" Jul 22 12:40:57.061503 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:57.061447 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gsw5" podUID="28d7c937-a848-4d26-861f-29f9d2b0baf7" Jul 22 12:40:57.250587 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:57.250554 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f3b3f37_8156_495b_b73b_011f8decc3fa.slice/crio-bfb16626aac9908615e78b8c831865c2be1658af7e85fe8ff6a75269f19fad27 WatchSource:0}: Error finding container bfb16626aac9908615e78b8c831865c2be1658af7e85fe8ff6a75269f19fad27: Status 404 returned error can't find the container with id bfb16626aac9908615e78b8c831865c2be1658af7e85fe8ff6a75269f19fad27 Jul 22 12:40:57.255323 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:57.255007 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55113322_3d54_4862_a5da_17d438ad52a7.slice/crio-ab2747d1046bc793931da21bf6b21d8a7f945f5b7624acec58a1788750acff29 WatchSource:0}: Error finding container ab2747d1046bc793931da21bf6b21d8a7f945f5b7624acec58a1788750acff29: Status 404 returned error can't find the container with id ab2747d1046bc793931da21bf6b21d8a7f945f5b7624acec58a1788750acff29 Jul 22 12:40:57.259774 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:57.258675 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b80aabc_b7ef_4649_bb58_4e0a39906c4f.slice/crio-67d5a48e9e885d2924f83a31aeef20257903f5d065fdb4c8128ec0dbbeffab37 WatchSource:0}: Error finding container 67d5a48e9e885d2924f83a31aeef20257903f5d065fdb4c8128ec0dbbeffab37: Status 404 returned error can't find the container with id 67d5a48e9e885d2924f83a31aeef20257903f5d065fdb4c8128ec0dbbeffab37 Jul 22 12:40:57.259980 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:57.259890 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc504a9e7_e48d_496d_8a1d_3b307361714c.slice/crio-3f00cd69dff0f643f9bbd8cb526e838be63b5b9190a7989108225b6ba8b9eca8 WatchSource:0}: Error finding container 3f00cd69dff0f643f9bbd8cb526e838be63b5b9190a7989108225b6ba8b9eca8: Status 404 returned error can't find the container with id 3f00cd69dff0f643f9bbd8cb526e838be63b5b9190a7989108225b6ba8b9eca8 Jul 22 12:40:57.261229 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:57.261147 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f69e77_f691_4a49_9cbf_8000995cd2a0.slice/crio-021f4a79e9235bb191f6f6bdd890df85e9dbf69b0f50a2be28abc0269e540568 WatchSource:0}: Error finding container 021f4a79e9235bb191f6f6bdd890df85e9dbf69b0f50a2be28abc0269e540568: Status 404 returned error can't find the container with id 021f4a79e9235bb191f6f6bdd890df85e9dbf69b0f50a2be28abc0269e540568 Jul 22 12:40:57.261517 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:57.261497 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb28e0ca4_8512_45da_91ed_2cde1491b2e2.slice/crio-2d06bb4ec99ab762d9ba38bcbc4f00dbc90fe64e28de9f5aabe64ad45a2f944c WatchSource:0}: Error finding container 2d06bb4ec99ab762d9ba38bcbc4f00dbc90fe64e28de9f5aabe64ad45a2f944c: Status 404 returned error can't find the container with id 2d06bb4ec99ab762d9ba38bcbc4f00dbc90fe64e28de9f5aabe64ad45a2f944c Jul 22 12:40:57.262405 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:57.262374 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fee5353_19d0_4530_baff_6831516314d3.slice/crio-86348ccab42caaf2d4d8873144ff8bb0e940d6d2a0a99d2efe243d6308b2551e WatchSource:0}: Error finding container 86348ccab42caaf2d4d8873144ff8bb0e940d6d2a0a99d2efe243d6308b2551e: Status 404 returned error can't find the container with id 86348ccab42caaf2d4d8873144ff8bb0e940d6d2a0a99d2efe243d6308b2551e Jul 22 12:40:57.265172 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:57.265128 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dc0c773_526a_4eee_8155_77cf64929958.slice/crio-e45573afc30d2a28c91eed284b471b9eb4a78a6cfc903ac0563fd7e248b0c262 WatchSource:0}: Error finding container e45573afc30d2a28c91eed284b471b9eb4a78a6cfc903ac0563fd7e248b0c262: Status 404 returned error can't find the container with id e45573afc30d2a28c91eed284b471b9eb4a78a6cfc903ac0563fd7e248b0c262 Jul 22 12:40:57.265850 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:40:57.265821 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5897b36_633d_4ccc_b69f_7b50e656545c.slice/crio-f787559109a6488adf783321d2a8123766472d1c83483b6defb053859a7cb805 WatchSource:0}: Error finding container f787559109a6488adf783321d2a8123766472d1c83483b6defb053859a7cb805: Status 404 returned error can't find the container with id f787559109a6488adf783321d2a8123766472d1c83483b6defb053859a7cb805 Jul 22 12:40:57.642945 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:57.642722 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c54132f-3e4d-427b-b38d-cd0daabb33d9-metrics-certs\") pod \"network-metrics-daemon-jch9g\" (UID: \"3c54132f-3e4d-427b-b38d-cd0daabb33d9\") " pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:40:57.642945 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:57.642872 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:40:57.643158 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:57.642947 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28d7c937-a848-4d26-861f-29f9d2b0baf7-original-pull-secret\") pod \"global-pull-secret-syncer-7gsw5\" (UID: \"28d7c937-a848-4d26-861f-29f9d2b0baf7\") " pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:40:57.643158 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:57.643007 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c54132f-3e4d-427b-b38d-cd0daabb33d9-metrics-certs podName:3c54132f-3e4d-427b-b38d-cd0daabb33d9 nodeName:}" failed. No retries permitted until 2026-07-22 12:40:59.642988223 +0000 UTC m=+8.044219123 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c54132f-3e4d-427b-b38d-cd0daabb33d9-metrics-certs") pod "network-metrics-daemon-jch9g" (UID: "3c54132f-3e4d-427b-b38d-cd0daabb33d9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:40:57.643158 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:57.643062 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 12:40:57.643158 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:57.643117 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28d7c937-a848-4d26-861f-29f9d2b0baf7-original-pull-secret podName:28d7c937-a848-4d26-861f-29f9d2b0baf7 nodeName:}" failed. No retries permitted until 2026-07-22 12:40:59.643102621 +0000 UTC m=+8.044333535 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28d7c937-a848-4d26-861f-29f9d2b0baf7-original-pull-secret") pod "global-pull-secret-syncer-7gsw5" (UID: "28d7c937-a848-4d26-861f-29f9d2b0baf7") : object "kube-system"/"original-pull-secret" not registered Jul 22 12:40:57.744380 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:57.744259 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxmtx\" (UniqueName: \"kubernetes.io/projected/9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed-kube-api-access-hxmtx\") pod \"network-check-target-6ghfv\" (UID: \"9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed\") " pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:40:57.744544 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:57.744470 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 12:40:57.744544 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:57.744494 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 12:40:57.744544 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:57.744508 2579 projected.go:196] Error preparing data for projected volume kube-api-access-hxmtx for pod openshift-network-diagnostics/network-check-target-6ghfv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:40:57.744708 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:57.744591 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed-kube-api-access-hxmtx podName:9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed nodeName:}" failed. No retries permitted until 2026-07-22 12:40:59.744571914 +0000 UTC m=+8.145802817 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxmtx" (UniqueName: "kubernetes.io/projected/9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed-kube-api-access-hxmtx") pod "network-check-target-6ghfv" (UID: "9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:40:57.933558 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:57.933509 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5c5j7" event={"ID":"6f3b3f37-8156-495b-b73b-011f8decc3fa","Type":"ContainerStarted","Data":"bfb16626aac9908615e78b8c831865c2be1658af7e85fe8ff6a75269f19fad27"} Jul 22 12:40:57.940724 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:57.940690 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" event={"ID":"d5897b36-633d-4ccc-b69f-7b50e656545c","Type":"ContainerStarted","Data":"f787559109a6488adf783321d2a8123766472d1c83483b6defb053859a7cb805"} Jul 22 12:40:57.953980 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:57.953947 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pgrh7" event={"ID":"2fee5353-19d0-4530-baff-6831516314d3","Type":"ContainerStarted","Data":"86348ccab42caaf2d4d8873144ff8bb0e940d6d2a0a99d2efe243d6308b2551e"} Jul 22 12:40:57.961400 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:57.961368 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sfx4m" event={"ID":"24f69e77-f691-4a49-9cbf-8000995cd2a0","Type":"ContainerStarted","Data":"021f4a79e9235bb191f6f6bdd890df85e9dbf69b0f50a2be28abc0269e540568"} Jul 22 12:40:57.962492 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:57.962439 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-21 12:35:56 +0000 UTC" deadline="2028-03-13 01:18:06.97386494 +0000 UTC" Jul 22 12:40:57.962492 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:57.962493 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14388h37m9.011376095s" Jul 22 12:40:57.963349 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:57.963251 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" event={"ID":"55113322-3d54-4862-a5da-17d438ad52a7","Type":"ContainerStarted","Data":"ab2747d1046bc793931da21bf6b21d8a7f945f5b7624acec58a1788750acff29"} Jul 22 12:40:57.970458 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:57.970428 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-34.ec2.internal" event={"ID":"ea81ad4ab5ce353c7d7f37c0cd1009b2","Type":"ContainerStarted","Data":"912c55df08f295d73dceda88b43553c2394b9077e2c913775a967b157e1f32f1"} Jul 22 12:40:57.975374 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:57.975346 2579 generic.go:356] "Generic (PLEG): container finished" podID="8eac028a224ef233879714f64598e3db" containerID="d57e23d3c894ff2477a4ce31f9025c498db3d63648d96a344c14199e554077a3" exitCode=0 Jul 22 12:40:57.975470 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:57.975430 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-34.ec2.internal" event={"ID":"8eac028a224ef233879714f64598e3db","Type":"ContainerDied","Data":"d57e23d3c894ff2477a4ce31f9025c498db3d63648d96a344c14199e554077a3"} Jul 22 12:40:57.981363 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:57.981339 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ns7kz" event={"ID":"3dc0c773-526a-4eee-8155-77cf64929958","Type":"ContainerStarted","Data":"e45573afc30d2a28c91eed284b471b9eb4a78a6cfc903ac0563fd7e248b0c262"} Jul 22 12:40:57.987080 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:57.987019 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" event={"ID":"b28e0ca4-8512-45da-91ed-2cde1491b2e2","Type":"ContainerStarted","Data":"2d06bb4ec99ab762d9ba38bcbc4f00dbc90fe64e28de9f5aabe64ad45a2f944c"} Jul 22 12:40:58.002153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:58.002127 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w6nhq" event={"ID":"c504a9e7-e48d-496d-8a1d-3b307361714c","Type":"ContainerStarted","Data":"3f00cd69dff0f643f9bbd8cb526e838be63b5b9190a7989108225b6ba8b9eca8"} Jul 22 12:40:58.013375 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:58.013321 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-34.ec2.internal" podStartSLOduration=3.013302909 podStartE2EDuration="3.013302909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:40:57.993943059 +0000 UTC m=+6.395173985" watchObservedRunningTime="2026-07-22 12:40:58.013302909 +0000 UTC m=+6.414533836" Jul 22 12:40:58.014735 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:58.014680 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fgc67" event={"ID":"8b80aabc-b7ef-4649-bb58-4e0a39906c4f","Type":"ContainerStarted","Data":"67d5a48e9e885d2924f83a31aeef20257903f5d065fdb4c8128ec0dbbeffab37"} Jul 22 12:40:58.063545 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:58.063512 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:40:58.063704 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:58.063683 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jch9g" podUID="3c54132f-3e4d-427b-b38d-cd0daabb33d9" Jul 22 12:40:59.036671 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:59.036630 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-34.ec2.internal" event={"ID":"8eac028a224ef233879714f64598e3db","Type":"ContainerStarted","Data":"70df217cfa514a5d787533a31facf8134827e715886e12fc642ce785a9e4a692"} Jul 22 12:40:59.061699 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:59.061109 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:40:59.061699 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:59.061229 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gsw5" podUID="28d7c937-a848-4d26-861f-29f9d2b0baf7" Jul 22 12:40:59.061699 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:59.061570 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:40:59.061699 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:59.061651 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ghfv" podUID="9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed" Jul 22 12:40:59.661381 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:59.661337 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c54132f-3e4d-427b-b38d-cd0daabb33d9-metrics-certs\") pod \"network-metrics-daemon-jch9g\" (UID: \"3c54132f-3e4d-427b-b38d-cd0daabb33d9\") " pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:40:59.661579 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:59.661398 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28d7c937-a848-4d26-861f-29f9d2b0baf7-original-pull-secret\") pod \"global-pull-secret-syncer-7gsw5\" (UID: \"28d7c937-a848-4d26-861f-29f9d2b0baf7\") " pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:40:59.661579 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:59.661522 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 12:40:59.661703 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:59.661585 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28d7c937-a848-4d26-861f-29f9d2b0baf7-original-pull-secret podName:28d7c937-a848-4d26-861f-29f9d2b0baf7 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:03.661565922 +0000 UTC m=+12.062796826 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28d7c937-a848-4d26-861f-29f9d2b0baf7-original-pull-secret") pod "global-pull-secret-syncer-7gsw5" (UID: "28d7c937-a848-4d26-861f-29f9d2b0baf7") : object "kube-system"/"original-pull-secret" not registered Jul 22 12:40:59.662060 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:59.662041 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:40:59.662161 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:59.662095 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c54132f-3e4d-427b-b38d-cd0daabb33d9-metrics-certs podName:3c54132f-3e4d-427b-b38d-cd0daabb33d9 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:03.6620794 +0000 UTC m=+12.063310304 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c54132f-3e4d-427b-b38d-cd0daabb33d9-metrics-certs") pod "network-metrics-daemon-jch9g" (UID: "3c54132f-3e4d-427b-b38d-cd0daabb33d9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:40:59.762393 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:40:59.762350 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxmtx\" (UniqueName: \"kubernetes.io/projected/9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed-kube-api-access-hxmtx\") pod \"network-check-target-6ghfv\" (UID: \"9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed\") " pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:40:59.762630 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:59.762594 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 12:40:59.762630 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:59.762617 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 12:40:59.762630 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:59.762630 2579 projected.go:196] Error preparing data for projected volume kube-api-access-hxmtx for pod openshift-network-diagnostics/network-check-target-6ghfv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:40:59.762837 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:40:59.762691 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed-kube-api-access-hxmtx podName:9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed nodeName:}" failed. No retries permitted until 2026-07-22 12:41:03.762671269 +0000 UTC m=+12.163902184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxmtx" (UniqueName: "kubernetes.io/projected/9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed-kube-api-access-hxmtx") pod "network-check-target-6ghfv" (UID: "9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:41:00.062231 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:00.062152 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:41:00.062710 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:00.062300 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jch9g" podUID="3c54132f-3e4d-427b-b38d-cd0daabb33d9" Jul 22 12:41:01.061496 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:01.061460 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:41:01.061496 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:01.061484 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:41:01.061784 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:01.061608 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ghfv" podUID="9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed" Jul 22 12:41:01.061784 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:01.061715 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gsw5" podUID="28d7c937-a848-4d26-861f-29f9d2b0baf7" Jul 22 12:41:02.063049 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:02.063013 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:41:02.063538 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:02.063148 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jch9g" podUID="3c54132f-3e4d-427b-b38d-cd0daabb33d9" Jul 22 12:41:03.061858 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:03.061826 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:41:03.062067 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:03.061830 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:41:03.062067 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:03.061974 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ghfv" podUID="9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed" Jul 22 12:41:03.062067 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:03.062008 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gsw5" podUID="28d7c937-a848-4d26-861f-29f9d2b0baf7" Jul 22 12:41:03.695944 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:03.695405 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c54132f-3e4d-427b-b38d-cd0daabb33d9-metrics-certs\") pod \"network-metrics-daemon-jch9g\" (UID: \"3c54132f-3e4d-427b-b38d-cd0daabb33d9\") " pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:41:03.695944 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:03.695447 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28d7c937-a848-4d26-861f-29f9d2b0baf7-original-pull-secret\") pod \"global-pull-secret-syncer-7gsw5\" (UID: \"28d7c937-a848-4d26-861f-29f9d2b0baf7\") " pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:41:03.695944 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:03.695562 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 12:41:03.695944 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:03.695622 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28d7c937-a848-4d26-861f-29f9d2b0baf7-original-pull-secret podName:28d7c937-a848-4d26-861f-29f9d2b0baf7 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:11.695605245 +0000 UTC m=+20.096836158 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28d7c937-a848-4d26-861f-29f9d2b0baf7-original-pull-secret") pod "global-pull-secret-syncer-7gsw5" (UID: "28d7c937-a848-4d26-861f-29f9d2b0baf7") : object "kube-system"/"original-pull-secret" not registered Jul 22 12:41:03.695944 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:03.695899 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:41:03.696489 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:03.696075 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c54132f-3e4d-427b-b38d-cd0daabb33d9-metrics-certs podName:3c54132f-3e4d-427b-b38d-cd0daabb33d9 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:11.696053925 +0000 UTC m=+20.097284825 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c54132f-3e4d-427b-b38d-cd0daabb33d9-metrics-certs") pod "network-metrics-daemon-jch9g" (UID: "3c54132f-3e4d-427b-b38d-cd0daabb33d9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:41:03.796066 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:03.796025 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxmtx\" (UniqueName: \"kubernetes.io/projected/9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed-kube-api-access-hxmtx\") pod \"network-check-target-6ghfv\" (UID: \"9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed\") " pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:41:03.796239 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:03.796215 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 12:41:03.796239 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:03.796237 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 12:41:03.796340 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:03.796251 2579 projected.go:196] Error preparing data for projected volume kube-api-access-hxmtx for pod openshift-network-diagnostics/network-check-target-6ghfv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:41:03.796340 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:03.796311 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed-kube-api-access-hxmtx podName:9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed nodeName:}" failed. No retries permitted until 2026-07-22 12:41:11.796292104 +0000 UTC m=+20.197523020 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxmtx" (UniqueName: "kubernetes.io/projected/9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed-kube-api-access-hxmtx") pod "network-check-target-6ghfv" (UID: "9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:41:04.061888 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:04.061835 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:41:04.062087 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:04.062003 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jch9g" podUID="3c54132f-3e4d-427b-b38d-cd0daabb33d9" Jul 22 12:41:05.062205 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:05.062168 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:41:05.062607 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:05.062291 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ghfv" podUID="9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed" Jul 22 12:41:05.062607 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:05.062359 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:41:05.062607 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:05.062440 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gsw5" podUID="28d7c937-a848-4d26-861f-29f9d2b0baf7" Jul 22 12:41:06.061990 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:06.061960 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:41:06.062168 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:06.062112 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jch9g" podUID="3c54132f-3e4d-427b-b38d-cd0daabb33d9" Jul 22 12:41:07.061418 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:07.061347 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:41:07.061861 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:07.061482 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ghfv" podUID="9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed" Jul 22 12:41:07.061861 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:07.061490 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:41:07.061861 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:07.061573 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gsw5" podUID="28d7c937-a848-4d26-861f-29f9d2b0baf7" Jul 22 12:41:08.061315 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:08.061284 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:41:08.061477 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:08.061398 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jch9g" podUID="3c54132f-3e4d-427b-b38d-cd0daabb33d9" Jul 22 12:41:09.061110 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:09.061074 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:41:09.061285 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:09.061116 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:41:09.061285 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:09.061190 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ghfv" podUID="9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed" Jul 22 12:41:09.061388 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:09.061325 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gsw5" podUID="28d7c937-a848-4d26-861f-29f9d2b0baf7" Jul 22 12:41:10.061792 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:10.061766 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:41:10.062234 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:10.061874 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jch9g" podUID="3c54132f-3e4d-427b-b38d-cd0daabb33d9" Jul 22 12:41:11.061323 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:11.061293 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:41:11.061323 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:11.061326 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:41:11.061519 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:11.061407 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ghfv" podUID="9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed" Jul 22 12:41:11.061575 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:11.061522 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gsw5" podUID="28d7c937-a848-4d26-861f-29f9d2b0baf7" Jul 22 12:41:11.757623 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:11.757582 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c54132f-3e4d-427b-b38d-cd0daabb33d9-metrics-certs\") pod \"network-metrics-daemon-jch9g\" (UID: \"3c54132f-3e4d-427b-b38d-cd0daabb33d9\") " pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:41:11.757623 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:11.757623 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28d7c937-a848-4d26-861f-29f9d2b0baf7-original-pull-secret\") pod \"global-pull-secret-syncer-7gsw5\" (UID: \"28d7c937-a848-4d26-861f-29f9d2b0baf7\") " pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:41:11.758070 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:11.757778 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 12:41:11.758070 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:11.757781 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:41:11.758070 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:11.757829 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28d7c937-a848-4d26-861f-29f9d2b0baf7-original-pull-secret podName:28d7c937-a848-4d26-861f-29f9d2b0baf7 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:27.757814293 +0000 UTC m=+36.159045194 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28d7c937-a848-4d26-861f-29f9d2b0baf7-original-pull-secret") pod "global-pull-secret-syncer-7gsw5" (UID: "28d7c937-a848-4d26-861f-29f9d2b0baf7") : object "kube-system"/"original-pull-secret" not registered Jul 22 12:41:11.758070 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:11.757845 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c54132f-3e4d-427b-b38d-cd0daabb33d9-metrics-certs podName:3c54132f-3e4d-427b-b38d-cd0daabb33d9 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:27.7578384 +0000 UTC m=+36.159069297 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c54132f-3e4d-427b-b38d-cd0daabb33d9-metrics-certs") pod "network-metrics-daemon-jch9g" (UID: "3c54132f-3e4d-427b-b38d-cd0daabb33d9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:41:11.858717 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:11.858685 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxmtx\" (UniqueName: \"kubernetes.io/projected/9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed-kube-api-access-hxmtx\") pod \"network-check-target-6ghfv\" (UID: \"9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed\") " pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:41:11.858871 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:11.858853 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 12:41:11.858942 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:11.858878 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 12:41:11.858942 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:11.858892 2579 projected.go:196] Error preparing data for projected volume kube-api-access-hxmtx for pod openshift-network-diagnostics/network-check-target-6ghfv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:41:11.859046 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:11.858956 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed-kube-api-access-hxmtx podName:9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed nodeName:}" failed. No retries permitted until 2026-07-22 12:41:27.858938703 +0000 UTC m=+36.260169618 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxmtx" (UniqueName: "kubernetes.io/projected/9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed-kube-api-access-hxmtx") pod "network-check-target-6ghfv" (UID: "9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:41:12.061958 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:12.061912 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:41:12.062111 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:12.062061 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jch9g" podUID="3c54132f-3e4d-427b-b38d-cd0daabb33d9" Jul 22 12:41:13.061759 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:13.061732 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:41:13.062172 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:13.061733 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:41:13.062172 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:13.061828 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ghfv" podUID="9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed" Jul 22 12:41:13.062172 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:13.061955 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gsw5" podUID="28d7c937-a848-4d26-861f-29f9d2b0baf7" Jul 22 12:41:14.062019 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:14.061988 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:41:14.062447 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:14.062111 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jch9g" podUID="3c54132f-3e4d-427b-b38d-cd0daabb33d9" Jul 22 12:41:15.061541 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:15.061499 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:41:15.061738 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:15.061628 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ghfv" podUID="9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed" Jul 22 12:41:15.061738 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:15.061516 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:41:15.061849 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:15.061792 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gsw5" podUID="28d7c937-a848-4d26-861f-29f9d2b0baf7" Jul 22 12:41:16.061431 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:16.061404 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:41:16.061844 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:16.061592 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jch9g" podUID="3c54132f-3e4d-427b-b38d-cd0daabb33d9" Jul 22 12:41:17.062054 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.061756 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:41:17.062634 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.061779 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:41:17.062634 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:17.062145 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ghfv" podUID="9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed" Jul 22 12:41:17.062634 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:17.062198 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gsw5" podUID="28d7c937-a848-4d26-861f-29f9d2b0baf7" Jul 22 12:41:17.071994 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.071960 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ns7kz" event={"ID":"3dc0c773-526a-4eee-8155-77cf64929958","Type":"ContainerStarted","Data":"b2bc31db1d734068683b58e8c54e5785feaf6151fe8daab5efed7a6f30c51270"} Jul 22 12:41:17.074917 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.074892 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ddj6_b28e0ca4-8512-45da-91ed-2cde1491b2e2/ovn-acl-logging/0.log" Jul 22 12:41:17.075317 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.075294 2579 generic.go:356] "Generic (PLEG): container finished" podID="b28e0ca4-8512-45da-91ed-2cde1491b2e2" containerID="8a3c190424d0d38f6a1f06c1f85ca27d107ade7ac480cd86b5e88fcf4e183b86" exitCode=1 Jul 22 12:41:17.075402 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.075368 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" event={"ID":"b28e0ca4-8512-45da-91ed-2cde1491b2e2","Type":"ContainerStarted","Data":"423b6fdc43212cc319833decb81577c15524a33f69f71921e66c16d9d49edaf9"} Jul 22 12:41:17.075446 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.075419 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" event={"ID":"b28e0ca4-8512-45da-91ed-2cde1491b2e2","Type":"ContainerStarted","Data":"d6231ab9fda2d13f6d6e3d48788aad0ef80fe04ef70011548add842af57ef82d"} Jul 22 12:41:17.075478 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.075451 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" event={"ID":"b28e0ca4-8512-45da-91ed-2cde1491b2e2","Type":"ContainerStarted","Data":"d8d7b403eb4f5b512b755e7dd0d87bab260a4191f0ba438376f6b02c447b431a"} Jul 22 12:41:17.075526 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.075487 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" event={"ID":"b28e0ca4-8512-45da-91ed-2cde1491b2e2","Type":"ContainerStarted","Data":"c9d1219002201ad2dfe160b3613cbcc3295b739dff23149adaf8ec2aaef79de8"} Jul 22 12:41:17.075567 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.075524 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" event={"ID":"b28e0ca4-8512-45da-91ed-2cde1491b2e2","Type":"ContainerDied","Data":"8a3c190424d0d38f6a1f06c1f85ca27d107ade7ac480cd86b5e88fcf4e183b86"} Jul 22 12:41:17.075610 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.075574 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" event={"ID":"b28e0ca4-8512-45da-91ed-2cde1491b2e2","Type":"ContainerStarted","Data":"405b931b653fcbe9865215986013bd3938cf3eeec848991b715c53c1b7dd39ce"} Jul 22 12:41:17.077079 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.077043 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w6nhq" event={"ID":"c504a9e7-e48d-496d-8a1d-3b307361714c","Type":"ContainerStarted","Data":"afd4d0a792a7b615fcdd7c19d3c84671cab5f95addd2849ac324bc049beeef78"} Jul 22 12:41:17.078487 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.078449 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fgc67" event={"ID":"8b80aabc-b7ef-4649-bb58-4e0a39906c4f","Type":"ContainerStarted","Data":"37e66700a2af17f2626edf07ce1d4956a92af6d111d1cbc3f37dbaa2ceef81fa"} Jul 22 12:41:17.079917 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.079886 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" event={"ID":"d5897b36-633d-4ccc-b69f-7b50e656545c","Type":"ContainerStarted","Data":"1edccf3a8b6a0c15d4482af7a60e50935cd69c8447cdf3189c5ec0e0907c25f5"} Jul 22 12:41:17.081392 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.081368 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pgrh7" event={"ID":"2fee5353-19d0-4530-baff-6831516314d3","Type":"ContainerStarted","Data":"15a090125ec56b5996b0cb223f0ae1b6cf5df858674d74cf360463fe3b6066de"} Jul 22 12:41:17.082892 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.082873 2579 generic.go:356] "Generic (PLEG): container finished" podID="24f69e77-f691-4a49-9cbf-8000995cd2a0" containerID="69a7f304994bf2e235d25165d55ba7f49a43d22611a1750853730f522f0a1a3f" exitCode=0 Jul 22 12:41:17.082992 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.082918 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sfx4m" event={"ID":"24f69e77-f691-4a49-9cbf-8000995cd2a0","Type":"ContainerDied","Data":"69a7f304994bf2e235d25165d55ba7f49a43d22611a1750853730f522f0a1a3f"} Jul 22 12:41:17.087771 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.087696 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" event={"ID":"55113322-3d54-4862-a5da-17d438ad52a7","Type":"ContainerStarted","Data":"85a5a2c941b5b6dabc79c9d738fd5fa261240f06a995ffd984ed0d525f448e76"} Jul 22 12:41:17.104828 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.104778 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-34.ec2.internal" podStartSLOduration=22.10476393 podStartE2EDuration="22.10476393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:40:59.059336909 +0000 UTC m=+7.460567835" watchObservedRunningTime="2026-07-22 12:41:17.10476393 +0000 UTC m=+25.505994851" Jul 22 12:41:17.105421 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.105395 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ns7kz" podStartSLOduration=14.573388926 podStartE2EDuration="25.105386634s" totalImagesPullingTime="10.531997708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:17.104703605 +0000 UTC m=+25.505934524" watchObservedRunningTime="2026-07-22 12:41:17.105386634 +0000 UTC m=+25.506617553" Jul 22 12:41:17.184819 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.184765 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pgrh7" podStartSLOduration=6.285359404 podStartE2EDuration="25.18474532s" totalImagesPullingTime="18.899385916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:17.148491582 +0000 UTC m=+25.549722502" watchObservedRunningTime="2026-07-22 12:41:17.18474532 +0000 UTC m=+25.585976243" Jul 22 12:41:17.266061 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.266016 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fgc67" podStartSLOduration=5.513102694 podStartE2EDuration="24.266001665s" totalImagesPullingTime="18.752898971s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:17.265999689 +0000 UTC m=+25.667230609" watchObservedRunningTime="2026-07-22 12:41:17.266001665 +0000 UTC m=+25.667232586" Jul 22 12:41:17.266361 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.266339 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wgxgn" podStartSLOduration=6.526834257 podStartE2EDuration="25.266333239s" totalImagesPullingTime="18.739498982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:17.230443503 +0000 UTC m=+25.631674424" watchObservedRunningTime="2026-07-22 12:41:17.266333239 +0000 UTC m=+25.667564158" Jul 22 12:41:17.284680 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.284645 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-w6nhq" podStartSLOduration=6.519318177 podStartE2EDuration="25.284632561s" totalImagesPullingTime="18.765314384s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:17.284086283 +0000 UTC m=+25.685317204" watchObservedRunningTime="2026-07-22 12:41:17.284632561 +0000 UTC m=+25.685863480" Jul 22 12:41:17.710692 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.710669 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 22 12:41:17.919618 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.919402 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-22T12:41:17.710687816Z","UUID":"d5b17850-26ae-4561-bbf2-7d3cc8c1de30","Handler":null,"Name":"","Endpoint":""} Jul 22 12:41:17.921018 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.920998 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 22 12:41:17.921143 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:17.921031 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 22 12:41:18.061560 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:18.061531 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:41:18.061782 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:18.061679 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jch9g" podUID="3c54132f-3e4d-427b-b38d-cd0daabb33d9" Jul 22 12:41:18.091637 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:18.091606 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5c5j7" event={"ID":"6f3b3f37-8156-495b-b73b-011f8decc3fa","Type":"ContainerStarted","Data":"b3a28195f622c7762a7cc99a6ba33ae01a376f3ca3290d8d30145b27d34d7b2d"} Jul 22 12:41:18.093590 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:18.093554 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" event={"ID":"55113322-3d54-4862-a5da-17d438ad52a7","Type":"ContainerStarted","Data":"a6857249ec49e46ac222beb330e75a222677fcd772548ea01c84d14713f7721d"} Jul 22 12:41:18.107446 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:18.107405 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5c5j7" podStartSLOduration=6.331978552 podStartE2EDuration="25.107391038s" totalImagesPullingTime="18.775412486s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:18.107265282 +0000 UTC m=+26.508496204" watchObservedRunningTime="2026-07-22 12:41:18.107391038 +0000 UTC m=+26.508621957" Jul 22 12:41:19.062207 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:19.062170 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:41:19.062207 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:19.062204 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:41:19.062415 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:19.062287 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gsw5" podUID="28d7c937-a848-4d26-861f-29f9d2b0baf7" Jul 22 12:41:19.062474 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:19.062441 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ghfv" podUID="9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed" Jul 22 12:41:19.098984 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:19.098961 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ddj6_b28e0ca4-8512-45da-91ed-2cde1491b2e2/ovn-acl-logging/0.log" Jul 22 12:41:19.099479 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:19.099450 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" event={"ID":"b28e0ca4-8512-45da-91ed-2cde1491b2e2","Type":"ContainerStarted","Data":"14073a3825ad019f85a01870d19ed6fa5576ba8b56cdcb1ab7e807fc7eda2fcc"} Jul 22 12:41:19.101618 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:19.101587 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" event={"ID":"55113322-3d54-4862-a5da-17d438ad52a7","Type":"ContainerStarted","Data":"07023cd3b11da0a764d52d05f12eb7afb2ba301cdbb806134da0dce2f26e355d"} Jul 22 12:41:19.121631 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:19.121581 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-82v8w" podStartSLOduration=5.963059862 podStartE2EDuration="27.121568809s" totalImagesPullingTime="21.158508947s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:40:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:19.121225874 +0000 UTC m=+27.522456797" watchObservedRunningTime="2026-07-22 12:41:19.121568809 +0000 UTC m=+27.522799729" Jul 22 12:41:20.061535 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:20.061507 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:41:20.061714 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:20.061636 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jch9g" podUID="3c54132f-3e4d-427b-b38d-cd0daabb33d9" Jul 22 12:41:20.962276 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:20.962086 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ns7kz" Jul 22 12:41:20.962750 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:20.962727 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ns7kz" Jul 22 12:41:21.061357 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:21.061327 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:41:21.061535 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:21.061369 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:41:21.061535 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:21.061430 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ghfv" podUID="9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed" Jul 22 12:41:21.061535 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:21.061469 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gsw5" podUID="28d7c937-a848-4d26-861f-29f9d2b0baf7" Jul 22 12:41:21.106135 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:21.106105 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ns7kz" Jul 22 12:41:21.106635 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:21.106615 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ns7kz" Jul 22 12:41:22.062589 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:22.062337 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:41:22.063282 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:22.062687 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jch9g" podUID="3c54132f-3e4d-427b-b38d-cd0daabb33d9" Jul 22 12:41:22.111267 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:22.111239 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ddj6_b28e0ca4-8512-45da-91ed-2cde1491b2e2/ovn-acl-logging/0.log" Jul 22 12:41:22.111615 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:22.111582 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" event={"ID":"b28e0ca4-8512-45da-91ed-2cde1491b2e2","Type":"ContainerStarted","Data":"fb89ae94a651e01b7c6d10284b51532586a1b3dd6647fb4d5ec2de95254aafed"} Jul 22 12:41:22.111979 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:22.111950 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:41:22.112065 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:22.112031 2579 scope.go:117] "RemoveContainer" containerID="8a3c190424d0d38f6a1f06c1f85ca27d107ade7ac480cd86b5e88fcf4e183b86" Jul 22 12:41:22.113424 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:22.113401 2579 generic.go:356] "Generic (PLEG): container finished" podID="24f69e77-f691-4a49-9cbf-8000995cd2a0" containerID="ab189bf2c400879a92569cbabac67de30b2042791266471c33f4a8373e95ce38" exitCode=0 Jul 22 12:41:22.113526 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:22.113486 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sfx4m" event={"ID":"24f69e77-f691-4a49-9cbf-8000995cd2a0","Type":"ContainerDied","Data":"ab189bf2c400879a92569cbabac67de30b2042791266471c33f4a8373e95ce38"} Jul 22 12:41:22.129187 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:22.129168 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:41:23.061848 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:23.061814 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:41:23.061848 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:23.061845 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:41:23.062086 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:23.061968 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ghfv" podUID="9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed" Jul 22 12:41:23.062136 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:23.062086 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gsw5" podUID="28d7c937-a848-4d26-861f-29f9d2b0baf7" Jul 22 12:41:23.118012 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:23.117977 2579 generic.go:356] "Generic (PLEG): container finished" podID="24f69e77-f691-4a49-9cbf-8000995cd2a0" containerID="e7d5db02cd39fea27b3458692832bac46c7552b2800c1aaeb8cf1bbc11f8b5d3" exitCode=0 Jul 22 12:41:23.118430 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:23.118063 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sfx4m" event={"ID":"24f69e77-f691-4a49-9cbf-8000995cd2a0","Type":"ContainerDied","Data":"e7d5db02cd39fea27b3458692832bac46c7552b2800c1aaeb8cf1bbc11f8b5d3"} Jul 22 12:41:23.121900 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:23.121884 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ddj6_b28e0ca4-8512-45da-91ed-2cde1491b2e2/ovn-acl-logging/0.log" Jul 22 12:41:23.122304 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:23.122278 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" event={"ID":"b28e0ca4-8512-45da-91ed-2cde1491b2e2","Type":"ContainerStarted","Data":"c569cac23b082d8ff4d90732877b6dc72d8900060fe32b2ae19c3dc56c689ff2"} Jul 22 12:41:23.122383 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:23.122364 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:41:23.122383 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:23.122295 2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 22 12:41:23.137887 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:23.137865 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:41:23.182431 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:23.182334 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" podStartSLOduration=12.358315711 podStartE2EDuration="31.182319676s" totalImagesPullingTime="18.824003965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:23.180644525 +0000 UTC m=+31.581875480" watchObservedRunningTime="2026-07-22 12:41:23.182319676 +0000 UTC m=+31.583550595" Jul 22 12:41:23.305897 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:23.305865 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7gsw5"] Jul 22 12:41:23.306044 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:23.305971 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:41:23.306087 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:23.306051 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gsw5" podUID="28d7c937-a848-4d26-861f-29f9d2b0baf7" Jul 22 12:41:23.309011 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:23.308985 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jch9g"] Jul 22 12:41:23.309125 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:23.309077 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:41:23.309193 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:23.309176 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jch9g" podUID="3c54132f-3e4d-427b-b38d-cd0daabb33d9" Jul 22 12:41:23.319265 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:23.319240 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6ghfv"] Jul 22 12:41:23.319341 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:23.319308 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:41:23.319398 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:23.319382 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ghfv" podUID="9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed" Jul 22 12:41:24.126378 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:24.126346 2579 generic.go:356] "Generic (PLEG): container finished" podID="24f69e77-f691-4a49-9cbf-8000995cd2a0" containerID="ad744a835ebee57f520f174179efaea1baae78dbd2614f140d34702648da625f" exitCode=0 Jul 22 12:41:24.126729 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:24.126444 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sfx4m" event={"ID":"24f69e77-f691-4a49-9cbf-8000995cd2a0","Type":"ContainerDied","Data":"ad744a835ebee57f520f174179efaea1baae78dbd2614f140d34702648da625f"} Jul 22 12:41:24.126729 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:24.126546 2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 22 12:41:25.061775 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:25.061745 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:41:25.061904 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:25.061773 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:41:25.061904 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:25.061773 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:41:25.061904 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:25.061886 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jch9g" podUID="3c54132f-3e4d-427b-b38d-cd0daabb33d9" Jul 22 12:41:25.062082 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:25.062023 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ghfv" podUID="9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed" Jul 22 12:41:25.062136 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:25.062111 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gsw5" podUID="28d7c937-a848-4d26-861f-29f9d2b0baf7" Jul 22 12:41:25.128916 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:25.128887 2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 22 12:41:25.936100 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:25.935910 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:41:26.144066 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:26.144022 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" podUID="b28e0ca4-8512-45da-91ed-2cde1491b2e2" containerName="ovnkube-controller" probeResult="failure" output="" Jul 22 12:41:27.061348 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:27.061314 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:41:27.061575 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:27.061314 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:41:27.061575 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:27.061443 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jch9g" podUID="3c54132f-3e4d-427b-b38d-cd0daabb33d9" Jul 22 12:41:27.061575 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:27.061521 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7gsw5" podUID="28d7c937-a848-4d26-861f-29f9d2b0baf7" Jul 22 12:41:27.061575 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:27.061315 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:41:27.061762 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:27.061609 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6ghfv" podUID="9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed" Jul 22 12:41:27.777774 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:27.777690 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c54132f-3e4d-427b-b38d-cd0daabb33d9-metrics-certs\") pod \"network-metrics-daemon-jch9g\" (UID: \"3c54132f-3e4d-427b-b38d-cd0daabb33d9\") " pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:41:27.777774 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:27.777730 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28d7c937-a848-4d26-861f-29f9d2b0baf7-original-pull-secret\") pod \"global-pull-secret-syncer-7gsw5\" (UID: \"28d7c937-a848-4d26-861f-29f9d2b0baf7\") " pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:41:27.778361 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:27.777839 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:41:27.778361 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:27.777904 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 22 12:41:27.778361 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:27.777912 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c54132f-3e4d-427b-b38d-cd0daabb33d9-metrics-certs podName:3c54132f-3e4d-427b-b38d-cd0daabb33d9 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:59.777891208 +0000 UTC m=+68.179122108 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c54132f-3e4d-427b-b38d-cd0daabb33d9-metrics-certs") pod "network-metrics-daemon-jch9g" (UID: "3c54132f-3e4d-427b-b38d-cd0daabb33d9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 22 12:41:27.778361 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:27.777980 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28d7c937-a848-4d26-861f-29f9d2b0baf7-original-pull-secret podName:28d7c937-a848-4d26-861f-29f9d2b0baf7 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:59.777964162 +0000 UTC m=+68.179195074 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/28d7c937-a848-4d26-861f-29f9d2b0baf7-original-pull-secret") pod "global-pull-secret-syncer-7gsw5" (UID: "28d7c937-a848-4d26-861f-29f9d2b0baf7") : object "kube-system"/"original-pull-secret" not registered Jul 22 12:41:27.878060 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:27.878023 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxmtx\" (UniqueName: \"kubernetes.io/projected/9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed-kube-api-access-hxmtx\") pod \"network-check-target-6ghfv\" (UID: \"9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed\") " pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:41:27.878266 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:27.878148 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 22 12:41:27.878266 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:27.878171 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 22 12:41:27.878266 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:27.878183 2579 projected.go:196] Error preparing data for projected volume kube-api-access-hxmtx for pod openshift-network-diagnostics/network-check-target-6ghfv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:41:27.878266 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:27.878234 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed-kube-api-access-hxmtx podName:9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed nodeName:}" failed. No retries permitted until 2026-07-22 12:41:59.87821582 +0000 UTC m=+68.279446735 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hxmtx" (UniqueName: "kubernetes.io/projected/9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed-kube-api-access-hxmtx") pod "network-check-target-6ghfv" (UID: "9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 22 12:41:28.892246 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:28.892220 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-34.ec2.internal" event="NodeReady" Jul 22 12:41:28.892719 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:28.892360 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 22 12:41:28.941056 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:28.941029 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b47b6758f-f82nc"] Jul 22 12:41:28.956875 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:28.956855 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:28.964049 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:28.964019 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 22 12:41:28.964146 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:28.964058 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 22 12:41:28.967995 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:28.967978 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fgszx\"" Jul 22 12:41:28.973241 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:28.973226 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 22 12:41:28.981186 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:28.981167 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-w8md4"] Jul 22 12:41:29.004181 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.004159 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7q7fk"] Jul 22 12:41:29.004298 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.004279 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w8md4" Jul 22 12:41:29.015439 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.015423 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 22 12:41:29.016306 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.016286 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 22 12:41:29.026008 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.025990 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kbrpx\"" Jul 22 12:41:29.026299 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.026283 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7q7fk" Jul 22 12:41:29.029527 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.029510 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 22 12:41:29.029618 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.029544 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 22 12:41:29.029990 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.029976 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gqts7\"" Jul 22 12:41:29.030069 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.030027 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 22 12:41:29.034302 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.034282 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w8md4"] Jul 22 12:41:29.060033 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.060010 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 22 12:41:29.061221 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.061133 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:41:29.061317 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.061245 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:41:29.061317 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.061295 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:41:29.063408 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.063382 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7q7fk"] Jul 22 12:41:29.064550 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.064323 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 12:41:29.064550 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.064353 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 12:41:29.064550 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.064462 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sv8qm\"" Jul 22 12:41:29.064986 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.064757 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-72nf2\"" Jul 22 12:41:29.064986 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.064823 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 12:41:29.064986 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.064938 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 12:41:29.085685 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.085663 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3febf38a-9737-4258-8621-a343c7803c28-ca-trust-extracted\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:29.085795 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.085730 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3febf38a-9737-4258-8621-a343c7803c28-trusted-ca\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:29.085795 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.085759 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3febf38a-9737-4258-8621-a343c7803c28-installation-pull-secrets\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:29.085895 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.085801 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-bound-sa-token\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:29.085895 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.085825 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3febf38a-9737-4258-8621-a343c7803c28-registry-certificates\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:29.085895 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.085852 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3febf38a-9737-4258-8621-a343c7803c28-image-registry-private-configuration\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:29.085895 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.085886 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmdcz\" (UniqueName: \"kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-kube-api-access-dmdcz\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:29.086055 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.085969 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-registry-tls\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:29.088570 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.088550 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b47b6758f-f82nc"] Jul 22 12:41:29.140609 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.140574 2579 generic.go:356] "Generic (PLEG): container finished" podID="24f69e77-f691-4a49-9cbf-8000995cd2a0" containerID="140e3dca21cd8237a0e992e78b32aacf44affbf95532a9d2a33d63f30b5b68b9" exitCode=0 Jul 22 12:41:29.140742 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.140622 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sfx4m" event={"ID":"24f69e77-f691-4a49-9cbf-8000995cd2a0","Type":"ContainerDied","Data":"140e3dca21cd8237a0e992e78b32aacf44affbf95532a9d2a33d63f30b5b68b9"} Jul 22 12:41:29.187253 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.187218 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x28bs\" (UniqueName: \"kubernetes.io/projected/e9effaf4-801b-4f94-856f-9a934fd06a77-kube-api-access-x28bs\") pod \"ingress-canary-7q7fk\" (UID: \"e9effaf4-801b-4f94-856f-9a934fd06a77\") " pod="openshift-ingress-canary/ingress-canary-7q7fk" Jul 22 12:41:29.187365 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.187280 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3febf38a-9737-4258-8621-a343c7803c28-trusted-ca\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:29.187365 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.187309 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/394708f4-ca0a-4fb2-a3d2-a22224322e3d-tmp-dir\") pod \"dns-default-w8md4\" (UID: \"394708f4-ca0a-4fb2-a3d2-a22224322e3d\") " pod="openshift-dns/dns-default-w8md4" Jul 22 12:41:29.187365 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.187338 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3febf38a-9737-4258-8621-a343c7803c28-installation-pull-secrets\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:29.187576 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.187378 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-bound-sa-token\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:29.187576 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.187406 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3febf38a-9737-4258-8621-a343c7803c28-registry-certificates\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:29.187576 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.187429 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3febf38a-9737-4258-8621-a343c7803c28-image-registry-private-configuration\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:29.187576 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.187445 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmdcz\" (UniqueName: \"kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-kube-api-access-dmdcz\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:29.187576 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.187460 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9effaf4-801b-4f94-856f-9a934fd06a77-cert\") pod \"ingress-canary-7q7fk\" (UID: \"e9effaf4-801b-4f94-856f-9a934fd06a77\") " pod="openshift-ingress-canary/ingress-canary-7q7fk" Jul 22 12:41:29.187576 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.187480 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2k7z\" (UniqueName: \"kubernetes.io/projected/394708f4-ca0a-4fb2-a3d2-a22224322e3d-kube-api-access-m2k7z\") pod \"dns-default-w8md4\" (UID: \"394708f4-ca0a-4fb2-a3d2-a22224322e3d\") " pod="openshift-dns/dns-default-w8md4" Jul 22 12:41:29.188237 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.188200 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-registry-tls\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:29.188347 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:29.188306 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 12:41:29.188347 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:29.188323 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b47b6758f-f82nc: secret "image-registry-tls" not found Jul 22 12:41:29.188347 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.188302 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/394708f4-ca0a-4fb2-a3d2-a22224322e3d-config-volume\") pod \"dns-default-w8md4\" (UID: \"394708f4-ca0a-4fb2-a3d2-a22224322e3d\") " pod="openshift-dns/dns-default-w8md4" Jul 22 12:41:29.188480 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:29.188425 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-registry-tls podName:3febf38a-9737-4258-8621-a343c7803c28 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:29.688404598 +0000 UTC m=+38.089635510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-registry-tls") pod "image-registry-b47b6758f-f82nc" (UID: "3febf38a-9737-4258-8621-a343c7803c28") : secret "image-registry-tls" not found Jul 22 12:41:29.188480 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.188444 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/394708f4-ca0a-4fb2-a3d2-a22224322e3d-metrics-tls\") pod \"dns-default-w8md4\" (UID: \"394708f4-ca0a-4fb2-a3d2-a22224322e3d\") " pod="openshift-dns/dns-default-w8md4" Jul 22 12:41:29.188568 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.188505 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3febf38a-9737-4258-8621-a343c7803c28-ca-trust-extracted\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:29.189233 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.189193 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3febf38a-9737-4258-8621-a343c7803c28-ca-trust-extracted\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:29.189519 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.189498 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3febf38a-9737-4258-8621-a343c7803c28-registry-certificates\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:29.189597 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.189545 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3febf38a-9737-4258-8621-a343c7803c28-trusted-ca\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:29.192219 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.192201 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3febf38a-9737-4258-8621-a343c7803c28-installation-pull-secrets\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:29.192346 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.192229 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3febf38a-9737-4258-8621-a343c7803c28-image-registry-private-configuration\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:29.196539 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.196515 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmdcz\" (UniqueName: \"kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-kube-api-access-dmdcz\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:29.196632 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.196555 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-bound-sa-token\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:29.288989 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.288966 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x28bs\" (UniqueName: \"kubernetes.io/projected/e9effaf4-801b-4f94-856f-9a934fd06a77-kube-api-access-x28bs\") pod \"ingress-canary-7q7fk\" (UID: \"e9effaf4-801b-4f94-856f-9a934fd06a77\") " pod="openshift-ingress-canary/ingress-canary-7q7fk" Jul 22 12:41:29.289088 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.289006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/394708f4-ca0a-4fb2-a3d2-a22224322e3d-tmp-dir\") pod \"dns-default-w8md4\" (UID: \"394708f4-ca0a-4fb2-a3d2-a22224322e3d\") " pod="openshift-dns/dns-default-w8md4" Jul 22 12:41:29.289167 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.289142 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9effaf4-801b-4f94-856f-9a934fd06a77-cert\") pod \"ingress-canary-7q7fk\" (UID: \"e9effaf4-801b-4f94-856f-9a934fd06a77\") " pod="openshift-ingress-canary/ingress-canary-7q7fk" Jul 22 12:41:29.289224 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.289189 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2k7z\" (UniqueName: \"kubernetes.io/projected/394708f4-ca0a-4fb2-a3d2-a22224322e3d-kube-api-access-m2k7z\") pod \"dns-default-w8md4\" (UID: \"394708f4-ca0a-4fb2-a3d2-a22224322e3d\") " pod="openshift-dns/dns-default-w8md4" Jul 22 12:41:29.289278 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.289238 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/394708f4-ca0a-4fb2-a3d2-a22224322e3d-config-volume\") pod \"dns-default-w8md4\" (UID: \"394708f4-ca0a-4fb2-a3d2-a22224322e3d\") " pod="openshift-dns/dns-default-w8md4" Jul 22 12:41:29.289278 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.289264 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/394708f4-ca0a-4fb2-a3d2-a22224322e3d-metrics-tls\") pod \"dns-default-w8md4\" (UID: \"394708f4-ca0a-4fb2-a3d2-a22224322e3d\") " pod="openshift-dns/dns-default-w8md4" Jul 22 12:41:29.289383 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:29.289365 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 12:41:29.289430 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.289410 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/394708f4-ca0a-4fb2-a3d2-a22224322e3d-tmp-dir\") pod \"dns-default-w8md4\" (UID: \"394708f4-ca0a-4fb2-a3d2-a22224322e3d\") " pod="openshift-dns/dns-default-w8md4" Jul 22 12:41:29.289482 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:29.289441 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/394708f4-ca0a-4fb2-a3d2-a22224322e3d-metrics-tls podName:394708f4-ca0a-4fb2-a3d2-a22224322e3d nodeName:}" failed. No retries permitted until 2026-07-22 12:41:29.789421149 +0000 UTC m=+38.190652050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/394708f4-ca0a-4fb2-a3d2-a22224322e3d-metrics-tls") pod "dns-default-w8md4" (UID: "394708f4-ca0a-4fb2-a3d2-a22224322e3d") : secret "dns-default-metrics-tls" not found Jul 22 12:41:29.289779 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.289760 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/394708f4-ca0a-4fb2-a3d2-a22224322e3d-config-volume\") pod \"dns-default-w8md4\" (UID: \"394708f4-ca0a-4fb2-a3d2-a22224322e3d\") " pod="openshift-dns/dns-default-w8md4" Jul 22 12:41:29.291372 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.291354 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e9effaf4-801b-4f94-856f-9a934fd06a77-cert\") pod \"ingress-canary-7q7fk\" (UID: \"e9effaf4-801b-4f94-856f-9a934fd06a77\") " pod="openshift-ingress-canary/ingress-canary-7q7fk" Jul 22 12:41:29.298232 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.298213 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x28bs\" (UniqueName: \"kubernetes.io/projected/e9effaf4-801b-4f94-856f-9a934fd06a77-kube-api-access-x28bs\") pod \"ingress-canary-7q7fk\" (UID: \"e9effaf4-801b-4f94-856f-9a934fd06a77\") " pod="openshift-ingress-canary/ingress-canary-7q7fk" Jul 22 12:41:29.301411 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.301391 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2k7z\" (UniqueName: \"kubernetes.io/projected/394708f4-ca0a-4fb2-a3d2-a22224322e3d-kube-api-access-m2k7z\") pod \"dns-default-w8md4\" (UID: \"394708f4-ca0a-4fb2-a3d2-a22224322e3d\") " pod="openshift-dns/dns-default-w8md4" Jul 22 12:41:29.334704 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.334682 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7q7fk" Jul 22 12:41:29.492409 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.492219 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7q7fk"] Jul 22 12:41:29.495834 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:41:29.495810 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9effaf4_801b_4f94_856f_9a934fd06a77.slice/crio-98d5d5723a2a517edce1628abc25f73d34c781d44c9b16304090669a27d30052 WatchSource:0}: Error finding container 98d5d5723a2a517edce1628abc25f73d34c781d44c9b16304090669a27d30052: Status 404 returned error can't find the container with id 98d5d5723a2a517edce1628abc25f73d34c781d44c9b16304090669a27d30052 Jul 22 12:41:29.692728 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.692612 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-registry-tls\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:29.692902 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:29.692780 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 12:41:29.692902 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:29.692803 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b47b6758f-f82nc: secret "image-registry-tls" not found Jul 22 12:41:29.692902 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:29.692871 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-registry-tls podName:3febf38a-9737-4258-8621-a343c7803c28 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:30.69285406 +0000 UTC m=+39.094084971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-registry-tls") pod "image-registry-b47b6758f-f82nc" (UID: "3febf38a-9737-4258-8621-a343c7803c28") : secret "image-registry-tls" not found Jul 22 12:41:29.793248 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:29.793211 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/394708f4-ca0a-4fb2-a3d2-a22224322e3d-metrics-tls\") pod \"dns-default-w8md4\" (UID: \"394708f4-ca0a-4fb2-a3d2-a22224322e3d\") " pod="openshift-dns/dns-default-w8md4" Jul 22 12:41:29.793390 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:29.793356 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 12:41:29.793454 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:29.793415 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/394708f4-ca0a-4fb2-a3d2-a22224322e3d-metrics-tls podName:394708f4-ca0a-4fb2-a3d2-a22224322e3d nodeName:}" failed. No retries permitted until 2026-07-22 12:41:30.793401363 +0000 UTC m=+39.194632261 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/394708f4-ca0a-4fb2-a3d2-a22224322e3d-metrics-tls") pod "dns-default-w8md4" (UID: "394708f4-ca0a-4fb2-a3d2-a22224322e3d") : secret "dns-default-metrics-tls" not found Jul 22 12:41:30.143942 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:30.143891 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7q7fk" event={"ID":"e9effaf4-801b-4f94-856f-9a934fd06a77","Type":"ContainerStarted","Data":"98d5d5723a2a517edce1628abc25f73d34c781d44c9b16304090669a27d30052"} Jul 22 12:41:30.146780 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:30.146755 2579 generic.go:356] "Generic (PLEG): container finished" podID="24f69e77-f691-4a49-9cbf-8000995cd2a0" containerID="3c6ca909abdb3331dd68693e3ffc3227abb8f7f8d7cd3968f5e6df7a53e5fa49" exitCode=0 Jul 22 12:41:30.146883 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:30.146796 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sfx4m" event={"ID":"24f69e77-f691-4a49-9cbf-8000995cd2a0","Type":"ContainerDied","Data":"3c6ca909abdb3331dd68693e3ffc3227abb8f7f8d7cd3968f5e6df7a53e5fa49"} Jul 22 12:41:30.700244 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:30.700209 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-registry-tls\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:30.700414 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:30.700359 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 12:41:30.700414 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:30.700377 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b47b6758f-f82nc: secret "image-registry-tls" not found Jul 22 12:41:30.700501 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:30.700428 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-registry-tls podName:3febf38a-9737-4258-8621-a343c7803c28 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:32.700414105 +0000 UTC m=+41.101645003 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-registry-tls") pod "image-registry-b47b6758f-f82nc" (UID: "3febf38a-9737-4258-8621-a343c7803c28") : secret "image-registry-tls" not found Jul 22 12:41:30.801063 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:30.801022 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/394708f4-ca0a-4fb2-a3d2-a22224322e3d-metrics-tls\") pod \"dns-default-w8md4\" (UID: \"394708f4-ca0a-4fb2-a3d2-a22224322e3d\") " pod="openshift-dns/dns-default-w8md4" Jul 22 12:41:30.801227 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:30.801184 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 12:41:30.801273 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:30.801259 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/394708f4-ca0a-4fb2-a3d2-a22224322e3d-metrics-tls podName:394708f4-ca0a-4fb2-a3d2-a22224322e3d nodeName:}" failed. No retries permitted until 2026-07-22 12:41:32.80124413 +0000 UTC m=+41.202475031 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/394708f4-ca0a-4fb2-a3d2-a22224322e3d-metrics-tls") pod "dns-default-w8md4" (UID: "394708f4-ca0a-4fb2-a3d2-a22224322e3d") : secret "dns-default-metrics-tls" not found Jul 22 12:41:31.153356 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:31.153305 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sfx4m" event={"ID":"24f69e77-f691-4a49-9cbf-8000995cd2a0","Type":"ContainerStarted","Data":"8c805722ceb673315c79d59cd54a2baa3c815fd7c040c1fa9a61bfc8135b4c1d"} Jul 22 12:41:31.183169 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:31.183117 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sfx4m" podStartSLOduration=10.308356455 podStartE2EDuration="39.183101239s" totalImagesPullingTime="28.874744784s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:40:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:31.18114177 +0000 UTC m=+39.582372700" watchObservedRunningTime="2026-07-22 12:41:31.183101239 +0000 UTC m=+39.584332158" Jul 22 12:41:32.714852 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:32.714820 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-registry-tls\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:32.715165 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:32.714972 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 12:41:32.715165 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:32.714989 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b47b6758f-f82nc: secret "image-registry-tls" not found Jul 22 12:41:32.715165 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:32.715039 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-registry-tls podName:3febf38a-9737-4258-8621-a343c7803c28 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:36.71502525 +0000 UTC m=+45.116256153 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-registry-tls") pod "image-registry-b47b6758f-f82nc" (UID: "3febf38a-9737-4258-8621-a343c7803c28") : secret "image-registry-tls" not found Jul 22 12:41:32.815510 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:32.815481 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/394708f4-ca0a-4fb2-a3d2-a22224322e3d-metrics-tls\") pod \"dns-default-w8md4\" (UID: \"394708f4-ca0a-4fb2-a3d2-a22224322e3d\") " pod="openshift-dns/dns-default-w8md4" Jul 22 12:41:32.815651 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:32.815608 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 12:41:32.815689 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:32.815657 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/394708f4-ca0a-4fb2-a3d2-a22224322e3d-metrics-tls podName:394708f4-ca0a-4fb2-a3d2-a22224322e3d nodeName:}" failed. No retries permitted until 2026-07-22 12:41:36.815644321 +0000 UTC m=+45.216875223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/394708f4-ca0a-4fb2-a3d2-a22224322e3d-metrics-tls") pod "dns-default-w8md4" (UID: "394708f4-ca0a-4fb2-a3d2-a22224322e3d") : secret "dns-default-metrics-tls" not found Jul 22 12:41:33.159125 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:33.159085 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7q7fk" event={"ID":"e9effaf4-801b-4f94-856f-9a934fd06a77","Type":"ContainerStarted","Data":"7712999a2b11cbb1d40185c2422359c069e48c2c441a0f6d5fe7c2a17d099ea9"} Jul 22 12:41:33.181338 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:33.181291 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7q7fk" podStartSLOduration=1.993544435 podStartE2EDuration="5.181275557s" totalImagesPullingTime="3.187731122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:33.180706894 +0000 UTC m=+41.581937813" watchObservedRunningTime="2026-07-22 12:41:33.181275557 +0000 UTC m=+41.582506477" Jul 22 12:41:33.704343 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:33.704310 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dkt6h"] Jul 22 12:41:33.726079 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:33.726052 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dkt6h" Jul 22 12:41:33.729529 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:33.729499 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 22 12:41:33.729529 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:33.729518 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mvgjr\"" Jul 22 12:41:33.729700 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:33.729504 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 22 12:41:33.729700 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:33.729661 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dkt6h"] Jul 22 12:41:33.730040 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:33.730021 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 22 12:41:33.730125 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:33.730026 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 22 12:41:33.823376 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:33.823348 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cfcc578f-7c22-4248-a8d0-7695852ec582-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dkt6h\" (UID: \"cfcc578f-7c22-4248-a8d0-7695852ec582\") " pod="openshift-insights/insights-runtime-extractor-dkt6h" Jul 22 12:41:33.823376 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:33.823380 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cfcc578f-7c22-4248-a8d0-7695852ec582-crio-socket\") pod \"insights-runtime-extractor-dkt6h\" (UID: \"cfcc578f-7c22-4248-a8d0-7695852ec582\") " pod="openshift-insights/insights-runtime-extractor-dkt6h" Jul 22 12:41:33.823569 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:33.823403 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cfcc578f-7c22-4248-a8d0-7695852ec582-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dkt6h\" (UID: \"cfcc578f-7c22-4248-a8d0-7695852ec582\") " pod="openshift-insights/insights-runtime-extractor-dkt6h" Jul 22 12:41:33.823569 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:33.823499 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cfcc578f-7c22-4248-a8d0-7695852ec582-data-volume\") pod \"insights-runtime-extractor-dkt6h\" (UID: \"cfcc578f-7c22-4248-a8d0-7695852ec582\") " pod="openshift-insights/insights-runtime-extractor-dkt6h" Jul 22 12:41:33.823569 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:33.823528 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqf5m\" (UniqueName: \"kubernetes.io/projected/cfcc578f-7c22-4248-a8d0-7695852ec582-kube-api-access-jqf5m\") pod \"insights-runtime-extractor-dkt6h\" (UID: \"cfcc578f-7c22-4248-a8d0-7695852ec582\") " pod="openshift-insights/insights-runtime-extractor-dkt6h" Jul 22 12:41:33.924326 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:33.924294 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cfcc578f-7c22-4248-a8d0-7695852ec582-data-volume\") pod \"insights-runtime-extractor-dkt6h\" (UID: \"cfcc578f-7c22-4248-a8d0-7695852ec582\") " pod="openshift-insights/insights-runtime-extractor-dkt6h" Jul 22 12:41:33.924326 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:33.924330 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqf5m\" (UniqueName: \"kubernetes.io/projected/cfcc578f-7c22-4248-a8d0-7695852ec582-kube-api-access-jqf5m\") pod \"insights-runtime-extractor-dkt6h\" (UID: \"cfcc578f-7c22-4248-a8d0-7695852ec582\") " pod="openshift-insights/insights-runtime-extractor-dkt6h" Jul 22 12:41:33.924496 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:33.924394 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cfcc578f-7c22-4248-a8d0-7695852ec582-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dkt6h\" (UID: \"cfcc578f-7c22-4248-a8d0-7695852ec582\") " pod="openshift-insights/insights-runtime-extractor-dkt6h" Jul 22 12:41:33.924496 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:33.924411 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cfcc578f-7c22-4248-a8d0-7695852ec582-crio-socket\") pod \"insights-runtime-extractor-dkt6h\" (UID: \"cfcc578f-7c22-4248-a8d0-7695852ec582\") " pod="openshift-insights/insights-runtime-extractor-dkt6h" Jul 22 12:41:33.924496 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:33.924433 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cfcc578f-7c22-4248-a8d0-7695852ec582-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dkt6h\" (UID: \"cfcc578f-7c22-4248-a8d0-7695852ec582\") " pod="openshift-insights/insights-runtime-extractor-dkt6h" Jul 22 12:41:33.924626 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:33.924538 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 12:41:33.924626 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:33.924586 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 12:41:33.924626 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:33.924606 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfcc578f-7c22-4248-a8d0-7695852ec582-insights-runtime-extractor-tls podName:cfcc578f-7c22-4248-a8d0-7695852ec582 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:34.424589026 +0000 UTC m=+42.825819945 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cfcc578f-7c22-4248-a8d0-7695852ec582-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dkt6h" (UID: "cfcc578f-7c22-4248-a8d0-7695852ec582") : secret "insights-runtime-extractor-tls" not found Jul 22 12:41:33.924737 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:33.924632 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cfcc578f-7c22-4248-a8d0-7695852ec582-crio-socket\") pod \"insights-runtime-extractor-dkt6h\" (UID: \"cfcc578f-7c22-4248-a8d0-7695852ec582\") " pod="openshift-insights/insights-runtime-extractor-dkt6h" Jul 22 12:41:33.924737 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:33.924645 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cfcc578f-7c22-4248-a8d0-7695852ec582-kube-rbac-proxy-cm podName:cfcc578f-7c22-4248-a8d0-7695852ec582 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:34.424632355 +0000 UTC m=+42.825863275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cfcc578f-7c22-4248-a8d0-7695852ec582-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dkt6h" (UID: "cfcc578f-7c22-4248-a8d0-7695852ec582") : configmap "kube-rbac-proxy" not found Jul 22 12:41:33.924737 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:33.924684 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cfcc578f-7c22-4248-a8d0-7695852ec582-data-volume\") pod \"insights-runtime-extractor-dkt6h\" (UID: \"cfcc578f-7c22-4248-a8d0-7695852ec582\") " pod="openshift-insights/insights-runtime-extractor-dkt6h" Jul 22 12:41:33.946326 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:33.946300 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqf5m\" (UniqueName: \"kubernetes.io/projected/cfcc578f-7c22-4248-a8d0-7695852ec582-kube-api-access-jqf5m\") pod \"insights-runtime-extractor-dkt6h\" (UID: \"cfcc578f-7c22-4248-a8d0-7695852ec582\") " pod="openshift-insights/insights-runtime-extractor-dkt6h" Jul 22 12:41:34.428725 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:34.428686 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cfcc578f-7c22-4248-a8d0-7695852ec582-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dkt6h\" (UID: \"cfcc578f-7c22-4248-a8d0-7695852ec582\") " pod="openshift-insights/insights-runtime-extractor-dkt6h" Jul 22 12:41:34.428725 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:34.428729 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cfcc578f-7c22-4248-a8d0-7695852ec582-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dkt6h\" (UID: \"cfcc578f-7c22-4248-a8d0-7695852ec582\") " pod="openshift-insights/insights-runtime-extractor-dkt6h" Jul 22 12:41:34.428966 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:34.428811 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 12:41:34.428966 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:34.428880 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cfcc578f-7c22-4248-a8d0-7695852ec582-kube-rbac-proxy-cm podName:cfcc578f-7c22-4248-a8d0-7695852ec582 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:35.428865173 +0000 UTC m=+43.830096074 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cfcc578f-7c22-4248-a8d0-7695852ec582-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dkt6h" (UID: "cfcc578f-7c22-4248-a8d0-7695852ec582") : configmap "kube-rbac-proxy" not found Jul 22 12:41:34.428966 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:34.428948 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 12:41:34.429070 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:34.429005 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfcc578f-7c22-4248-a8d0-7695852ec582-insights-runtime-extractor-tls podName:cfcc578f-7c22-4248-a8d0-7695852ec582 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:35.428991945 +0000 UTC m=+43.830222843 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cfcc578f-7c22-4248-a8d0-7695852ec582-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dkt6h" (UID: "cfcc578f-7c22-4248-a8d0-7695852ec582") : secret "insights-runtime-extractor-tls" not found Jul 22 12:41:35.010913 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:35.010882 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vnhnm"] Jul 22 12:41:35.026636 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:35.026614 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vnhnm" Jul 22 12:41:35.029960 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:35.029936 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 22 12:41:35.031188 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:35.031168 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vnhnm"] Jul 22 12:41:35.032126 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:35.032107 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 22 12:41:35.032218 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:35.032135 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-r7kzt\"" Jul 22 12:41:35.070478 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:35.070457 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fgc67_8b80aabc-b7ef-4649-bb58-4e0a39906c4f/dns-node-resolver/0.log" Jul 22 12:41:35.135008 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:35.134975 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2hrg\" (UniqueName: \"kubernetes.io/projected/eb4e1304-4566-4840-93a8-10a898a6f343-kube-api-access-r2hrg\") pod \"migrator-7cc9c7ffd7-vnhnm\" (UID: \"eb4e1304-4566-4840-93a8-10a898a6f343\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vnhnm" Jul 22 12:41:35.235852 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:35.235820 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2hrg\" (UniqueName: \"kubernetes.io/projected/eb4e1304-4566-4840-93a8-10a898a6f343-kube-api-access-r2hrg\") pod \"migrator-7cc9c7ffd7-vnhnm\" (UID: \"eb4e1304-4566-4840-93a8-10a898a6f343\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vnhnm" Jul 22 12:41:35.245550 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:35.245526 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2hrg\" (UniqueName: \"kubernetes.io/projected/eb4e1304-4566-4840-93a8-10a898a6f343-kube-api-access-r2hrg\") pod \"migrator-7cc9c7ffd7-vnhnm\" (UID: \"eb4e1304-4566-4840-93a8-10a898a6f343\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vnhnm" Jul 22 12:41:35.335728 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:35.335696 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vnhnm" Jul 22 12:41:35.438273 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:35.438244 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cfcc578f-7c22-4248-a8d0-7695852ec582-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dkt6h\" (UID: \"cfcc578f-7c22-4248-a8d0-7695852ec582\") " pod="openshift-insights/insights-runtime-extractor-dkt6h" Jul 22 12:41:35.438403 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:35.438283 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cfcc578f-7c22-4248-a8d0-7695852ec582-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dkt6h\" (UID: \"cfcc578f-7c22-4248-a8d0-7695852ec582\") " pod="openshift-insights/insights-runtime-extractor-dkt6h" Jul 22 12:41:35.438403 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:35.438399 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 12:41:35.438487 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:35.438405 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 12:41:35.438487 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:35.438450 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfcc578f-7c22-4248-a8d0-7695852ec582-insights-runtime-extractor-tls podName:cfcc578f-7c22-4248-a8d0-7695852ec582 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:37.438437629 +0000 UTC m=+45.839668527 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cfcc578f-7c22-4248-a8d0-7695852ec582-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dkt6h" (UID: "cfcc578f-7c22-4248-a8d0-7695852ec582") : secret "insights-runtime-extractor-tls" not found Jul 22 12:41:35.438487 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:35.438475 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cfcc578f-7c22-4248-a8d0-7695852ec582-kube-rbac-proxy-cm podName:cfcc578f-7c22-4248-a8d0-7695852ec582 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:37.438456407 +0000 UTC m=+45.839687308 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cfcc578f-7c22-4248-a8d0-7695852ec582-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dkt6h" (UID: "cfcc578f-7c22-4248-a8d0-7695852ec582") : configmap "kube-rbac-proxy" not found Jul 22 12:41:35.490533 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:35.490504 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vnhnm"] Jul 22 12:41:35.493231 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:41:35.493204 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb4e1304_4566_4840_93a8_10a898a6f343.slice/crio-5079d4ec6cc095eb2fe93a42bb453fc06f7091096a1f86129516d84ed6afafe1 WatchSource:0}: Error finding container 5079d4ec6cc095eb2fe93a42bb453fc06f7091096a1f86129516d84ed6afafe1: Status 404 returned error can't find the container with id 5079d4ec6cc095eb2fe93a42bb453fc06f7091096a1f86129516d84ed6afafe1 Jul 22 12:41:36.167712 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:36.167681 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vnhnm" event={"ID":"eb4e1304-4566-4840-93a8-10a898a6f343","Type":"ContainerStarted","Data":"5079d4ec6cc095eb2fe93a42bb453fc06f7091096a1f86129516d84ed6afafe1"} Jul 22 12:41:36.465328 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:36.465261 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w6nhq_c504a9e7-e48d-496d-8a1d-3b307361714c/node-ca/0.log" Jul 22 12:41:36.748723 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:36.748643 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-registry-tls\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:36.748863 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:36.748810 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 12:41:36.748863 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:36.748825 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b47b6758f-f82nc: secret "image-registry-tls" not found Jul 22 12:41:36.748949 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:36.748885 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-registry-tls podName:3febf38a-9737-4258-8621-a343c7803c28 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:44.748866643 +0000 UTC m=+53.150097559 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-registry-tls") pod "image-registry-b47b6758f-f82nc" (UID: "3febf38a-9737-4258-8621-a343c7803c28") : secret "image-registry-tls" not found Jul 22 12:41:36.849792 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:36.849761 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/394708f4-ca0a-4fb2-a3d2-a22224322e3d-metrics-tls\") pod \"dns-default-w8md4\" (UID: \"394708f4-ca0a-4fb2-a3d2-a22224322e3d\") " pod="openshift-dns/dns-default-w8md4" Jul 22 12:41:36.849950 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:36.849879 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 22 12:41:36.849950 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:36.849944 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/394708f4-ca0a-4fb2-a3d2-a22224322e3d-metrics-tls podName:394708f4-ca0a-4fb2-a3d2-a22224322e3d nodeName:}" failed. No retries permitted until 2026-07-22 12:41:44.849916762 +0000 UTC m=+53.251147660 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/394708f4-ca0a-4fb2-a3d2-a22224322e3d-metrics-tls") pod "dns-default-w8md4" (UID: "394708f4-ca0a-4fb2-a3d2-a22224322e3d") : secret "dns-default-metrics-tls" not found Jul 22 12:41:36.864668 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:36.864646 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7q7fk_e9effaf4-801b-4f94-856f-9a934fd06a77/serve-healthcheck-canary/0.log" Jul 22 12:41:37.171434 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:37.171407 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vnhnm" event={"ID":"eb4e1304-4566-4840-93a8-10a898a6f343","Type":"ContainerStarted","Data":"5ec4aace4194805ceac5bcc3925cd1ead73f0adcb2eb1f8631a79c7ee87b4c4b"} Jul 22 12:41:37.454149 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:37.454114 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cfcc578f-7c22-4248-a8d0-7695852ec582-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dkt6h\" (UID: \"cfcc578f-7c22-4248-a8d0-7695852ec582\") " pod="openshift-insights/insights-runtime-extractor-dkt6h" Jul 22 12:41:37.454311 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:37.454159 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cfcc578f-7c22-4248-a8d0-7695852ec582-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dkt6h\" (UID: \"cfcc578f-7c22-4248-a8d0-7695852ec582\") " pod="openshift-insights/insights-runtime-extractor-dkt6h" Jul 22 12:41:37.454311 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:37.454283 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 12:41:37.454399 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:37.454333 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfcc578f-7c22-4248-a8d0-7695852ec582-insights-runtime-extractor-tls podName:cfcc578f-7c22-4248-a8d0-7695852ec582 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:41.454318509 +0000 UTC m=+49.855549406 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cfcc578f-7c22-4248-a8d0-7695852ec582-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dkt6h" (UID: "cfcc578f-7c22-4248-a8d0-7695852ec582") : secret "insights-runtime-extractor-tls" not found Jul 22 12:41:37.454399 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:37.454362 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 22 12:41:37.454474 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:37.454418 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cfcc578f-7c22-4248-a8d0-7695852ec582-kube-rbac-proxy-cm podName:cfcc578f-7c22-4248-a8d0-7695852ec582 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:41.454405667 +0000 UTC m=+49.855636565 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/cfcc578f-7c22-4248-a8d0-7695852ec582-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dkt6h" (UID: "cfcc578f-7c22-4248-a8d0-7695852ec582") : configmap "kube-rbac-proxy" not found Jul 22 12:41:38.178229 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:38.178191 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vnhnm" event={"ID":"eb4e1304-4566-4840-93a8-10a898a6f343","Type":"ContainerStarted","Data":"38da40412175bb07f04ba7db1d5ea359ac91b64dafec4aa5ee3055c9eb45779c"} Jul 22 12:41:38.197630 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:38.197577 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-vnhnm" podStartSLOduration=2.72140802 podStartE2EDuration="4.197563165s" totalImagesPullingTime="1.476155145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:41:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:38.197322497 +0000 UTC m=+46.598553418" watchObservedRunningTime="2026-07-22 12:41:38.197563165 +0000 UTC m=+46.598794084" Jul 22 12:41:41.482602 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:41.482562 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cfcc578f-7c22-4248-a8d0-7695852ec582-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dkt6h\" (UID: \"cfcc578f-7c22-4248-a8d0-7695852ec582\") " pod="openshift-insights/insights-runtime-extractor-dkt6h" Jul 22 12:41:41.482602 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:41.482606 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cfcc578f-7c22-4248-a8d0-7695852ec582-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dkt6h\" (UID: \"cfcc578f-7c22-4248-a8d0-7695852ec582\") " pod="openshift-insights/insights-runtime-extractor-dkt6h" Jul 22 12:41:41.483034 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:41.482695 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 22 12:41:41.483034 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:41.482741 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfcc578f-7c22-4248-a8d0-7695852ec582-insights-runtime-extractor-tls podName:cfcc578f-7c22-4248-a8d0-7695852ec582 nodeName:}" failed. No retries permitted until 2026-07-22 12:41:49.482728909 +0000 UTC m=+57.883959807 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/cfcc578f-7c22-4248-a8d0-7695852ec582-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dkt6h" (UID: "cfcc578f-7c22-4248-a8d0-7695852ec582") : secret "insights-runtime-extractor-tls" not found Jul 22 12:41:41.483296 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:41.483276 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cfcc578f-7c22-4248-a8d0-7695852ec582-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dkt6h\" (UID: \"cfcc578f-7c22-4248-a8d0-7695852ec582\") " pod="openshift-insights/insights-runtime-extractor-dkt6h" Jul 22 12:41:44.803158 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:44.803125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-registry-tls\") pod \"image-registry-b47b6758f-f82nc\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:44.803611 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:44.803250 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 22 12:41:44.803611 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:44.803261 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b47b6758f-f82nc: secret "image-registry-tls" not found Jul 22 12:41:44.803611 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:44.803307 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-registry-tls podName:3febf38a-9737-4258-8621-a343c7803c28 nodeName:}" failed. No retries permitted until 2026-07-22 12:42:00.803294492 +0000 UTC m=+69.204525390 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-registry-tls") pod "image-registry-b47b6758f-f82nc" (UID: "3febf38a-9737-4258-8621-a343c7803c28") : secret "image-registry-tls" not found Jul 22 12:41:44.903598 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:44.903562 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/394708f4-ca0a-4fb2-a3d2-a22224322e3d-metrics-tls\") pod \"dns-default-w8md4\" (UID: \"394708f4-ca0a-4fb2-a3d2-a22224322e3d\") " pod="openshift-dns/dns-default-w8md4" Jul 22 12:41:44.905836 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:44.905802 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/394708f4-ca0a-4fb2-a3d2-a22224322e3d-metrics-tls\") pod \"dns-default-w8md4\" (UID: \"394708f4-ca0a-4fb2-a3d2-a22224322e3d\") " pod="openshift-dns/dns-default-w8md4" Jul 22 12:41:44.911656 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:44.911637 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-w8md4" Jul 22 12:41:45.071914 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:45.071845 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-w8md4"] Jul 22 12:41:45.074996 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:41:45.074958 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394708f4_ca0a_4fb2_a3d2_a22224322e3d.slice/crio-65627f9430cbad0b45ff3fedb73e2526708b25fb36ab42a2a4c468f4090b735d WatchSource:0}: Error finding container 65627f9430cbad0b45ff3fedb73e2526708b25fb36ab42a2a4c468f4090b735d: Status 404 returned error can't find the container with id 65627f9430cbad0b45ff3fedb73e2526708b25fb36ab42a2a4c468f4090b735d Jul 22 12:41:45.194398 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:45.194358 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w8md4" event={"ID":"394708f4-ca0a-4fb2-a3d2-a22224322e3d","Type":"ContainerStarted","Data":"65627f9430cbad0b45ff3fedb73e2526708b25fb36ab42a2a4c468f4090b735d"} Jul 22 12:41:47.201653 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:47.201568 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w8md4" event={"ID":"394708f4-ca0a-4fb2-a3d2-a22224322e3d","Type":"ContainerStarted","Data":"0ca3cd570faeb2c8ff00804aa681c1e039242d4197314471550425afac6f749b"} Jul 22 12:41:47.201653 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:47.201618 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-w8md4" event={"ID":"394708f4-ca0a-4fb2-a3d2-a22224322e3d","Type":"ContainerStarted","Data":"da4135c583a4e0a0005bdcd85f83b88d006f9ab72699692521865039cfbf4dd4"} Jul 22 12:41:47.201653 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:47.201654 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-w8md4" Jul 22 12:41:47.230215 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:47.230170 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-w8md4" podStartSLOduration=17.496403023 podStartE2EDuration="19.230154107s" totalImagesPullingTime="1.733751084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:41:47.229599131 +0000 UTC m=+55.630830050" watchObservedRunningTime="2026-07-22 12:41:47.230154107 +0000 UTC m=+55.631385027" Jul 22 12:41:49.542185 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:49.542146 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cfcc578f-7c22-4248-a8d0-7695852ec582-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dkt6h\" (UID: \"cfcc578f-7c22-4248-a8d0-7695852ec582\") " pod="openshift-insights/insights-runtime-extractor-dkt6h" Jul 22 12:41:49.544388 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:49.544358 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cfcc578f-7c22-4248-a8d0-7695852ec582-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dkt6h\" (UID: \"cfcc578f-7c22-4248-a8d0-7695852ec582\") " pod="openshift-insights/insights-runtime-extractor-dkt6h" Jul 22 12:41:49.636533 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:49.636500 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dkt6h" Jul 22 12:41:49.762694 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:49.762638 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dkt6h"] Jul 22 12:41:49.765250 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:41:49.765218 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfcc578f_7c22_4248_a8d0_7695852ec582.slice/crio-963ce0f994af7682085cfdfde939ea392709d8d92629d8ce28db69a9f1ea4f48 WatchSource:0}: Error finding container 963ce0f994af7682085cfdfde939ea392709d8d92629d8ce28db69a9f1ea4f48: Status 404 returned error can't find the container with id 963ce0f994af7682085cfdfde939ea392709d8d92629d8ce28db69a9f1ea4f48 Jul 22 12:41:50.214152 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:50.214115 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dkt6h" event={"ID":"cfcc578f-7c22-4248-a8d0-7695852ec582","Type":"ContainerStarted","Data":"963ce0f994af7682085cfdfde939ea392709d8d92629d8ce28db69a9f1ea4f48"} Jul 22 12:41:55.233296 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:55.233253 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dkt6h" event={"ID":"cfcc578f-7c22-4248-a8d0-7695852ec582","Type":"ContainerStarted","Data":"08f0075ba1dcfeea4ca7dcf2df789a8a79a9cf136c8de83dc837c6b6bd36e94e"} Jul 22 12:41:56.141554 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:56.141523 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ddj6" Jul 22 12:41:57.208845 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:57.208805 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-w8md4" Jul 22 12:41:58.245255 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:58.245168 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dkt6h" event={"ID":"cfcc578f-7c22-4248-a8d0-7695852ec582","Type":"ContainerStarted","Data":"041aec5f80f587f286faeee2f9762db8536698be2e2bbbd5ecf48cc769585443"} Jul 22 12:41:58.908081 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:58.908050 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b47b6758f-f82nc"] Jul 22 12:41:58.908283 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:41:58.908261 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-b47b6758f-f82nc" podUID="3febf38a-9737-4258-8621-a343c7803c28" Jul 22 12:41:59.248871 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.248784 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:59.272608 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.272577 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:41:59.322603 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.322573 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmdcz\" (UniqueName: \"kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-kube-api-access-dmdcz\") pod \"3febf38a-9737-4258-8621-a343c7803c28\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " Jul 22 12:41:59.322761 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.322616 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3febf38a-9737-4258-8621-a343c7803c28-image-registry-private-configuration\") pod \"3febf38a-9737-4258-8621-a343c7803c28\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " Jul 22 12:41:59.322761 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.322656 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3febf38a-9737-4258-8621-a343c7803c28-ca-trust-extracted\") pod \"3febf38a-9737-4258-8621-a343c7803c28\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " Jul 22 12:41:59.322761 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.322716 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3febf38a-9737-4258-8621-a343c7803c28-installation-pull-secrets\") pod \"3febf38a-9737-4258-8621-a343c7803c28\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " Jul 22 12:41:59.322919 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.322766 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3febf38a-9737-4258-8621-a343c7803c28-trusted-ca\") pod \"3febf38a-9737-4258-8621-a343c7803c28\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " Jul 22 12:41:59.322919 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.322793 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-bound-sa-token\") pod \"3febf38a-9737-4258-8621-a343c7803c28\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " Jul 22 12:41:59.322919 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.322835 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3febf38a-9737-4258-8621-a343c7803c28-registry-certificates\") pod \"3febf38a-9737-4258-8621-a343c7803c28\" (UID: \"3febf38a-9737-4258-8621-a343c7803c28\") " Jul 22 12:41:59.323170 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.323024 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3febf38a-9737-4258-8621-a343c7803c28-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "3febf38a-9737-4258-8621-a343c7803c28" (UID: "3febf38a-9737-4258-8621-a343c7803c28"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:41:59.323379 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.323330 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3febf38a-9737-4258-8621-a343c7803c28-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "3febf38a-9737-4258-8621-a343c7803c28" (UID: "3febf38a-9737-4258-8621-a343c7803c28"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:41:59.323379 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.323348 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3febf38a-9737-4258-8621-a343c7803c28-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "3febf38a-9737-4258-8621-a343c7803c28" (UID: "3febf38a-9737-4258-8621-a343c7803c28"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:41:59.325396 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.325359 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "3febf38a-9737-4258-8621-a343c7803c28" (UID: "3febf38a-9737-4258-8621-a343c7803c28"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:41:59.325396 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.325373 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3febf38a-9737-4258-8621-a343c7803c28-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "3febf38a-9737-4258-8621-a343c7803c28" (UID: "3febf38a-9737-4258-8621-a343c7803c28"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:41:59.325559 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.325432 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3febf38a-9737-4258-8621-a343c7803c28-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "3febf38a-9737-4258-8621-a343c7803c28" (UID: "3febf38a-9737-4258-8621-a343c7803c28"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:41:59.325559 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.325511 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-kube-api-access-dmdcz" (OuterVolumeSpecName: "kube-api-access-dmdcz") pod "3febf38a-9737-4258-8621-a343c7803c28" (UID: "3febf38a-9737-4258-8621-a343c7803c28"). InnerVolumeSpecName "kube-api-access-dmdcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:41:59.423889 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.423860 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3febf38a-9737-4258-8621-a343c7803c28-trusted-ca\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:41:59.423889 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.423889 2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-bound-sa-token\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:41:59.424110 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.423903 2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3febf38a-9737-4258-8621-a343c7803c28-registry-certificates\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:41:59.424110 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.423916 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmdcz\" (UniqueName: \"kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-kube-api-access-dmdcz\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:41:59.424110 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.423943 2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3febf38a-9737-4258-8621-a343c7803c28-image-registry-private-configuration\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:41:59.424110 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.423958 2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3febf38a-9737-4258-8621-a343c7803c28-ca-trust-extracted\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:41:59.424110 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.423967 2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3febf38a-9737-4258-8621-a343c7803c28-installation-pull-secrets\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:41:59.827407 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.827368 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28d7c937-a848-4d26-861f-29f9d2b0baf7-original-pull-secret\") pod \"global-pull-secret-syncer-7gsw5\" (UID: \"28d7c937-a848-4d26-861f-29f9d2b0baf7\") " pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:41:59.827582 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.827492 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c54132f-3e4d-427b-b38d-cd0daabb33d9-metrics-certs\") pod \"network-metrics-daemon-jch9g\" (UID: \"3c54132f-3e4d-427b-b38d-cd0daabb33d9\") " pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:41:59.830445 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.830416 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 22 12:41:59.830561 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.830424 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 22 12:41:59.840510 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.840469 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/28d7c937-a848-4d26-861f-29f9d2b0baf7-original-pull-secret\") pod \"global-pull-secret-syncer-7gsw5\" (UID: \"28d7c937-a848-4d26-861f-29f9d2b0baf7\") " pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:41:59.840626 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.840528 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c54132f-3e4d-427b-b38d-cd0daabb33d9-metrics-certs\") pod \"network-metrics-daemon-jch9g\" (UID: \"3c54132f-3e4d-427b-b38d-cd0daabb33d9\") " pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:41:59.928103 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.928060 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxmtx\" (UniqueName: \"kubernetes.io/projected/9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed-kube-api-access-hxmtx\") pod \"network-check-target-6ghfv\" (UID: \"9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed\") " pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:41:59.931403 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.931375 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 22 12:41:59.941420 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.941397 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 22 12:41:59.952039 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.952015 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxmtx\" (UniqueName: \"kubernetes.io/projected/9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed-kube-api-access-hxmtx\") pod \"network-check-target-6ghfv\" (UID: \"9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed\") " pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:41:59.977214 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.977194 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sv8qm\"" Jul 22 12:41:59.979304 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.979289 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7gsw5" Jul 22 12:41:59.983969 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.983956 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:41:59.986652 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.986637 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-72nf2\"" Jul 22 12:41:59.993817 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:41:59.993803 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jch9g" Jul 22 12:42:00.252236 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:00.252125 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b47b6758f-f82nc" Jul 22 12:42:00.292524 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:00.292492 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b47b6758f-f82nc"] Jul 22 12:42:00.302157 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:00.302125 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-b47b6758f-f82nc"] Jul 22 12:42:00.330746 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:00.330719 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3febf38a-9737-4258-8621-a343c7803c28-registry-tls\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:42:01.787724 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:01.787587 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jch9g"] Jul 22 12:42:01.793555 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:42:01.793525 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c54132f_3e4d_427b_b38d_cd0daabb33d9.slice/crio-bb8f164627070faa47b0c9435c976e9dd77fc3f5ea4d840d45093f7567dc1017 WatchSource:0}: Error finding container bb8f164627070faa47b0c9435c976e9dd77fc3f5ea4d840d45093f7567dc1017: Status 404 returned error can't find the container with id bb8f164627070faa47b0c9435c976e9dd77fc3f5ea4d840d45093f7567dc1017 Jul 22 12:42:01.800418 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:01.800394 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7gsw5"] Jul 22 12:42:01.804405 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:42:01.804378 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28d7c937_a848_4d26_861f_29f9d2b0baf7.slice/crio-4ff4a9913c3380dad9fe34e83e83a0649ce440e645733a3368fdd21080f006e7 WatchSource:0}: Error finding container 4ff4a9913c3380dad9fe34e83e83a0649ce440e645733a3368fdd21080f006e7: Status 404 returned error can't find the container with id 4ff4a9913c3380dad9fe34e83e83a0649ce440e645733a3368fdd21080f006e7 Jul 22 12:42:01.816845 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:01.816818 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6ghfv"] Jul 22 12:42:01.819499 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:42:01.819478 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce1b3d5_d39a_4bcb_8e36_431eb89f19ed.slice/crio-eeeeccdc8cb30b912d261589f250d7d4e6f396fc91b13229fbec91070e3ec852 WatchSource:0}: Error finding container eeeeccdc8cb30b912d261589f250d7d4e6f396fc91b13229fbec91070e3ec852: Status 404 returned error can't find the container with id eeeeccdc8cb30b912d261589f250d7d4e6f396fc91b13229fbec91070e3ec852 Jul 22 12:42:02.065398 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:02.065368 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3febf38a-9737-4258-8621-a343c7803c28" path="/var/lib/kubelet/pods/3febf38a-9737-4258-8621-a343c7803c28/volumes" Jul 22 12:42:02.261402 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:02.261349 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dkt6h" event={"ID":"cfcc578f-7c22-4248-a8d0-7695852ec582","Type":"ContainerStarted","Data":"21d97b350b165791572c6b26a555e852feff4bde395316e45bab54e8af608171"} Jul 22 12:42:02.262750 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:02.262720 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6ghfv" event={"ID":"9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed","Type":"ContainerStarted","Data":"eeeeccdc8cb30b912d261589f250d7d4e6f396fc91b13229fbec91070e3ec852"} Jul 22 12:42:02.263937 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:02.263905 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jch9g" event={"ID":"3c54132f-3e4d-427b-b38d-cd0daabb33d9","Type":"ContainerStarted","Data":"bb8f164627070faa47b0c9435c976e9dd77fc3f5ea4d840d45093f7567dc1017"} Jul 22 12:42:02.264983 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:02.264959 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7gsw5" event={"ID":"28d7c937-a848-4d26-861f-29f9d2b0baf7","Type":"ContainerStarted","Data":"4ff4a9913c3380dad9fe34e83e83a0649ce440e645733a3368fdd21080f006e7"} Jul 22 12:42:02.282504 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:02.282461 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dkt6h" podStartSLOduration=17.445551872 podStartE2EDuration="29.282448136s" totalImagesPullingTime="11.836896264s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:41:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:02.282327775 +0000 UTC m=+70.683558695" watchObservedRunningTime="2026-07-22 12:42:02.282448136 +0000 UTC m=+70.683679055" Jul 22 12:42:03.273630 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:03.273596 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jch9g" event={"ID":"3c54132f-3e4d-427b-b38d-cd0daabb33d9","Type":"ContainerStarted","Data":"0cd5b435b322beb762932761ff5f7df2d61812eb8f0288210cd4292aafa15945"} Jul 22 12:42:03.273949 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:03.273656 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jch9g" event={"ID":"3c54132f-3e4d-427b-b38d-cd0daabb33d9","Type":"ContainerStarted","Data":"a5e117eff81e397a1157f8ccae5bbfc028645e5efc902b64a6dcba92a45effee"} Jul 22 12:42:03.295804 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:03.295565 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jch9g" podStartSLOduration=70.043742833 podStartE2EDuration="1m11.295543705s" totalImagesPullingTime="1.251800872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:03.293846317 +0000 UTC m=+71.695077228" watchObservedRunningTime="2026-07-22 12:42:03.295543705 +0000 UTC m=+71.696774626" Jul 22 12:42:05.692889 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:05.692825 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dptcc"] Jul 22 12:42:05.696200 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:05.696173 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dptcc" Jul 22 12:42:05.699709 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:05.699539 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 22 12:42:05.699709 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:05.699585 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vtfsw\"" Jul 22 12:42:05.713285 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:05.713259 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dptcc"] Jul 22 12:42:05.773250 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:05.773218 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1713bba6-d614-4d6b-89be-4880d38a9f29-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-dptcc\" (UID: \"1713bba6-d614-4d6b-89be-4880d38a9f29\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dptcc" Jul 22 12:42:05.873801 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:05.873768 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1713bba6-d614-4d6b-89be-4880d38a9f29-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-dptcc\" (UID: \"1713bba6-d614-4d6b-89be-4880d38a9f29\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dptcc" Jul 22 12:42:05.876496 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:05.876471 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1713bba6-d614-4d6b-89be-4880d38a9f29-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-dptcc\" (UID: \"1713bba6-d614-4d6b-89be-4880d38a9f29\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dptcc" Jul 22 12:42:06.007622 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:06.007537 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dptcc" Jul 22 12:42:06.430815 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:06.430787 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dptcc"] Jul 22 12:42:06.433443 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:42:06.433420 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1713bba6_d614_4d6b_89be_4880d38a9f29.slice/crio-c798d121d6bb78bd0b7b4f22bf95b733589d6e5dacef326b92fd500221e3f686 WatchSource:0}: Error finding container c798d121d6bb78bd0b7b4f22bf95b733589d6e5dacef326b92fd500221e3f686: Status 404 returned error can't find the container with id c798d121d6bb78bd0b7b4f22bf95b733589d6e5dacef326b92fd500221e3f686 Jul 22 12:42:07.289229 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:07.289157 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7gsw5" event={"ID":"28d7c937-a848-4d26-861f-29f9d2b0baf7","Type":"ContainerStarted","Data":"43d0b46c7929b77e807c4df7c3e54932f1dbe7f2b7a8da050296a3f03acb5d06"} Jul 22 12:42:07.290589 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:07.290563 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dptcc" event={"ID":"1713bba6-d614-4d6b-89be-4880d38a9f29","Type":"ContainerStarted","Data":"c798d121d6bb78bd0b7b4f22bf95b733589d6e5dacef326b92fd500221e3f686"} Jul 22 12:42:07.292165 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:07.292140 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6ghfv" event={"ID":"9ce1b3d5-d39a-4bcb-8e36-431eb89f19ed","Type":"ContainerStarted","Data":"e482c498b5a1cb35422c9f5402559c62087384827233b67c499a0cc68d1712ec"} Jul 22 12:42:07.292297 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:07.292281 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:42:07.306165 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:07.306098 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7gsw5" podStartSLOduration=69.789188646 podStartE2EDuration="1m14.306048975s" totalImagesPullingTime="4.516860329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 12:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:07.305982667 +0000 UTC m=+75.707213599" watchObservedRunningTime="2026-07-22 12:42:07.306048975 +0000 UTC m=+75.707279896" Jul 22 12:42:07.323049 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:07.323009 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6ghfv" podStartSLOduration=70.829527855 podStartE2EDuration="1m15.32299659s" totalImagesPullingTime="4.493468735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:40:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:07.322916465 +0000 UTC m=+75.724147386" watchObservedRunningTime="2026-07-22 12:42:07.32299659 +0000 UTC m=+75.724227510" Jul 22 12:42:08.296869 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:08.296819 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dptcc" event={"ID":"1713bba6-d614-4d6b-89be-4880d38a9f29","Type":"ContainerStarted","Data":"60af2b4beb5ac0b7b057c2c5e68dfd414eafa75c8a776e1f8d3813f4a52f6fcf"} Jul 22 12:42:08.297335 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:08.297206 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dptcc" Jul 22 12:42:08.301707 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:08.301688 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dptcc" Jul 22 12:42:08.314495 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:08.314452 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-dptcc" podStartSLOduration=2.021427323 podStartE2EDuration="3.314440017s" totalImagesPullingTime="1.293012694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:08.313108741 +0000 UTC m=+76.714339663" watchObservedRunningTime="2026-07-22 12:42:08.314440017 +0000 UTC m=+76.715670936" Jul 22 12:42:08.779253 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:08.779171 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-llw8h"] Jul 22 12:42:08.782220 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:08.782204 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-llw8h" Jul 22 12:42:08.788084 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:08.788064 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 22 12:42:08.788673 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:08.788381 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vt5nn\"" Jul 22 12:42:08.788673 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:08.788422 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 22 12:42:08.788846 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:08.788830 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 22 12:42:08.789272 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:08.789256 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 22 12:42:08.789515 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:08.789500 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 22 12:42:08.799475 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:08.799454 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-llw8h"] Jul 22 12:42:08.894443 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:08.894393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-llw8h\" (UID: \"379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-llw8h" Jul 22 12:42:08.894586 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:08.894473 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-llw8h\" (UID: \"379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-llw8h" Jul 22 12:42:08.894586 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:08.894520 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5jc4\" (UniqueName: \"kubernetes.io/projected/379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d-kube-api-access-p5jc4\") pod \"prometheus-operator-5945b5cfd-llw8h\" (UID: \"379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-llw8h" Jul 22 12:42:08.894586 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:08.894559 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-llw8h\" (UID: \"379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-llw8h" Jul 22 12:42:08.995349 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:08.995320 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-llw8h\" (UID: \"379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-llw8h" Jul 22 12:42:08.995488 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:08.995354 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-llw8h\" (UID: \"379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-llw8h" Jul 22 12:42:08.995488 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:08.995384 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5jc4\" (UniqueName: \"kubernetes.io/projected/379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d-kube-api-access-p5jc4\") pod \"prometheus-operator-5945b5cfd-llw8h\" (UID: \"379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-llw8h" Jul 22 12:42:08.995488 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:08.995408 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-llw8h\" (UID: \"379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-llw8h" Jul 22 12:42:08.995581 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:42:08.995489 2579 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 22 12:42:08.995581 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:42:08.995547 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d-prometheus-operator-tls podName:379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d nodeName:}" failed. No retries permitted until 2026-07-22 12:42:09.495529093 +0000 UTC m=+77.896759992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-llw8h" (UID: "379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d") : secret "prometheus-operator-tls" not found Jul 22 12:42:08.996184 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:08.996164 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-llw8h\" (UID: \"379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-llw8h" Jul 22 12:42:08.997772 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:08.997748 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-llw8h\" (UID: \"379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-llw8h" Jul 22 12:42:09.006379 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:09.006357 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5jc4\" (UniqueName: \"kubernetes.io/projected/379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d-kube-api-access-p5jc4\") pod \"prometheus-operator-5945b5cfd-llw8h\" (UID: \"379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-llw8h" Jul 22 12:42:09.498847 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:09.498807 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-llw8h\" (UID: \"379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-llw8h" Jul 22 12:42:09.501188 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:09.501163 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-llw8h\" (UID: \"379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-llw8h" Jul 22 12:42:09.690976 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:09.690943 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-llw8h" Jul 22 12:42:09.814757 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:09.814732 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-llw8h"] Jul 22 12:42:09.817188 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:42:09.817165 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379c5fcb_5fc6_44d5_88b1_d0ab4f6e1f6d.slice/crio-bc4fc323f2bdcabf07d83b58ed1bf8b8feb88fdea9c3476c71467f4d6154eafd WatchSource:0}: Error finding container bc4fc323f2bdcabf07d83b58ed1bf8b8feb88fdea9c3476c71467f4d6154eafd: Status 404 returned error can't find the container with id bc4fc323f2bdcabf07d83b58ed1bf8b8feb88fdea9c3476c71467f4d6154eafd Jul 22 12:42:10.305973 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:10.305939 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-llw8h" event={"ID":"379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d","Type":"ContainerStarted","Data":"bc4fc323f2bdcabf07d83b58ed1bf8b8feb88fdea9c3476c71467f4d6154eafd"} Jul 22 12:42:11.312076 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:11.312037 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-llw8h" event={"ID":"379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d","Type":"ContainerStarted","Data":"89fd3e53d4a54a4a122ea3d350d2ff99a25af4095e445eeb28a1bbb3ca5036d0"} Jul 22 12:42:11.312515 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:11.312106 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-llw8h" event={"ID":"379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d","Type":"ContainerStarted","Data":"619b8e8016b3b3691363ac932994f81eee82b8c1542ff8d9f3bfeb91072792e3"} Jul 22 12:42:11.347837 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:11.347274 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-llw8h" podStartSLOduration=2.060089457 podStartE2EDuration="3.347254619s" totalImagesPullingTime="1.287165162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:42:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:11.34716591 +0000 UTC m=+79.748396831" watchObservedRunningTime="2026-07-22 12:42:11.347254619 +0000 UTC m=+79.748485539" Jul 22 12:42:13.176973 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.176919 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-jnzn4"] Jul 22 12:42:13.179995 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.179979 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-jnzn4" Jul 22 12:42:13.183542 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.183519 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-p9q2j\"" Jul 22 12:42:13.183668 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.183564 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 22 12:42:13.183668 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.183584 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 22 12:42:13.198008 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.197983 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-jnzn4"] Jul 22 12:42:13.199889 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.199870 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-75nsf"] Jul 22 12:42:13.203153 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.203135 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-75nsf" Jul 22 12:42:13.206251 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.206233 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-89qvp\"" Jul 22 12:42:13.206522 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.206503 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 22 12:42:13.206606 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.206542 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 22 12:42:13.207300 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.207284 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 22 12:42:13.217077 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.217058 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-48jh9"] Jul 22 12:42:13.220209 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.220192 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.221479 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.221460 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-75nsf"] Jul 22 12:42:13.222497 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.222476 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cf037ef-07c5-4294-933e-6c61ef336250-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-jnzn4\" (UID: \"3cf037ef-07c5-4294-933e-6c61ef336250\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-jnzn4" Jul 22 12:42:13.222591 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.222511 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cf037ef-07c5-4294-933e-6c61ef336250-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-jnzn4\" (UID: \"3cf037ef-07c5-4294-933e-6c61ef336250\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-jnzn4" Jul 22 12:42:13.222591 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.222550 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xb2r\" (UniqueName: \"kubernetes.io/projected/3cf037ef-07c5-4294-933e-6c61ef336250-kube-api-access-2xb2r\") pod \"openshift-state-metrics-79f6958cdf-jnzn4\" (UID: \"3cf037ef-07c5-4294-933e-6c61ef336250\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-jnzn4" Jul 22 12:42:13.222703 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.222613 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cf037ef-07c5-4294-933e-6c61ef336250-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-jnzn4\" (UID: \"3cf037ef-07c5-4294-933e-6c61ef336250\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-jnzn4" Jul 22 12:42:13.222770 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.222748 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 22 12:42:13.223405 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.223384 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2xzrv\"" Jul 22 12:42:13.223549 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.223529 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 22 12:42:13.223959 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.223938 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 22 12:42:13.323219 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.323184 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cf037ef-07c5-4294-933e-6c61ef336250-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-jnzn4\" (UID: \"3cf037ef-07c5-4294-933e-6c61ef336250\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-jnzn4" Jul 22 12:42:13.323219 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.323222 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cf037ef-07c5-4294-933e-6c61ef336250-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-jnzn4\" (UID: \"3cf037ef-07c5-4294-933e-6c61ef336250\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-jnzn4" Jul 22 12:42:13.323428 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.323242 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cf037ef-07c5-4294-933e-6c61ef336250-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-jnzn4\" (UID: \"3cf037ef-07c5-4294-933e-6c61ef336250\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-jnzn4" Jul 22 12:42:13.323428 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.323272 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c98bde4d-82db-4b9d-969a-892fc9ef54f2-metrics-client-ca\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.323428 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.323299 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42f58d4d-89d6-46e4-8598-0a428e9ed439-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-75nsf\" (UID: \"42f58d4d-89d6-46e4-8598-0a428e9ed439\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75nsf" Jul 22 12:42:13.323428 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.323325 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4jjt\" (UniqueName: \"kubernetes.io/projected/42f58d4d-89d6-46e4-8598-0a428e9ed439-kube-api-access-r4jjt\") pod \"kube-state-metrics-56c5594674-75nsf\" (UID: \"42f58d4d-89d6-46e4-8598-0a428e9ed439\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75nsf" Jul 22 12:42:13.323428 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.323351 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c98bde4d-82db-4b9d-969a-892fc9ef54f2-root\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.323428 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.323415 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/42f58d4d-89d6-46e4-8598-0a428e9ed439-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-75nsf\" (UID: \"42f58d4d-89d6-46e4-8598-0a428e9ed439\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75nsf" Jul 22 12:42:13.323636 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.323508 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c98bde4d-82db-4b9d-969a-892fc9ef54f2-sys\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.323636 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.323583 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c98bde4d-82db-4b9d-969a-892fc9ef54f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.323636 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.323617 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/42f58d4d-89d6-46e4-8598-0a428e9ed439-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-75nsf\" (UID: \"42f58d4d-89d6-46e4-8598-0a428e9ed439\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75nsf" Jul 22 12:42:13.323784 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.323666 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xb2r\" (UniqueName: \"kubernetes.io/projected/3cf037ef-07c5-4294-933e-6c61ef336250-kube-api-access-2xb2r\") pod \"openshift-state-metrics-79f6958cdf-jnzn4\" (UID: \"3cf037ef-07c5-4294-933e-6c61ef336250\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-jnzn4" Jul 22 12:42:13.323784 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.323700 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c98bde4d-82db-4b9d-969a-892fc9ef54f2-node-exporter-tls\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.323784 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.323762 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/42f58d4d-89d6-46e4-8598-0a428e9ed439-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-75nsf\" (UID: \"42f58d4d-89d6-46e4-8598-0a428e9ed439\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75nsf" Jul 22 12:42:13.323945 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.323791 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c98bde4d-82db-4b9d-969a-892fc9ef54f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.323945 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.323820 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff589\" (UniqueName: \"kubernetes.io/projected/c98bde4d-82db-4b9d-969a-892fc9ef54f2-kube-api-access-ff589\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.323945 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.323845 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c98bde4d-82db-4b9d-969a-892fc9ef54f2-node-exporter-textfile\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.323945 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.323871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42f58d4d-89d6-46e4-8598-0a428e9ed439-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-75nsf\" (UID: \"42f58d4d-89d6-46e4-8598-0a428e9ed439\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75nsf" Jul 22 12:42:13.323945 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.323918 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c98bde4d-82db-4b9d-969a-892fc9ef54f2-node-exporter-wtmp\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.324201 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.324003 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3cf037ef-07c5-4294-933e-6c61ef336250-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-jnzn4\" (UID: \"3cf037ef-07c5-4294-933e-6c61ef336250\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-jnzn4" Jul 22 12:42:13.325621 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.325602 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3cf037ef-07c5-4294-933e-6c61ef336250-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-jnzn4\" (UID: \"3cf037ef-07c5-4294-933e-6c61ef336250\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-jnzn4" Jul 22 12:42:13.325735 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.325720 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3cf037ef-07c5-4294-933e-6c61ef336250-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-jnzn4\" (UID: \"3cf037ef-07c5-4294-933e-6c61ef336250\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-jnzn4" Jul 22 12:42:13.333640 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.333618 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xb2r\" (UniqueName: \"kubernetes.io/projected/3cf037ef-07c5-4294-933e-6c61ef336250-kube-api-access-2xb2r\") pod \"openshift-state-metrics-79f6958cdf-jnzn4\" (UID: \"3cf037ef-07c5-4294-933e-6c61ef336250\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-jnzn4" Jul 22 12:42:13.424726 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.424678 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42f58d4d-89d6-46e4-8598-0a428e9ed439-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-75nsf\" (UID: \"42f58d4d-89d6-46e4-8598-0a428e9ed439\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75nsf" Jul 22 12:42:13.424726 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.424728 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4jjt\" (UniqueName: \"kubernetes.io/projected/42f58d4d-89d6-46e4-8598-0a428e9ed439-kube-api-access-r4jjt\") pod \"kube-state-metrics-56c5594674-75nsf\" (UID: \"42f58d4d-89d6-46e4-8598-0a428e9ed439\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75nsf" Jul 22 12:42:13.424980 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.424755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c98bde4d-82db-4b9d-969a-892fc9ef54f2-root\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.424980 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.424810 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c98bde4d-82db-4b9d-969a-892fc9ef54f2-root\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.424980 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.424814 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/42f58d4d-89d6-46e4-8598-0a428e9ed439-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-75nsf\" (UID: \"42f58d4d-89d6-46e4-8598-0a428e9ed439\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75nsf" Jul 22 12:42:13.424980 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.424861 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c98bde4d-82db-4b9d-969a-892fc9ef54f2-sys\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.424980 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.424898 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c98bde4d-82db-4b9d-969a-892fc9ef54f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.424980 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.424959 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/42f58d4d-89d6-46e4-8598-0a428e9ed439-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-75nsf\" (UID: \"42f58d4d-89d6-46e4-8598-0a428e9ed439\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75nsf" Jul 22 12:42:13.424980 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.424974 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c98bde4d-82db-4b9d-969a-892fc9ef54f2-sys\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.425315 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.425004 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c98bde4d-82db-4b9d-969a-892fc9ef54f2-node-exporter-tls\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.425315 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.425034 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/42f58d4d-89d6-46e4-8598-0a428e9ed439-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-75nsf\" (UID: \"42f58d4d-89d6-46e4-8598-0a428e9ed439\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75nsf" Jul 22 12:42:13.425315 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.425065 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c98bde4d-82db-4b9d-969a-892fc9ef54f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.425315 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.425098 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff589\" (UniqueName: \"kubernetes.io/projected/c98bde4d-82db-4b9d-969a-892fc9ef54f2-kube-api-access-ff589\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.425315 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:42:13.425105 2579 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 22 12:42:13.425315 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:42:13.425185 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42f58d4d-89d6-46e4-8598-0a428e9ed439-kube-state-metrics-tls podName:42f58d4d-89d6-46e4-8598-0a428e9ed439 nodeName:}" failed. No retries permitted until 2026-07-22 12:42:13.925164512 +0000 UTC m=+82.326395413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/42f58d4d-89d6-46e4-8598-0a428e9ed439-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-75nsf" (UID: "42f58d4d-89d6-46e4-8598-0a428e9ed439") : secret "kube-state-metrics-tls" not found Jul 22 12:42:13.425315 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:42:13.425215 2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 22 12:42:13.425315 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:42:13.425250 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c98bde4d-82db-4b9d-969a-892fc9ef54f2-node-exporter-tls podName:c98bde4d-82db-4b9d-969a-892fc9ef54f2 nodeName:}" failed. No retries permitted until 2026-07-22 12:42:13.925238986 +0000 UTC m=+82.326469893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c98bde4d-82db-4b9d-969a-892fc9ef54f2-node-exporter-tls") pod "node-exporter-48jh9" (UID: "c98bde4d-82db-4b9d-969a-892fc9ef54f2") : secret "node-exporter-tls" not found Jul 22 12:42:13.425315 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.425302 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c98bde4d-82db-4b9d-969a-892fc9ef54f2-node-exporter-textfile\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.425812 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.425333 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42f58d4d-89d6-46e4-8598-0a428e9ed439-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-75nsf\" (UID: \"42f58d4d-89d6-46e4-8598-0a428e9ed439\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75nsf" Jul 22 12:42:13.425812 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.425368 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c98bde4d-82db-4b9d-969a-892fc9ef54f2-node-exporter-wtmp\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.425812 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.425415 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c98bde4d-82db-4b9d-969a-892fc9ef54f2-metrics-client-ca\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.425812 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.425412 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/42f58d4d-89d6-46e4-8598-0a428e9ed439-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-75nsf\" (UID: \"42f58d4d-89d6-46e4-8598-0a428e9ed439\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75nsf" Jul 22 12:42:13.425812 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.425539 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c98bde4d-82db-4b9d-969a-892fc9ef54f2-node-exporter-wtmp\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.425812 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.425589 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42f58d4d-89d6-46e4-8598-0a428e9ed439-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-75nsf\" (UID: \"42f58d4d-89d6-46e4-8598-0a428e9ed439\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75nsf" Jul 22 12:42:13.425812 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.425609 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c98bde4d-82db-4b9d-969a-892fc9ef54f2-node-exporter-textfile\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.425812 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.425757 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c98bde4d-82db-4b9d-969a-892fc9ef54f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.426095 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.425890 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c98bde4d-82db-4b9d-969a-892fc9ef54f2-metrics-client-ca\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.426095 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.426012 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/42f58d4d-89d6-46e4-8598-0a428e9ed439-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-75nsf\" (UID: \"42f58d4d-89d6-46e4-8598-0a428e9ed439\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75nsf" Jul 22 12:42:13.427364 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.427315 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c98bde4d-82db-4b9d-969a-892fc9ef54f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.427463 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.427444 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42f58d4d-89d6-46e4-8598-0a428e9ed439-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-75nsf\" (UID: \"42f58d4d-89d6-46e4-8598-0a428e9ed439\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75nsf" Jul 22 12:42:13.436113 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.436090 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff589\" (UniqueName: \"kubernetes.io/projected/c98bde4d-82db-4b9d-969a-892fc9ef54f2-kube-api-access-ff589\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.436577 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.436557 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4jjt\" (UniqueName: \"kubernetes.io/projected/42f58d4d-89d6-46e4-8598-0a428e9ed439-kube-api-access-r4jjt\") pod \"kube-state-metrics-56c5594674-75nsf\" (UID: \"42f58d4d-89d6-46e4-8598-0a428e9ed439\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75nsf" Jul 22 12:42:13.488488 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.488466 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-jnzn4" Jul 22 12:42:13.642408 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.642337 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-jnzn4"] Jul 22 12:42:13.644853 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:42:13.644822 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf037ef_07c5_4294_933e_6c61ef336250.slice/crio-40fd00872a6b503d7ef2fbf0dc956dd6bb88b4385f9bde0840f4d93dfffd253b WatchSource:0}: Error finding container 40fd00872a6b503d7ef2fbf0dc956dd6bb88b4385f9bde0840f4d93dfffd253b: Status 404 returned error can't find the container with id 40fd00872a6b503d7ef2fbf0dc956dd6bb88b4385f9bde0840f4d93dfffd253b Jul 22 12:42:13.930221 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.930184 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/42f58d4d-89d6-46e4-8598-0a428e9ed439-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-75nsf\" (UID: \"42f58d4d-89d6-46e4-8598-0a428e9ed439\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75nsf" Jul 22 12:42:13.930381 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.930231 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c98bde4d-82db-4b9d-969a-892fc9ef54f2-node-exporter-tls\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:13.930381 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:42:13.930353 2579 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 22 12:42:13.930450 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:42:13.930430 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42f58d4d-89d6-46e4-8598-0a428e9ed439-kube-state-metrics-tls podName:42f58d4d-89d6-46e4-8598-0a428e9ed439 nodeName:}" failed. No retries permitted until 2026-07-22 12:42:14.930410373 +0000 UTC m=+83.331641275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/42f58d4d-89d6-46e4-8598-0a428e9ed439-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-75nsf" (UID: "42f58d4d-89d6-46e4-8598-0a428e9ed439") : secret "kube-state-metrics-tls" not found Jul 22 12:42:13.932537 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:13.932519 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c98bde4d-82db-4b9d-969a-892fc9ef54f2-node-exporter-tls\") pod \"node-exporter-48jh9\" (UID: \"c98bde4d-82db-4b9d-969a-892fc9ef54f2\") " pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:14.129650 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:14.129620 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-48jh9" Jul 22 12:42:14.139654 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:42:14.139631 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc98bde4d_82db_4b9d_969a_892fc9ef54f2.slice/crio-32e797ebe0c353a8fd4ea28fc218c4236575a8949d4128e26d393ab0e8b92cda WatchSource:0}: Error finding container 32e797ebe0c353a8fd4ea28fc218c4236575a8949d4128e26d393ab0e8b92cda: Status 404 returned error can't find the container with id 32e797ebe0c353a8fd4ea28fc218c4236575a8949d4128e26d393ab0e8b92cda Jul 22 12:42:14.324995 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:14.324836 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-jnzn4" event={"ID":"3cf037ef-07c5-4294-933e-6c61ef336250","Type":"ContainerStarted","Data":"8b40c5f40954b123523813f8cfffa75a4c47fc7a7b39768e3887be70c609f1db"} Jul 22 12:42:14.324995 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:14.324919 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-jnzn4" event={"ID":"3cf037ef-07c5-4294-933e-6c61ef336250","Type":"ContainerStarted","Data":"db7ebc3842a2b8c207beeeeeac2960a0a39a9b46b54e58ca020853277f1c3365"} Jul 22 12:42:14.324995 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:14.324987 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-jnzn4" event={"ID":"3cf037ef-07c5-4294-933e-6c61ef336250","Type":"ContainerStarted","Data":"40fd00872a6b503d7ef2fbf0dc956dd6bb88b4385f9bde0840f4d93dfffd253b"} Jul 22 12:42:14.326344 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:14.326314 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-48jh9" event={"ID":"c98bde4d-82db-4b9d-969a-892fc9ef54f2","Type":"ContainerStarted","Data":"32e797ebe0c353a8fd4ea28fc218c4236575a8949d4128e26d393ab0e8b92cda"} Jul 22 12:42:14.940318 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:14.940283 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/42f58d4d-89d6-46e4-8598-0a428e9ed439-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-75nsf\" (UID: \"42f58d4d-89d6-46e4-8598-0a428e9ed439\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75nsf" Jul 22 12:42:14.943041 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:14.943008 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/42f58d4d-89d6-46e4-8598-0a428e9ed439-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-75nsf\" (UID: \"42f58d4d-89d6-46e4-8598-0a428e9ed439\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-75nsf" Jul 22 12:42:15.011283 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.011208 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-75nsf" Jul 22 12:42:15.048867 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.048837 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68bc864c5-ckqws"] Jul 22 12:42:15.053046 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.053025 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:42:15.055672 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.055645 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 22 12:42:15.055808 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.055789 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 22 12:42:15.056968 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.056949 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 22 12:42:15.057081 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.056971 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 22 12:42:15.057142 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.057092 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 22 12:42:15.057195 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.057138 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-22jtw\"" Jul 22 12:42:15.057448 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.057431 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 22 12:42:15.057799 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.057781 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 22 12:42:15.063236 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.062838 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 22 12:42:15.064288 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.064266 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68bc864c5-ckqws"] Jul 22 12:42:15.142553 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.142518 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-trusted-ca-bundle\") pod \"console-68bc864c5-ckqws\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:42:15.142715 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.142602 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-console-serving-cert\") pod \"console-68bc864c5-ckqws\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:42:15.142715 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.142639 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-console-oauth-config\") pod \"console-68bc864c5-ckqws\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:42:15.142846 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.142728 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-console-config\") pod \"console-68bc864c5-ckqws\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:42:15.142846 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.142751 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-service-ca\") pod \"console-68bc864c5-ckqws\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:42:15.142846 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.142770 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-oauth-serving-cert\") pod \"console-68bc864c5-ckqws\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:42:15.142846 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.142794 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5qr7\" (UniqueName: \"kubernetes.io/projected/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-kube-api-access-t5qr7\") pod \"console-68bc864c5-ckqws\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:42:15.165285 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.165221 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-75nsf"] Jul 22 12:42:15.243229 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.243191 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-trusted-ca-bundle\") pod \"console-68bc864c5-ckqws\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:42:15.243229 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.243234 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-console-serving-cert\") pod \"console-68bc864c5-ckqws\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:42:15.243471 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.243257 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-console-oauth-config\") pod \"console-68bc864c5-ckqws\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:42:15.243471 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.243300 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-console-config\") pod \"console-68bc864c5-ckqws\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:42:15.243471 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.243341 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-service-ca\") pod \"console-68bc864c5-ckqws\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:42:15.243471 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.243364 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-oauth-serving-cert\") pod \"console-68bc864c5-ckqws\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:42:15.243699 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.243666 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5qr7\" (UniqueName: \"kubernetes.io/projected/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-kube-api-access-t5qr7\") pod \"console-68bc864c5-ckqws\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:42:15.244150 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.244122 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-trusted-ca-bundle\") pod \"console-68bc864c5-ckqws\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:42:15.244150 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.244134 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-console-config\") pod \"console-68bc864c5-ckqws\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:42:15.244302 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.244122 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-service-ca\") pod \"console-68bc864c5-ckqws\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:42:15.244399 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.244367 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-oauth-serving-cert\") pod \"console-68bc864c5-ckqws\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:42:15.245799 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.245777 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-console-oauth-config\") pod \"console-68bc864c5-ckqws\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:42:15.245955 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.245918 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-console-serving-cert\") pod \"console-68bc864c5-ckqws\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:42:15.257424 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.257399 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5qr7\" (UniqueName: \"kubernetes.io/projected/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-kube-api-access-t5qr7\") pod \"console-68bc864c5-ckqws\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:42:15.333836 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.333795 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-jnzn4" event={"ID":"3cf037ef-07c5-4294-933e-6c61ef336250","Type":"ContainerStarted","Data":"ccc8e5b9c0451a80fe554d8bb334e9639b82b4d07076349a6244ce2698e0b6e3"} Jul 22 12:42:15.366732 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.366703 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:42:15.369182 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:42:15.369151 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f58d4d_89d6_46e4_8598_0a428e9ed439.slice/crio-cda21420af8c584484e54c3fe4ba6f20838cb6599d350790d640f16a27c3b3f5 WatchSource:0}: Error finding container cda21420af8c584484e54c3fe4ba6f20838cb6599d350790d640f16a27c3b3f5: Status 404 returned error can't find the container with id cda21420af8c584484e54c3fe4ba6f20838cb6599d350790d640f16a27c3b3f5 Jul 22 12:42:15.507724 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.507661 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-jnzn4" podStartSLOduration=1.5614677989999999 podStartE2EDuration="2.507640001s" totalImagesPullingTime="946.172202ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:42:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:15.357090984 +0000 UTC m=+83.758321907" watchObservedRunningTime="2026-07-22 12:42:15.507640001 +0000 UTC m=+83.908870921" Jul 22 12:42:15.509234 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:15.509209 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68bc864c5-ckqws"] Jul 22 12:42:15.511148 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:42:15.511119 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9abd37_29a1_4dc3_9b53_3805dfd3692d.slice/crio-2745c2b7e092cc8871d4ba21d5ea375363e7f20dfafc8f74fe246c2cd5fb83be WatchSource:0}: Error finding container 2745c2b7e092cc8871d4ba21d5ea375363e7f20dfafc8f74fe246c2cd5fb83be: Status 404 returned error can't find the container with id 2745c2b7e092cc8871d4ba21d5ea375363e7f20dfafc8f74fe246c2cd5fb83be Jul 22 12:42:16.339876 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.339837 2579 generic.go:356] "Generic (PLEG): container finished" podID="c98bde4d-82db-4b9d-969a-892fc9ef54f2" containerID="b4614ade9de7df47b7bf25bba309c41ba663681506a267824d53519b55c70601" exitCode=0 Jul 22 12:42:16.340280 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.339973 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-48jh9" event={"ID":"c98bde4d-82db-4b9d-969a-892fc9ef54f2","Type":"ContainerDied","Data":"b4614ade9de7df47b7bf25bba309c41ba663681506a267824d53519b55c70601"} Jul 22 12:42:16.341736 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.341710 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-75nsf" event={"ID":"42f58d4d-89d6-46e4-8598-0a428e9ed439","Type":"ContainerStarted","Data":"cda21420af8c584484e54c3fe4ba6f20838cb6599d350790d640f16a27c3b3f5"} Jul 22 12:42:16.343278 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.343244 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68bc864c5-ckqws" event={"ID":"bc9abd37-29a1-4dc3-9b53-3805dfd3692d","Type":"ContainerStarted","Data":"2745c2b7e092cc8871d4ba21d5ea375363e7f20dfafc8f74fe246c2cd5fb83be"} Jul 22 12:42:16.350334 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.349225 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6979d99b4-hwtbm"] Jul 22 12:42:16.354370 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.354348 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:16.357368 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.357348 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 22 12:42:16.357485 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.357463 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 22 12:42:16.357587 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.357571 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bj6co6c32al0b\"" Jul 22 12:42:16.357697 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.357628 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-hz2tm\"" Jul 22 12:42:16.357795 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.357687 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 22 12:42:16.358199 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.358179 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 22 12:42:16.358306 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.358289 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 22 12:42:16.376393 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.376367 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6979d99b4-hwtbm"] Jul 22 12:42:16.454322 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.454289 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b0204876-d2f4-486c-95cb-7a6f0726e566-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6979d99b4-hwtbm\" (UID: \"b0204876-d2f4-486c-95cb-7a6f0726e566\") " pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:16.454497 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.454350 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk6z9\" (UniqueName: \"kubernetes.io/projected/b0204876-d2f4-486c-95cb-7a6f0726e566-kube-api-access-dk6z9\") pod \"thanos-querier-6979d99b4-hwtbm\" (UID: \"b0204876-d2f4-486c-95cb-7a6f0726e566\") " pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:16.454497 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.454466 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b0204876-d2f4-486c-95cb-7a6f0726e566-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6979d99b4-hwtbm\" (UID: \"b0204876-d2f4-486c-95cb-7a6f0726e566\") " pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:16.454649 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.454614 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b0204876-d2f4-486c-95cb-7a6f0726e566-secret-thanos-querier-tls\") pod \"thanos-querier-6979d99b4-hwtbm\" (UID: \"b0204876-d2f4-486c-95cb-7a6f0726e566\") " pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:16.454716 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.454647 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0204876-d2f4-486c-95cb-7a6f0726e566-metrics-client-ca\") pod \"thanos-querier-6979d99b4-hwtbm\" (UID: \"b0204876-d2f4-486c-95cb-7a6f0726e566\") " pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:16.454823 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.454799 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b0204876-d2f4-486c-95cb-7a6f0726e566-secret-grpc-tls\") pod \"thanos-querier-6979d99b4-hwtbm\" (UID: \"b0204876-d2f4-486c-95cb-7a6f0726e566\") " pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:16.454901 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.454889 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b0204876-d2f4-486c-95cb-7a6f0726e566-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6979d99b4-hwtbm\" (UID: \"b0204876-d2f4-486c-95cb-7a6f0726e566\") " pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:16.455212 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.454984 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b0204876-d2f4-486c-95cb-7a6f0726e566-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6979d99b4-hwtbm\" (UID: \"b0204876-d2f4-486c-95cb-7a6f0726e566\") " pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:16.555596 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.555570 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk6z9\" (UniqueName: \"kubernetes.io/projected/b0204876-d2f4-486c-95cb-7a6f0726e566-kube-api-access-dk6z9\") pod \"thanos-querier-6979d99b4-hwtbm\" (UID: \"b0204876-d2f4-486c-95cb-7a6f0726e566\") " pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:16.555724 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.555617 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b0204876-d2f4-486c-95cb-7a6f0726e566-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6979d99b4-hwtbm\" (UID: \"b0204876-d2f4-486c-95cb-7a6f0726e566\") " pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:16.555724 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.555647 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b0204876-d2f4-486c-95cb-7a6f0726e566-secret-thanos-querier-tls\") pod \"thanos-querier-6979d99b4-hwtbm\" (UID: \"b0204876-d2f4-486c-95cb-7a6f0726e566\") " pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:16.555724 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.555676 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0204876-d2f4-486c-95cb-7a6f0726e566-metrics-client-ca\") pod \"thanos-querier-6979d99b4-hwtbm\" (UID: \"b0204876-d2f4-486c-95cb-7a6f0726e566\") " pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:16.555724 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.555702 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b0204876-d2f4-486c-95cb-7a6f0726e566-secret-grpc-tls\") pod \"thanos-querier-6979d99b4-hwtbm\" (UID: \"b0204876-d2f4-486c-95cb-7a6f0726e566\") " pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:16.555909 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.555744 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b0204876-d2f4-486c-95cb-7a6f0726e566-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6979d99b4-hwtbm\" (UID: \"b0204876-d2f4-486c-95cb-7a6f0726e566\") " pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:16.555909 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.555780 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b0204876-d2f4-486c-95cb-7a6f0726e566-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6979d99b4-hwtbm\" (UID: \"b0204876-d2f4-486c-95cb-7a6f0726e566\") " pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:16.555909 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.555829 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b0204876-d2f4-486c-95cb-7a6f0726e566-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6979d99b4-hwtbm\" (UID: \"b0204876-d2f4-486c-95cb-7a6f0726e566\") " pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:16.557355 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.557327 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b0204876-d2f4-486c-95cb-7a6f0726e566-metrics-client-ca\") pod \"thanos-querier-6979d99b4-hwtbm\" (UID: \"b0204876-d2f4-486c-95cb-7a6f0726e566\") " pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:16.559233 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.559205 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b0204876-d2f4-486c-95cb-7a6f0726e566-secret-grpc-tls\") pod \"thanos-querier-6979d99b4-hwtbm\" (UID: \"b0204876-d2f4-486c-95cb-7a6f0726e566\") " pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:16.559346 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.559293 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b0204876-d2f4-486c-95cb-7a6f0726e566-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6979d99b4-hwtbm\" (UID: \"b0204876-d2f4-486c-95cb-7a6f0726e566\") " pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:16.559745 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.559725 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/b0204876-d2f4-486c-95cb-7a6f0726e566-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6979d99b4-hwtbm\" (UID: \"b0204876-d2f4-486c-95cb-7a6f0726e566\") " pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:16.559793 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.559730 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/b0204876-d2f4-486c-95cb-7a6f0726e566-secret-thanos-querier-tls\") pod \"thanos-querier-6979d99b4-hwtbm\" (UID: \"b0204876-d2f4-486c-95cb-7a6f0726e566\") " pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:16.560288 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.560246 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/b0204876-d2f4-486c-95cb-7a6f0726e566-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6979d99b4-hwtbm\" (UID: \"b0204876-d2f4-486c-95cb-7a6f0726e566\") " pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:16.561588 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.561568 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b0204876-d2f4-486c-95cb-7a6f0726e566-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6979d99b4-hwtbm\" (UID: \"b0204876-d2f4-486c-95cb-7a6f0726e566\") " pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:16.567271 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.567226 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk6z9\" (UniqueName: \"kubernetes.io/projected/b0204876-d2f4-486c-95cb-7a6f0726e566-kube-api-access-dk6z9\") pod \"thanos-querier-6979d99b4-hwtbm\" (UID: \"b0204876-d2f4-486c-95cb-7a6f0726e566\") " pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:16.666202 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.665992 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:16.866690 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:16.866612 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6979d99b4-hwtbm"] Jul 22 12:42:16.871360 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:42:16.871336 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0204876_d2f4_486c_95cb_7a6f0726e566.slice/crio-b3e85b4d0cb061362e0d56846d51f72b39aa537a91873a332bb348317cc47259 WatchSource:0}: Error finding container b3e85b4d0cb061362e0d56846d51f72b39aa537a91873a332bb348317cc47259: Status 404 returned error can't find the container with id b3e85b4d0cb061362e0d56846d51f72b39aa537a91873a332bb348317cc47259 Jul 22 12:42:17.352852 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.352818 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-75nsf" event={"ID":"42f58d4d-89d6-46e4-8598-0a428e9ed439","Type":"ContainerStarted","Data":"efcde495050735f1c57503789434fb0b3fd78508782ca78592f5ba5d961538dd"} Jul 22 12:42:17.353296 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.352911 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-75nsf" event={"ID":"42f58d4d-89d6-46e4-8598-0a428e9ed439","Type":"ContainerStarted","Data":"4dd83051cb74f85c015a3a9452cc47c540607a74fddec31eff104c5fb0cc6821"} Jul 22 12:42:17.353296 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.352965 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-75nsf" event={"ID":"42f58d4d-89d6-46e4-8598-0a428e9ed439","Type":"ContainerStarted","Data":"c38f103ae5d29a3cea11f901e3226fac346890ed8a1417bb6b9d13fa43983bcc"} Jul 22 12:42:17.354340 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.354314 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" event={"ID":"b0204876-d2f4-486c-95cb-7a6f0726e566","Type":"ContainerStarted","Data":"b3e85b4d0cb061362e0d56846d51f72b39aa537a91873a332bb348317cc47259"} Jul 22 12:42:17.356570 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.356546 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-48jh9" event={"ID":"c98bde4d-82db-4b9d-969a-892fc9ef54f2","Type":"ContainerStarted","Data":"36f799821713aa8e624f16a6b28bc1a2df636c1e227c874405798489467d2767"} Jul 22 12:42:17.356668 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.356595 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-48jh9" event={"ID":"c98bde4d-82db-4b9d-969a-892fc9ef54f2","Type":"ContainerStarted","Data":"26fd648df336294b4f24ed4373a926d5a36f93fb8612977be1871a9e9dad777f"} Jul 22 12:42:17.379077 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.379028 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-75nsf" podStartSLOduration=3.171543565 podStartE2EDuration="4.379008394s" totalImagesPullingTime="1.207464829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:42:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:17.377477548 +0000 UTC m=+85.778708512" watchObservedRunningTime="2026-07-22 12:42:17.379008394 +0000 UTC m=+85.780239316" Jul 22 12:42:17.398596 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.398558 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-48jh9" podStartSLOduration=3.057271655 podStartE2EDuration="4.398544643s" totalImagesPullingTime="1.276910027s" totalInitContainerRuntime="64.362961ms" isStatefulPod=true podCreationTimestamp="2026-07-22 12:42:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:17.397494432 +0000 UTC m=+85.798725352" watchObservedRunningTime="2026-07-22 12:42:17.398544643 +0000 UTC m=+85.799775562" Jul 22 12:42:17.619169 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.619082 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-64766cc68b-h7chl"] Jul 22 12:42:17.622775 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.622749 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" Jul 22 12:42:17.625766 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.625742 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 22 12:42:17.625880 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.625799 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 22 12:42:17.626036 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.626019 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 22 12:42:17.627246 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.627059 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 22 12:42:17.627645 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.627387 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-f6dq2\"" Jul 22 12:42:17.627645 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.627517 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6nhs662tki0pi\"" Jul 22 12:42:17.631868 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.631847 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-64766cc68b-h7chl"] Jul 22 12:42:17.768295 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.768253 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50a8d9a2-5d26-4f2b-8996-d1bb986e41b7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64766cc68b-h7chl\" (UID: \"50a8d9a2-5d26-4f2b-8996-d1bb986e41b7\") " pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" Jul 22 12:42:17.768460 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.768325 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/50a8d9a2-5d26-4f2b-8996-d1bb986e41b7-secret-metrics-server-tls\") pod \"metrics-server-64766cc68b-h7chl\" (UID: \"50a8d9a2-5d26-4f2b-8996-d1bb986e41b7\") " pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" Jul 22 12:42:17.768460 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.768359 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzspd\" (UniqueName: \"kubernetes.io/projected/50a8d9a2-5d26-4f2b-8996-d1bb986e41b7-kube-api-access-kzspd\") pod \"metrics-server-64766cc68b-h7chl\" (UID: \"50a8d9a2-5d26-4f2b-8996-d1bb986e41b7\") " pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" Jul 22 12:42:17.768460 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.768397 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/50a8d9a2-5d26-4f2b-8996-d1bb986e41b7-audit-log\") pod \"metrics-server-64766cc68b-h7chl\" (UID: \"50a8d9a2-5d26-4f2b-8996-d1bb986e41b7\") " pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" Jul 22 12:42:17.768601 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.768481 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/50a8d9a2-5d26-4f2b-8996-d1bb986e41b7-metrics-server-audit-profiles\") pod \"metrics-server-64766cc68b-h7chl\" (UID: \"50a8d9a2-5d26-4f2b-8996-d1bb986e41b7\") " pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" Jul 22 12:42:17.768601 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.768512 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/50a8d9a2-5d26-4f2b-8996-d1bb986e41b7-client-ca-bundle\") pod \"metrics-server-64766cc68b-h7chl\" (UID: \"50a8d9a2-5d26-4f2b-8996-d1bb986e41b7\") " pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" Jul 22 12:42:17.768601 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.768568 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/50a8d9a2-5d26-4f2b-8996-d1bb986e41b7-secret-metrics-server-client-certs\") pod \"metrics-server-64766cc68b-h7chl\" (UID: \"50a8d9a2-5d26-4f2b-8996-d1bb986e41b7\") " pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" Jul 22 12:42:17.869777 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.869700 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50a8d9a2-5d26-4f2b-8996-d1bb986e41b7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64766cc68b-h7chl\" (UID: \"50a8d9a2-5d26-4f2b-8996-d1bb986e41b7\") " pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" Jul 22 12:42:17.869777 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.869743 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/50a8d9a2-5d26-4f2b-8996-d1bb986e41b7-secret-metrics-server-tls\") pod \"metrics-server-64766cc68b-h7chl\" (UID: \"50a8d9a2-5d26-4f2b-8996-d1bb986e41b7\") " pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" Jul 22 12:42:17.869777 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.869771 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzspd\" (UniqueName: \"kubernetes.io/projected/50a8d9a2-5d26-4f2b-8996-d1bb986e41b7-kube-api-access-kzspd\") pod \"metrics-server-64766cc68b-h7chl\" (UID: \"50a8d9a2-5d26-4f2b-8996-d1bb986e41b7\") " pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" Jul 22 12:42:17.870053 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.869975 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/50a8d9a2-5d26-4f2b-8996-d1bb986e41b7-audit-log\") pod \"metrics-server-64766cc68b-h7chl\" (UID: \"50a8d9a2-5d26-4f2b-8996-d1bb986e41b7\") " pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" Jul 22 12:42:17.870103 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.870064 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/50a8d9a2-5d26-4f2b-8996-d1bb986e41b7-metrics-server-audit-profiles\") pod \"metrics-server-64766cc68b-h7chl\" (UID: \"50a8d9a2-5d26-4f2b-8996-d1bb986e41b7\") " pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" Jul 22 12:42:17.870103 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.870087 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/50a8d9a2-5d26-4f2b-8996-d1bb986e41b7-client-ca-bundle\") pod \"metrics-server-64766cc68b-h7chl\" (UID: \"50a8d9a2-5d26-4f2b-8996-d1bb986e41b7\") " pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" Jul 22 12:42:17.870175 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.870122 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/50a8d9a2-5d26-4f2b-8996-d1bb986e41b7-secret-metrics-server-client-certs\") pod \"metrics-server-64766cc68b-h7chl\" (UID: \"50a8d9a2-5d26-4f2b-8996-d1bb986e41b7\") " pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" Jul 22 12:42:17.870989 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.870484 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50a8d9a2-5d26-4f2b-8996-d1bb986e41b7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-64766cc68b-h7chl\" (UID: \"50a8d9a2-5d26-4f2b-8996-d1bb986e41b7\") " pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" Jul 22 12:42:17.870989 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.870788 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/50a8d9a2-5d26-4f2b-8996-d1bb986e41b7-audit-log\") pod \"metrics-server-64766cc68b-h7chl\" (UID: \"50a8d9a2-5d26-4f2b-8996-d1bb986e41b7\") " pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" Jul 22 12:42:17.871125 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.871031 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/50a8d9a2-5d26-4f2b-8996-d1bb986e41b7-metrics-server-audit-profiles\") pod \"metrics-server-64766cc68b-h7chl\" (UID: \"50a8d9a2-5d26-4f2b-8996-d1bb986e41b7\") " pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" Jul 22 12:42:17.873129 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.873105 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/50a8d9a2-5d26-4f2b-8996-d1bb986e41b7-secret-metrics-server-client-certs\") pod \"metrics-server-64766cc68b-h7chl\" (UID: \"50a8d9a2-5d26-4f2b-8996-d1bb986e41b7\") " pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" Jul 22 12:42:17.873248 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.873228 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/50a8d9a2-5d26-4f2b-8996-d1bb986e41b7-client-ca-bundle\") pod \"metrics-server-64766cc68b-h7chl\" (UID: \"50a8d9a2-5d26-4f2b-8996-d1bb986e41b7\") " pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" Jul 22 12:42:17.873480 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.873465 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/50a8d9a2-5d26-4f2b-8996-d1bb986e41b7-secret-metrics-server-tls\") pod \"metrics-server-64766cc68b-h7chl\" (UID: \"50a8d9a2-5d26-4f2b-8996-d1bb986e41b7\") " pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" Jul 22 12:42:17.881676 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.881633 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzspd\" (UniqueName: \"kubernetes.io/projected/50a8d9a2-5d26-4f2b-8996-d1bb986e41b7-kube-api-access-kzspd\") pod \"metrics-server-64766cc68b-h7chl\" (UID: \"50a8d9a2-5d26-4f2b-8996-d1bb986e41b7\") " pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" Jul 22 12:42:17.935292 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:17.935263 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" Jul 22 12:42:18.023287 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:18.023249 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p94kz"] Jul 22 12:42:18.028443 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:18.028420 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p94kz" Jul 22 12:42:18.031336 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:18.031314 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-hh6cj\"" Jul 22 12:42:18.031584 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:18.031568 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 22 12:42:18.067445 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:18.067414 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p94kz"] Jul 22 12:42:18.172734 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:18.172655 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc5706d0-452b-4568-8460-30c35440cf14-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-p94kz\" (UID: \"dc5706d0-452b-4568-8460-30c35440cf14\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p94kz" Jul 22 12:42:18.172734 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:18.172696 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqwmp\" (UniqueName: \"kubernetes.io/projected/dc5706d0-452b-4568-8460-30c35440cf14-kube-api-access-zqwmp\") pod \"monitoring-plugin-7b7b7bc8bf-p94kz\" (UID: \"dc5706d0-452b-4568-8460-30c35440cf14\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p94kz" Jul 22 12:42:18.273826 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:18.273792 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc5706d0-452b-4568-8460-30c35440cf14-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-p94kz\" (UID: \"dc5706d0-452b-4568-8460-30c35440cf14\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p94kz" Jul 22 12:42:18.274012 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:18.273834 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqwmp\" (UniqueName: \"kubernetes.io/projected/dc5706d0-452b-4568-8460-30c35440cf14-kube-api-access-zqwmp\") pod \"monitoring-plugin-7b7b7bc8bf-p94kz\" (UID: \"dc5706d0-452b-4568-8460-30c35440cf14\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p94kz" Jul 22 12:42:18.274012 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:42:18.273980 2579 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 22 12:42:18.274123 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:42:18.274058 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc5706d0-452b-4568-8460-30c35440cf14-monitoring-plugin-cert podName:dc5706d0-452b-4568-8460-30c35440cf14 nodeName:}" failed. No retries permitted until 2026-07-22 12:42:18.774037229 +0000 UTC m=+87.175268130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/dc5706d0-452b-4568-8460-30c35440cf14-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-p94kz" (UID: "dc5706d0-452b-4568-8460-30c35440cf14") : secret "monitoring-plugin-cert" not found Jul 22 12:42:18.286483 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:18.286459 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqwmp\" (UniqueName: \"kubernetes.io/projected/dc5706d0-452b-4568-8460-30c35440cf14-kube-api-access-zqwmp\") pod \"monitoring-plugin-7b7b7bc8bf-p94kz\" (UID: \"dc5706d0-452b-4568-8460-30c35440cf14\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p94kz" Jul 22 12:42:18.469023 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:18.468993 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-64766cc68b-h7chl"] Jul 22 12:42:18.473097 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:42:18.473061 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a8d9a2_5d26_4f2b_8996_d1bb986e41b7.slice/crio-31fb0ccfb1bd026daf436aaf5e7bbc8f45c2cae5b6245a6f5cf762a29224d553 WatchSource:0}: Error finding container 31fb0ccfb1bd026daf436aaf5e7bbc8f45c2cae5b6245a6f5cf762a29224d553: Status 404 returned error can't find the container with id 31fb0ccfb1bd026daf436aaf5e7bbc8f45c2cae5b6245a6f5cf762a29224d553 Jul 22 12:42:18.777448 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:18.777363 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc5706d0-452b-4568-8460-30c35440cf14-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-p94kz\" (UID: \"dc5706d0-452b-4568-8460-30c35440cf14\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p94kz" Jul 22 12:42:18.779905 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:18.779873 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc5706d0-452b-4568-8460-30c35440cf14-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-p94kz\" (UID: \"dc5706d0-452b-4568-8460-30c35440cf14\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p94kz" Jul 22 12:42:18.942693 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:18.942660 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p94kz" Jul 22 12:42:19.094478 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:19.094443 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p94kz"] Jul 22 12:42:19.095528 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:42:19.095495 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc5706d0_452b_4568_8460_30c35440cf14.slice/crio-96a0f7741b3217680fb6c11838340c769db3b773362b0041b9e6e6c689e40117 WatchSource:0}: Error finding container 96a0f7741b3217680fb6c11838340c769db3b773362b0041b9e6e6c689e40117: Status 404 returned error can't find the container with id 96a0f7741b3217680fb6c11838340c769db3b773362b0041b9e6e6c689e40117 Jul 22 12:42:19.369439 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:19.368909 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68bc864c5-ckqws" event={"ID":"bc9abd37-29a1-4dc3-9b53-3805dfd3692d","Type":"ContainerStarted","Data":"5c45b54e41697e015430b26cedd2e8818b60459be6431c0f3362450007286dac"} Jul 22 12:42:19.370870 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:19.370792 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p94kz" event={"ID":"dc5706d0-452b-4568-8460-30c35440cf14","Type":"ContainerStarted","Data":"96a0f7741b3217680fb6c11838340c769db3b773362b0041b9e6e6c689e40117"} Jul 22 12:42:19.375316 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:19.375291 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" event={"ID":"50a8d9a2-5d26-4f2b-8996-d1bb986e41b7","Type":"ContainerStarted","Data":"31fb0ccfb1bd026daf436aaf5e7bbc8f45c2cae5b6245a6f5cf762a29224d553"} Jul 22 12:42:19.412602 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:19.412530 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68bc864c5-ckqws" podStartSLOduration=1.533056678 podStartE2EDuration="4.412506935s" totalImagesPullingTime="2.879450257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:42:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:19.409724176 +0000 UTC m=+87.810955113" watchObservedRunningTime="2026-07-22 12:42:19.412506935 +0000 UTC m=+87.813737855" Jul 22 12:42:21.384540 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:21.384504 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" event={"ID":"50a8d9a2-5d26-4f2b-8996-d1bb986e41b7","Type":"ContainerStarted","Data":"ff4919ea48c9f746d84fabea2867bae11563bedbfc00d46307fe1d2139a337dd"} Jul 22 12:42:21.387004 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:21.386972 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" event={"ID":"b0204876-d2f4-486c-95cb-7a6f0726e566","Type":"ContainerStarted","Data":"5361e1fb3e5711126a91d93d50a3c0fa27345376fc2759791a957d9b29d37cad"} Jul 22 12:42:21.387131 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:21.387047 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" event={"ID":"b0204876-d2f4-486c-95cb-7a6f0726e566","Type":"ContainerStarted","Data":"d8920a3c9cf3a9a39cb7b9707c5f0d2d4be5fbc3bd4e0b82a1a9bbc278f93743"} Jul 22 12:42:21.387131 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:21.387088 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" event={"ID":"b0204876-d2f4-486c-95cb-7a6f0726e566","Type":"ContainerStarted","Data":"f929773e72841bfea40a51868a4cc0b8122d52ae8de9edfedfe6fa58b9624514"} Jul 22 12:42:21.388886 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:21.388857 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p94kz" event={"ID":"dc5706d0-452b-4568-8460-30c35440cf14","Type":"ContainerStarted","Data":"9914c251c876471defae6916943b6d0d7f02c4b85e8b4a597b40360d1a573ad6"} Jul 22 12:42:21.389066 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:21.389047 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p94kz" Jul 22 12:42:21.395312 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:21.395287 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p94kz" Jul 22 12:42:21.420027 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:21.419974 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" podStartSLOduration=2.499624079 podStartE2EDuration="4.419958938s" totalImagesPullingTime="1.920334859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:21.417400286 +0000 UTC m=+89.818631207" watchObservedRunningTime="2026-07-22 12:42:21.419958938 +0000 UTC m=+89.821189852" Jul 22 12:42:21.436437 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:21.436379 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-p94kz" podStartSLOduration=2.630988389 podStartE2EDuration="4.436358773s" totalImagesPullingTime="1.805370384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:21.435553683 +0000 UTC m=+89.836784606" watchObservedRunningTime="2026-07-22 12:42:21.436358773 +0000 UTC m=+89.837589693" Jul 22 12:42:22.396315 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:22.396270 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" event={"ID":"b0204876-d2f4-486c-95cb-7a6f0726e566","Type":"ContainerStarted","Data":"bb0e1224d7eed3418f5a063f1827b0efba0dd077f6b2b2ad880f5fa1aa89949f"} Jul 22 12:42:22.396765 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:22.396345 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" event={"ID":"b0204876-d2f4-486c-95cb-7a6f0726e566","Type":"ContainerStarted","Data":"7341a0a793272a7183cd139eb78077472f1b5f3680421eb2482d6111c49ee38e"} Jul 22 12:42:22.396765 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:22.396376 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" event={"ID":"b0204876-d2f4-486c-95cb-7a6f0726e566","Type":"ContainerStarted","Data":"230268f280472247ca01afb3f6c5c1445b7ec8125d655e9bc4852bfd9fc9887d"} Jul 22 12:42:22.446173 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:22.446119 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" podStartSLOduration=1.903245401 podStartE2EDuration="6.446103568s" totalImagesPullingTime="4.542858167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:42:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:22.444203168 +0000 UTC m=+90.845434101" watchObservedRunningTime="2026-07-22 12:42:22.446103568 +0000 UTC m=+90.847334487" Jul 22 12:42:23.400417 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:23.400389 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:25.366894 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:25.366858 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:42:25.367284 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:25.367021 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:42:25.372194 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:25.372175 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:42:25.410119 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:25.410094 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:42:28.156660 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:28.156624 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65594785fd-bz47t"] Jul 22 12:42:28.233879 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:28.233843 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65594785fd-bz47t"] Jul 22 12:42:28.234025 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:28.233961 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:42:28.359732 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:28.359697 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p5fv\" (UniqueName: \"kubernetes.io/projected/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-kube-api-access-2p5fv\") pod \"console-65594785fd-bz47t\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:42:28.359880 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:28.359740 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-oauth-serving-cert\") pod \"console-65594785fd-bz47t\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:42:28.359880 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:28.359825 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-console-serving-cert\") pod \"console-65594785fd-bz47t\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:42:28.359880 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:28.359845 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-trusted-ca-bundle\") pod \"console-65594785fd-bz47t\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:42:28.359880 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:28.359873 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-service-ca\") pod \"console-65594785fd-bz47t\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:42:28.360056 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:28.359907 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-console-config\") pod \"console-65594785fd-bz47t\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:42:28.360056 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:28.359948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-console-oauth-config\") pod \"console-65594785fd-bz47t\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:42:28.460524 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:28.460453 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-oauth-serving-cert\") pod \"console-65594785fd-bz47t\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:42:28.460524 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:28.460505 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-console-serving-cert\") pod \"console-65594785fd-bz47t\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:42:28.460761 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:28.460653 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-trusted-ca-bundle\") pod \"console-65594785fd-bz47t\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:42:28.460761 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:28.460709 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-service-ca\") pod \"console-65594785fd-bz47t\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:42:28.460761 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:28.460748 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-console-config\") pod \"console-65594785fd-bz47t\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:42:28.460911 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:28.460853 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-console-oauth-config\") pod \"console-65594785fd-bz47t\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:42:28.461061 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:28.461041 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p5fv\" (UniqueName: \"kubernetes.io/projected/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-kube-api-access-2p5fv\") pod \"console-65594785fd-bz47t\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:42:28.461554 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:28.461529 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-service-ca\") pod \"console-65594785fd-bz47t\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:42:28.461678 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:28.461559 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-trusted-ca-bundle\") pod \"console-65594785fd-bz47t\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:42:28.461748 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:28.461700 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-oauth-serving-cert\") pod \"console-65594785fd-bz47t\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:42:28.461821 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:28.461795 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-console-config\") pod \"console-65594785fd-bz47t\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:42:28.463034 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:28.463016 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-console-serving-cert\") pod \"console-65594785fd-bz47t\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:42:28.463533 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:28.463512 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-console-oauth-config\") pod \"console-65594785fd-bz47t\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:42:28.470266 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:28.470245 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p5fv\" (UniqueName: \"kubernetes.io/projected/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-kube-api-access-2p5fv\") pod \"console-65594785fd-bz47t\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:42:28.542442 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:28.542410 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:42:28.723264 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:28.723238 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65594785fd-bz47t"] Jul 22 12:42:28.725457 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:42:28.725423 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb193e6c4_09bc_4462_bf47_0a43c2c22f3f.slice/crio-0a955c5debf9bc11ff747679c7deb4224d73676d036c758bcd25f8a2cf3afa42 WatchSource:0}: Error finding container 0a955c5debf9bc11ff747679c7deb4224d73676d036c758bcd25f8a2cf3afa42: Status 404 returned error can't find the container with id 0a955c5debf9bc11ff747679c7deb4224d73676d036c758bcd25f8a2cf3afa42 Jul 22 12:42:29.409591 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:29.409562 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6979d99b4-hwtbm" Jul 22 12:42:29.424380 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:29.424350 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65594785fd-bz47t" event={"ID":"b193e6c4-09bc-4462-bf47-0a43c2c22f3f","Type":"ContainerStarted","Data":"59db68ba972a385c51acfd582b4c9dfa79dbcef0efe4c2b922a96f76382685d8"} Jul 22 12:42:29.424520 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:29.424395 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65594785fd-bz47t" event={"ID":"b193e6c4-09bc-4462-bf47-0a43c2c22f3f","Type":"ContainerStarted","Data":"0a955c5debf9bc11ff747679c7deb4224d73676d036c758bcd25f8a2cf3afa42"} Jul 22 12:42:29.460950 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:29.458453 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65594785fd-bz47t" podStartSLOduration=1.458433682 podStartE2EDuration="1.458433682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:42:29.455374067 +0000 UTC m=+97.856604980" watchObservedRunningTime="2026-07-22 12:42:29.458433682 +0000 UTC m=+97.859664603" Jul 22 12:42:37.936260 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:37.936206 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" Jul 22 12:42:37.936658 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:37.936285 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" Jul 22 12:42:38.299828 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:38.299799 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6ghfv" Jul 22 12:42:38.543166 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:38.543136 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:42:38.543166 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:38.543185 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:42:38.548075 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:38.548052 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:42:39.466089 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:39.466059 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:42:39.534132 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:39.534101 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68bc864c5-ckqws"] Jul 22 12:42:57.941415 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:57.941380 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" Jul 22 12:42:57.945027 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:42:57.945010 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-64766cc68b-h7chl" Jul 22 12:43:04.555227 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:04.555165 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68bc864c5-ckqws" podUID="bc9abd37-29a1-4dc3-9b53-3805dfd3692d" containerName="console" containerID="cri-o://5c45b54e41697e015430b26cedd2e8818b60459be6431c0f3362450007286dac" gracePeriod=15 Jul 22 12:43:04.811494 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:04.811441 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:43:04.861437 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:04.861406 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-console-config\") pod \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " Jul 22 12:43:04.861573 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:04.861462 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-console-oauth-config\") pod \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " Jul 22 12:43:04.861573 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:04.861479 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5qr7\" (UniqueName: \"kubernetes.io/projected/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-kube-api-access-t5qr7\") pod \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " Jul 22 12:43:04.861644 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:04.861595 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-service-ca\") pod \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " Jul 22 12:43:04.861682 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:04.861667 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-oauth-serving-cert\") pod \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " Jul 22 12:43:04.861730 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:04.861720 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-trusted-ca-bundle\") pod \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " Jul 22 12:43:04.861791 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:04.861775 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-console-serving-cert\") pod \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\" (UID: \"bc9abd37-29a1-4dc3-9b53-3805dfd3692d\") " Jul 22 12:43:04.861964 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:04.861773 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-console-config" (OuterVolumeSpecName: "console-config") pod "bc9abd37-29a1-4dc3-9b53-3805dfd3692d" (UID: "bc9abd37-29a1-4dc3-9b53-3805dfd3692d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:43:04.862091 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:04.862042 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bc9abd37-29a1-4dc3-9b53-3805dfd3692d" (UID: "bc9abd37-29a1-4dc3-9b53-3805dfd3692d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:43:04.862091 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:04.861981 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-service-ca" (OuterVolumeSpecName: "service-ca") pod "bc9abd37-29a1-4dc3-9b53-3805dfd3692d" (UID: "bc9abd37-29a1-4dc3-9b53-3805dfd3692d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:43:04.862235 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:04.862183 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bc9abd37-29a1-4dc3-9b53-3805dfd3692d" (UID: "bc9abd37-29a1-4dc3-9b53-3805dfd3692d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:43:04.863883 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:04.863861 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bc9abd37-29a1-4dc3-9b53-3805dfd3692d" (UID: "bc9abd37-29a1-4dc3-9b53-3805dfd3692d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:43:04.864004 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:04.863976 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bc9abd37-29a1-4dc3-9b53-3805dfd3692d" (UID: "bc9abd37-29a1-4dc3-9b53-3805dfd3692d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:43:04.864061 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:04.863995 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-kube-api-access-t5qr7" (OuterVolumeSpecName: "kube-api-access-t5qr7") pod "bc9abd37-29a1-4dc3-9b53-3805dfd3692d" (UID: "bc9abd37-29a1-4dc3-9b53-3805dfd3692d"). InnerVolumeSpecName "kube-api-access-t5qr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:43:04.963038 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:04.962995 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-console-config\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:43:04.963038 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:04.963028 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-console-oauth-config\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:43:04.963038 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:04.963042 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5qr7\" (UniqueName: \"kubernetes.io/projected/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-kube-api-access-t5qr7\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:43:04.963277 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:04.963055 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-service-ca\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:43:04.963277 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:04.963067 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-oauth-serving-cert\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:43:04.963277 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:04.963078 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-trusted-ca-bundle\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:43:04.963277 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:04.963090 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc9abd37-29a1-4dc3-9b53-3805dfd3692d-console-serving-cert\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:43:05.562437 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:05.562400 2579 generic.go:356] "Generic (PLEG): container finished" podID="bc9abd37-29a1-4dc3-9b53-3805dfd3692d" containerID="5c45b54e41697e015430b26cedd2e8818b60459be6431c0f3362450007286dac" exitCode=0 Jul 22 12:43:05.562821 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:05.562446 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68bc864c5-ckqws" Jul 22 12:43:05.562821 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:05.562490 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68bc864c5-ckqws" event={"ID":"bc9abd37-29a1-4dc3-9b53-3805dfd3692d","Type":"ContainerDied","Data":"5c45b54e41697e015430b26cedd2e8818b60459be6431c0f3362450007286dac"} Jul 22 12:43:05.562821 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:05.562544 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68bc864c5-ckqws" event={"ID":"bc9abd37-29a1-4dc3-9b53-3805dfd3692d","Type":"ContainerDied","Data":"2745c2b7e092cc8871d4ba21d5ea375363e7f20dfafc8f74fe246c2cd5fb83be"} Jul 22 12:43:05.562821 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:05.562565 2579 scope.go:117] "RemoveContainer" containerID="5c45b54e41697e015430b26cedd2e8818b60459be6431c0f3362450007286dac" Jul 22 12:43:05.571627 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:05.571610 2579 scope.go:117] "RemoveContainer" containerID="5c45b54e41697e015430b26cedd2e8818b60459be6431c0f3362450007286dac" Jul 22 12:43:05.571912 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:43:05.571893 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c45b54e41697e015430b26cedd2e8818b60459be6431c0f3362450007286dac\": container with ID starting with 5c45b54e41697e015430b26cedd2e8818b60459be6431c0f3362450007286dac not found: ID does not exist" containerID="5c45b54e41697e015430b26cedd2e8818b60459be6431c0f3362450007286dac" Jul 22 12:43:05.571989 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:05.571920 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c45b54e41697e015430b26cedd2e8818b60459be6431c0f3362450007286dac"} err="failed to get container status \"5c45b54e41697e015430b26cedd2e8818b60459be6431c0f3362450007286dac\": rpc error: code = NotFound desc = could not find container \"5c45b54e41697e015430b26cedd2e8818b60459be6431c0f3362450007286dac\": container with ID starting with 5c45b54e41697e015430b26cedd2e8818b60459be6431c0f3362450007286dac not found: ID does not exist" Jul 22 12:43:05.585407 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:05.585378 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68bc864c5-ckqws"] Jul 22 12:43:05.589467 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:05.589448 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68bc864c5-ckqws"] Jul 22 12:43:06.065849 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:06.065818 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc9abd37-29a1-4dc3-9b53-3805dfd3692d" path="/var/lib/kubelet/pods/bc9abd37-29a1-4dc3-9b53-3805dfd3692d/volumes" Jul 22 12:43:35.288769 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.288733 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 12:43:35.289256 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.289192 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc9abd37-29a1-4dc3-9b53-3805dfd3692d" containerName="console" Jul 22 12:43:35.289256 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.289210 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc9abd37-29a1-4dc3-9b53-3805dfd3692d" containerName="console" Jul 22 12:43:35.289369 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.289301 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc9abd37-29a1-4dc3-9b53-3805dfd3692d" containerName="console" Jul 22 12:43:35.294110 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.294091 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.297117 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.297094 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 22 12:43:35.297245 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.297131 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 22 12:43:35.297314 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.297142 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-cckpq\"" Jul 22 12:43:35.297314 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.297155 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 22 12:43:35.297314 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.297290 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 22 12:43:35.297314 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.297156 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 22 12:43:35.297495 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.297205 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 22 12:43:35.297495 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.297200 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 22 12:43:35.298080 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.298064 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 22 12:43:35.304008 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.303992 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 22 12:43:35.310537 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.310512 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 12:43:35.405671 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.405639 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/21aa6be9-3e79-42eb-a1df-71a1e072b534-config-out\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.405671 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.405673 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/21aa6be9-3e79-42eb-a1df-71a1e072b534-config-volume\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.405862 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.405689 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/21aa6be9-3e79-42eb-a1df-71a1e072b534-web-config\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.405862 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.405706 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/21aa6be9-3e79-42eb-a1df-71a1e072b534-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.405862 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.405786 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/21aa6be9-3e79-42eb-a1df-71a1e072b534-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.405862 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.405832 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21aa6be9-3e79-42eb-a1df-71a1e072b534-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.405862 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.405854 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/21aa6be9-3e79-42eb-a1df-71a1e072b534-tls-assets\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.406038 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.405885 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/21aa6be9-3e79-42eb-a1df-71a1e072b534-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.406038 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.405904 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/21aa6be9-3e79-42eb-a1df-71a1e072b534-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.406038 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.405946 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shmm5\" (UniqueName: \"kubernetes.io/projected/21aa6be9-3e79-42eb-a1df-71a1e072b534-kube-api-access-shmm5\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.406038 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.405967 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/21aa6be9-3e79-42eb-a1df-71a1e072b534-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.406038 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.405988 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/21aa6be9-3e79-42eb-a1df-71a1e072b534-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.406038 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.406020 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21aa6be9-3e79-42eb-a1df-71a1e072b534-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.506373 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.506340 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21aa6be9-3e79-42eb-a1df-71a1e072b534-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.506514 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.506380 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/21aa6be9-3e79-42eb-a1df-71a1e072b534-config-out\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.506514 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.506397 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/21aa6be9-3e79-42eb-a1df-71a1e072b534-config-volume\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.506514 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.506416 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/21aa6be9-3e79-42eb-a1df-71a1e072b534-web-config\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.506514 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.506458 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/21aa6be9-3e79-42eb-a1df-71a1e072b534-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.506514 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.506489 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/21aa6be9-3e79-42eb-a1df-71a1e072b534-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.506514 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.506511 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21aa6be9-3e79-42eb-a1df-71a1e072b534-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.506788 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.506541 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/21aa6be9-3e79-42eb-a1df-71a1e072b534-tls-assets\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.506788 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.506586 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/21aa6be9-3e79-42eb-a1df-71a1e072b534-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.506887 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.506858 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/21aa6be9-3e79-42eb-a1df-71a1e072b534-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.507347 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.507320 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/21aa6be9-3e79-42eb-a1df-71a1e072b534-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.507560 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.507538 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/21aa6be9-3e79-42eb-a1df-71a1e072b534-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.507689 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.507671 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shmm5\" (UniqueName: \"kubernetes.io/projected/21aa6be9-3e79-42eb-a1df-71a1e072b534-kube-api-access-shmm5\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.507805 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.507790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/21aa6be9-3e79-42eb-a1df-71a1e072b534-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.507900 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.507886 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/21aa6be9-3e79-42eb-a1df-71a1e072b534-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.509283 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.509174 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/21aa6be9-3e79-42eb-a1df-71a1e072b534-config-out\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.509576 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.507543 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/21aa6be9-3e79-42eb-a1df-71a1e072b534-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.509678 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.509572 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/21aa6be9-3e79-42eb-a1df-71a1e072b534-config-volume\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.509741 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.509726 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/21aa6be9-3e79-42eb-a1df-71a1e072b534-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.509887 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.509861 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/21aa6be9-3e79-42eb-a1df-71a1e072b534-web-config\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.510182 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.510165 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/21aa6be9-3e79-42eb-a1df-71a1e072b534-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.510339 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.510255 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/21aa6be9-3e79-42eb-a1df-71a1e072b534-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.510452 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.510438 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/21aa6be9-3e79-42eb-a1df-71a1e072b534-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.510793 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.510772 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/21aa6be9-3e79-42eb-a1df-71a1e072b534-tls-assets\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.511519 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.511502 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/21aa6be9-3e79-42eb-a1df-71a1e072b534-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.518603 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.518583 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shmm5\" (UniqueName: \"kubernetes.io/projected/21aa6be9-3e79-42eb-a1df-71a1e072b534-kube-api-access-shmm5\") pod \"alertmanager-main-0\" (UID: \"21aa6be9-3e79-42eb-a1df-71a1e072b534\") " pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.604065 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.603993 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 22 12:43:35.742763 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:35.742653 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 22 12:43:35.745418 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:43:35.745387 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21aa6be9_3e79_42eb_a1df_71a1e072b534.slice/crio-a6244ae9e1a7745f28d7fcba08687a16b2174a4d373d4de111de5287a6def409 WatchSource:0}: Error finding container a6244ae9e1a7745f28d7fcba08687a16b2174a4d373d4de111de5287a6def409: Status 404 returned error can't find the container with id a6244ae9e1a7745f28d7fcba08687a16b2174a4d373d4de111de5287a6def409 Jul 22 12:43:36.676748 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:36.676703 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"21aa6be9-3e79-42eb-a1df-71a1e072b534","Type":"ContainerStarted","Data":"a6244ae9e1a7745f28d7fcba08687a16b2174a4d373d4de111de5287a6def409"} Jul 22 12:43:37.682543 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:37.682508 2579 generic.go:356] "Generic (PLEG): container finished" podID="21aa6be9-3e79-42eb-a1df-71a1e072b534" containerID="af500387d5bff814327ec36dc7e3203c24db48dd7b524b878e2aa86feb919a1e" exitCode=0 Jul 22 12:43:37.682976 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:37.682595 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"21aa6be9-3e79-42eb-a1df-71a1e072b534","Type":"ContainerDied","Data":"af500387d5bff814327ec36dc7e3203c24db48dd7b524b878e2aa86feb919a1e"} Jul 22 12:43:39.694023 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:39.693988 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"21aa6be9-3e79-42eb-a1df-71a1e072b534","Type":"ContainerStarted","Data":"39d54f2bca79faf56902cf306ff1747fa804bec878ad4899123c63532bdad806"} Jul 22 12:43:39.694461 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:39.694110 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"21aa6be9-3e79-42eb-a1df-71a1e072b534","Type":"ContainerStarted","Data":"11c1bcecdffbaf217779e5c850d53f525a5417b2ee38e2f2681d928457f6e481"} Jul 22 12:43:39.694461 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:39.694159 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"21aa6be9-3e79-42eb-a1df-71a1e072b534","Type":"ContainerStarted","Data":"a6bed860c53b2f896a9ec8078c2eebf2273b9e74ae5d1dee445844f5dcca5277"} Jul 22 12:43:39.694461 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:39.694207 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"21aa6be9-3e79-42eb-a1df-71a1e072b534","Type":"ContainerStarted","Data":"08756edc5bca1d3bff855267f9b91c386d15dd1f8b33e621473a7d568ef1a375"} Jul 22 12:43:39.694461 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:39.694246 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"21aa6be9-3e79-42eb-a1df-71a1e072b534","Type":"ContainerStarted","Data":"3e244f7585740d4925ead8f039aec9fab21d475a2ec1fbb651426d752371221a"} Jul 22 12:43:39.694461 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:39.694285 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"21aa6be9-3e79-42eb-a1df-71a1e072b534","Type":"ContainerStarted","Data":"52ba1b24cb4249127dfb6ef81e35c48174124b0ee690b738c63a19f2301b4f30"} Jul 22 12:43:39.727060 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:39.727019 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.014271638 podStartE2EDuration="4.727002044s" totalImagesPullingTime="2.696072739s" totalInitContainerRuntime="16.657667ms" isStatefulPod=false podCreationTimestamp="2026-07-22 12:43:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:43:39.724529562 +0000 UTC m=+168.125760481" watchObservedRunningTime="2026-07-22 12:43:39.727002044 +0000 UTC m=+168.128232965" Jul 22 12:43:50.544688 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:50.544615 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-579f767c9c-btscm"] Jul 22 12:43:50.548068 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:50.548049 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:43:50.565592 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:50.565568 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-579f767c9c-btscm"] Jul 22 12:43:50.628361 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:50.628329 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-service-ca\") pod \"console-579f767c9c-btscm\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:43:50.628518 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:50.628368 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-console-config\") pod \"console-579f767c9c-btscm\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:43:50.628518 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:50.628393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb8tf\" (UniqueName: \"kubernetes.io/projected/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-kube-api-access-gb8tf\") pod \"console-579f767c9c-btscm\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:43:50.628518 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:50.628416 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-trusted-ca-bundle\") pod \"console-579f767c9c-btscm\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:43:50.628518 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:50.628455 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-console-serving-cert\") pod \"console-579f767c9c-btscm\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:43:50.628518 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:50.628506 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-oauth-serving-cert\") pod \"console-579f767c9c-btscm\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:43:50.628716 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:50.628538 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-console-oauth-config\") pod \"console-579f767c9c-btscm\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:43:50.729699 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:50.729663 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-console-serving-cert\") pod \"console-579f767c9c-btscm\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:43:50.729853 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:50.729705 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-oauth-serving-cert\") pod \"console-579f767c9c-btscm\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:43:50.729853 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:50.729726 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-console-oauth-config\") pod \"console-579f767c9c-btscm\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:43:50.729853 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:50.729780 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-service-ca\") pod \"console-579f767c9c-btscm\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:43:50.729853 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:50.729801 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-console-config\") pod \"console-579f767c9c-btscm\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:43:50.729853 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:50.729816 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb8tf\" (UniqueName: \"kubernetes.io/projected/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-kube-api-access-gb8tf\") pod \"console-579f767c9c-btscm\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:43:50.730133 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:50.729874 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-trusted-ca-bundle\") pod \"console-579f767c9c-btscm\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:43:50.730534 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:50.730510 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-console-config\") pod \"console-579f767c9c-btscm\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:43:50.730658 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:50.730535 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-oauth-serving-cert\") pod \"console-579f767c9c-btscm\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:43:50.730658 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:50.730616 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-service-ca\") pod \"console-579f767c9c-btscm\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:43:50.730782 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:50.730750 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-trusted-ca-bundle\") pod \"console-579f767c9c-btscm\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:43:50.732774 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:50.732748 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-console-oauth-config\") pod \"console-579f767c9c-btscm\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:43:50.732847 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:50.732822 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-console-serving-cert\") pod \"console-579f767c9c-btscm\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:43:50.738580 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:50.738558 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb8tf\" (UniqueName: \"kubernetes.io/projected/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-kube-api-access-gb8tf\") pod \"console-579f767c9c-btscm\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:43:50.857563 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:50.857475 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:43:50.985487 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:50.985461 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-579f767c9c-btscm"] Jul 22 12:43:50.987945 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:43:50.987894 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab830ca0_eab6_498f_a8d7_5743e1f84ecd.slice/crio-56fa0fd1fe5d2b11f9ee5f00298e497297338ba85e7aa19176c0c0ef1f4321db WatchSource:0}: Error finding container 56fa0fd1fe5d2b11f9ee5f00298e497297338ba85e7aa19176c0c0ef1f4321db: Status 404 returned error can't find the container with id 56fa0fd1fe5d2b11f9ee5f00298e497297338ba85e7aa19176c0c0ef1f4321db Jul 22 12:43:51.742556 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:51.742521 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579f767c9c-btscm" event={"ID":"ab830ca0-eab6-498f-a8d7-5743e1f84ecd","Type":"ContainerStarted","Data":"51376a26d5b17f62654db0d58ce25b104fa096b85c4e262b76c9da60ee32ab29"} Jul 22 12:43:51.742918 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:51.742604 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579f767c9c-btscm" event={"ID":"ab830ca0-eab6-498f-a8d7-5743e1f84ecd","Type":"ContainerStarted","Data":"56fa0fd1fe5d2b11f9ee5f00298e497297338ba85e7aa19176c0c0ef1f4321db"} Jul 22 12:43:51.763782 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:43:51.763731 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-579f767c9c-btscm" podStartSLOduration=1.763717244 podStartE2EDuration="1.763717244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:43:51.76254001 +0000 UTC m=+180.163770932" watchObservedRunningTime="2026-07-22 12:43:51.763717244 +0000 UTC m=+180.164948228" Jul 22 12:44:00.858442 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:00.858406 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:44:00.858865 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:00.858463 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:44:00.863164 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:00.863139 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:44:01.785849 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:01.785822 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:44:01.849159 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:01.849126 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65594785fd-bz47t"] Jul 22 12:44:26.873755 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:26.873716 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65594785fd-bz47t" podUID="b193e6c4-09bc-4462-bf47-0a43c2c22f3f" containerName="console" containerID="cri-o://59db68ba972a385c51acfd582b4c9dfa79dbcef0efe4c2b922a96f76382685d8" gracePeriod=15 Jul 22 12:44:27.127158 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.127094 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:44:27.241432 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.241401 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-console-serving-cert\") pod \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " Jul 22 12:44:27.241607 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.241449 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-service-ca\") pod \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " Jul 22 12:44:27.241607 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.241478 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-oauth-serving-cert\") pod \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " Jul 22 12:44:27.241607 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.241507 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-console-config\") pod \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " Jul 22 12:44:27.241812 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.241606 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-trusted-ca-bundle\") pod \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " Jul 22 12:44:27.241812 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.241650 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-console-oauth-config\") pod \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " Jul 22 12:44:27.241812 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.241692 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2p5fv\" (UniqueName: \"kubernetes.io/projected/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-kube-api-access-2p5fv\") pod \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\" (UID: \"b193e6c4-09bc-4462-bf47-0a43c2c22f3f\") " Jul 22 12:44:27.241984 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.241867 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-console-config" (OuterVolumeSpecName: "console-config") pod "b193e6c4-09bc-4462-bf47-0a43c2c22f3f" (UID: "b193e6c4-09bc-4462-bf47-0a43c2c22f3f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:44:27.242049 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.241991 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b193e6c4-09bc-4462-bf47-0a43c2c22f3f" (UID: "b193e6c4-09bc-4462-bf47-0a43c2c22f3f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:44:27.242103 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.242077 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b193e6c4-09bc-4462-bf47-0a43c2c22f3f" (UID: "b193e6c4-09bc-4462-bf47-0a43c2c22f3f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:44:27.242249 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.242228 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-oauth-serving-cert\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:44:27.242435 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.242349 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-service-ca" (OuterVolumeSpecName: "service-ca") pod "b193e6c4-09bc-4462-bf47-0a43c2c22f3f" (UID: "b193e6c4-09bc-4462-bf47-0a43c2c22f3f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:44:27.242435 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.242390 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-console-config\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:44:27.242610 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.242566 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-trusted-ca-bundle\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:44:27.243796 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.243770 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b193e6c4-09bc-4462-bf47-0a43c2c22f3f" (UID: "b193e6c4-09bc-4462-bf47-0a43c2c22f3f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:44:27.244254 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.244229 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b193e6c4-09bc-4462-bf47-0a43c2c22f3f" (UID: "b193e6c4-09bc-4462-bf47-0a43c2c22f3f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:44:27.244332 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.244246 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-kube-api-access-2p5fv" (OuterVolumeSpecName: "kube-api-access-2p5fv") pod "b193e6c4-09bc-4462-bf47-0a43c2c22f3f" (UID: "b193e6c4-09bc-4462-bf47-0a43c2c22f3f"). InnerVolumeSpecName "kube-api-access-2p5fv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:44:27.343751 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.343704 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p5fv\" (UniqueName: \"kubernetes.io/projected/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-kube-api-access-2p5fv\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:44:27.343751 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.343748 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-console-serving-cert\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:44:27.343751 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.343759 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-service-ca\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:44:27.343751 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.343768 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b193e6c4-09bc-4462-bf47-0a43c2c22f3f-console-oauth-config\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:44:27.881693 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.881661 2579 generic.go:356] "Generic (PLEG): container finished" podID="b193e6c4-09bc-4462-bf47-0a43c2c22f3f" containerID="59db68ba972a385c51acfd582b4c9dfa79dbcef0efe4c2b922a96f76382685d8" exitCode=0 Jul 22 12:44:27.882082 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.881704 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65594785fd-bz47t" Jul 22 12:44:27.882082 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.881742 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65594785fd-bz47t" event={"ID":"b193e6c4-09bc-4462-bf47-0a43c2c22f3f","Type":"ContainerDied","Data":"59db68ba972a385c51acfd582b4c9dfa79dbcef0efe4c2b922a96f76382685d8"} Jul 22 12:44:27.882082 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.881809 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65594785fd-bz47t" event={"ID":"b193e6c4-09bc-4462-bf47-0a43c2c22f3f","Type":"ContainerDied","Data":"0a955c5debf9bc11ff747679c7deb4224d73676d036c758bcd25f8a2cf3afa42"} Jul 22 12:44:27.882082 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.881830 2579 scope.go:117] "RemoveContainer" containerID="59db68ba972a385c51acfd582b4c9dfa79dbcef0efe4c2b922a96f76382685d8" Jul 22 12:44:27.890989 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.890972 2579 scope.go:117] "RemoveContainer" containerID="59db68ba972a385c51acfd582b4c9dfa79dbcef0efe4c2b922a96f76382685d8" Jul 22 12:44:27.891237 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:44:27.891218 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59db68ba972a385c51acfd582b4c9dfa79dbcef0efe4c2b922a96f76382685d8\": container with ID starting with 59db68ba972a385c51acfd582b4c9dfa79dbcef0efe4c2b922a96f76382685d8 not found: ID does not exist" containerID="59db68ba972a385c51acfd582b4c9dfa79dbcef0efe4c2b922a96f76382685d8" Jul 22 12:44:27.891281 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.891248 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59db68ba972a385c51acfd582b4c9dfa79dbcef0efe4c2b922a96f76382685d8"} err="failed to get container status \"59db68ba972a385c51acfd582b4c9dfa79dbcef0efe4c2b922a96f76382685d8\": rpc error: code = NotFound desc = could not find container \"59db68ba972a385c51acfd582b4c9dfa79dbcef0efe4c2b922a96f76382685d8\": container with ID starting with 59db68ba972a385c51acfd582b4c9dfa79dbcef0efe4c2b922a96f76382685d8 not found: ID does not exist" Jul 22 12:44:27.904702 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.904681 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65594785fd-bz47t"] Jul 22 12:44:27.908408 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:27.908388 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65594785fd-bz47t"] Jul 22 12:44:28.066419 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:28.066391 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b193e6c4-09bc-4462-bf47-0a43c2c22f3f" path="/var/lib/kubelet/pods/b193e6c4-09bc-4462-bf47-0a43c2c22f3f/volumes" Jul 22 12:44:42.051421 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.051390 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d746d875b-rwtms"] Jul 22 12:44:42.051819 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.051736 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b193e6c4-09bc-4462-bf47-0a43c2c22f3f" containerName="console" Jul 22 12:44:42.051819 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.051747 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b193e6c4-09bc-4462-bf47-0a43c2c22f3f" containerName="console" Jul 22 12:44:42.051819 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.051796 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b193e6c4-09bc-4462-bf47-0a43c2c22f3f" containerName="console" Jul 22 12:44:42.056190 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.056174 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d746d875b-rwtms" Jul 22 12:44:42.058915 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.058884 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 22 12:44:42.060460 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.060276 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-pzdf6\"" Jul 22 12:44:42.060460 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.060317 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 22 12:44:42.060460 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.060321 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 22 12:44:42.060686 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.060323 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 22 12:44:42.067295 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.067277 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d746d875b-rwtms"] Jul 22 12:44:42.151681 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.151655 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb"] Jul 22 12:44:42.155121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.155101 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb" Jul 22 12:44:42.157720 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.157701 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 22 12:44:42.157720 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.157717 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 22 12:44:42.159428 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.159390 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 22 12:44:42.159525 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.159512 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 22 12:44:42.162206 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.162186 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1c2b80c7-8895-4ea3-9e27-ffcd86b33c52-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6d746d875b-rwtms\" (UID: \"1c2b80c7-8895-4ea3-9e27-ffcd86b33c52\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d746d875b-rwtms" Jul 22 12:44:42.162462 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.162436 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wrsj\" (UniqueName: \"kubernetes.io/projected/1c2b80c7-8895-4ea3-9e27-ffcd86b33c52-kube-api-access-9wrsj\") pod \"managed-serviceaccount-addon-agent-6d746d875b-rwtms\" (UID: \"1c2b80c7-8895-4ea3-9e27-ffcd86b33c52\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d746d875b-rwtms" Jul 22 12:44:42.169343 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.169324 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb"] Jul 22 12:44:42.262952 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.262900 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzj4l\" (UniqueName: \"kubernetes.io/projected/65954e92-f21b-40dc-8987-3255ee324b60-kube-api-access-hzj4l\") pod \"cluster-proxy-proxy-agent-7b6464dbd5-p5lsb\" (UID: \"65954e92-f21b-40dc-8987-3255ee324b60\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb" Jul 22 12:44:42.263112 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.262959 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/65954e92-f21b-40dc-8987-3255ee324b60-hub\") pod \"cluster-proxy-proxy-agent-7b6464dbd5-p5lsb\" (UID: \"65954e92-f21b-40dc-8987-3255ee324b60\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb" Jul 22 12:44:42.263112 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.262995 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1c2b80c7-8895-4ea3-9e27-ffcd86b33c52-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6d746d875b-rwtms\" (UID: \"1c2b80c7-8895-4ea3-9e27-ffcd86b33c52\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d746d875b-rwtms" Jul 22 12:44:42.263112 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.263015 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/65954e92-f21b-40dc-8987-3255ee324b60-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7b6464dbd5-p5lsb\" (UID: \"65954e92-f21b-40dc-8987-3255ee324b60\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb" Jul 22 12:44:42.263112 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.263070 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/65954e92-f21b-40dc-8987-3255ee324b60-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7b6464dbd5-p5lsb\" (UID: \"65954e92-f21b-40dc-8987-3255ee324b60\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb" Jul 22 12:44:42.263322 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.263112 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wrsj\" (UniqueName: \"kubernetes.io/projected/1c2b80c7-8895-4ea3-9e27-ffcd86b33c52-kube-api-access-9wrsj\") pod \"managed-serviceaccount-addon-agent-6d746d875b-rwtms\" (UID: \"1c2b80c7-8895-4ea3-9e27-ffcd86b33c52\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d746d875b-rwtms" Jul 22 12:44:42.263322 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.263160 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/65954e92-f21b-40dc-8987-3255ee324b60-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7b6464dbd5-p5lsb\" (UID: \"65954e92-f21b-40dc-8987-3255ee324b60\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb" Jul 22 12:44:42.263322 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.263184 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/65954e92-f21b-40dc-8987-3255ee324b60-ca\") pod \"cluster-proxy-proxy-agent-7b6464dbd5-p5lsb\" (UID: \"65954e92-f21b-40dc-8987-3255ee324b60\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb" Jul 22 12:44:42.265372 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.265351 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1c2b80c7-8895-4ea3-9e27-ffcd86b33c52-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6d746d875b-rwtms\" (UID: \"1c2b80c7-8895-4ea3-9e27-ffcd86b33c52\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d746d875b-rwtms" Jul 22 12:44:42.272188 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.272169 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wrsj\" (UniqueName: \"kubernetes.io/projected/1c2b80c7-8895-4ea3-9e27-ffcd86b33c52-kube-api-access-9wrsj\") pod \"managed-serviceaccount-addon-agent-6d746d875b-rwtms\" (UID: \"1c2b80c7-8895-4ea3-9e27-ffcd86b33c52\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d746d875b-rwtms" Jul 22 12:44:42.364499 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.364427 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/65954e92-f21b-40dc-8987-3255ee324b60-hub\") pod \"cluster-proxy-proxy-agent-7b6464dbd5-p5lsb\" (UID: \"65954e92-f21b-40dc-8987-3255ee324b60\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb" Jul 22 12:44:42.364499 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.364463 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/65954e92-f21b-40dc-8987-3255ee324b60-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7b6464dbd5-p5lsb\" (UID: \"65954e92-f21b-40dc-8987-3255ee324b60\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb" Jul 22 12:44:42.364499 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.364488 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/65954e92-f21b-40dc-8987-3255ee324b60-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7b6464dbd5-p5lsb\" (UID: \"65954e92-f21b-40dc-8987-3255ee324b60\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb" Jul 22 12:44:42.364690 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.364525 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/65954e92-f21b-40dc-8987-3255ee324b60-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7b6464dbd5-p5lsb\" (UID: \"65954e92-f21b-40dc-8987-3255ee324b60\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb" Jul 22 12:44:42.364690 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.364548 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/65954e92-f21b-40dc-8987-3255ee324b60-ca\") pod \"cluster-proxy-proxy-agent-7b6464dbd5-p5lsb\" (UID: \"65954e92-f21b-40dc-8987-3255ee324b60\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb" Jul 22 12:44:42.364690 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.364581 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzj4l\" (UniqueName: \"kubernetes.io/projected/65954e92-f21b-40dc-8987-3255ee324b60-kube-api-access-hzj4l\") pod \"cluster-proxy-proxy-agent-7b6464dbd5-p5lsb\" (UID: \"65954e92-f21b-40dc-8987-3255ee324b60\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb" Jul 22 12:44:42.365346 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.365318 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/65954e92-f21b-40dc-8987-3255ee324b60-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7b6464dbd5-p5lsb\" (UID: \"65954e92-f21b-40dc-8987-3255ee324b60\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb" Jul 22 12:44:42.366911 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.366890 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/65954e92-f21b-40dc-8987-3255ee324b60-ca\") pod \"cluster-proxy-proxy-agent-7b6464dbd5-p5lsb\" (UID: \"65954e92-f21b-40dc-8987-3255ee324b60\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb" Jul 22 12:44:42.367041 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.367023 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/65954e92-f21b-40dc-8987-3255ee324b60-hub\") pod \"cluster-proxy-proxy-agent-7b6464dbd5-p5lsb\" (UID: \"65954e92-f21b-40dc-8987-3255ee324b60\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb" Jul 22 12:44:42.367181 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.367164 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/65954e92-f21b-40dc-8987-3255ee324b60-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7b6464dbd5-p5lsb\" (UID: \"65954e92-f21b-40dc-8987-3255ee324b60\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb" Jul 22 12:44:42.367216 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.367199 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/65954e92-f21b-40dc-8987-3255ee324b60-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7b6464dbd5-p5lsb\" (UID: \"65954e92-f21b-40dc-8987-3255ee324b60\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb" Jul 22 12:44:42.373201 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.373185 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzj4l\" (UniqueName: \"kubernetes.io/projected/65954e92-f21b-40dc-8987-3255ee324b60-kube-api-access-hzj4l\") pod \"cluster-proxy-proxy-agent-7b6464dbd5-p5lsb\" (UID: \"65954e92-f21b-40dc-8987-3255ee324b60\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb" Jul 22 12:44:42.374040 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.374013 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d746d875b-rwtms" Jul 22 12:44:42.463671 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.463629 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb" Jul 22 12:44:42.504539 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.504503 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d746d875b-rwtms"] Jul 22 12:44:42.507411 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:44:42.507380 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c2b80c7_8895_4ea3_9e27_ffcd86b33c52.slice/crio-aa2371288e7b0176f14a5c52be7ca659bc4eeeb40ab999561254956fa47d1cb8 WatchSource:0}: Error finding container aa2371288e7b0176f14a5c52be7ca659bc4eeeb40ab999561254956fa47d1cb8: Status 404 returned error can't find the container with id aa2371288e7b0176f14a5c52be7ca659bc4eeeb40ab999561254956fa47d1cb8 Jul 22 12:44:42.600067 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.600042 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb"] Jul 22 12:44:42.603176 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:44:42.603152 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65954e92_f21b_40dc_8987_3255ee324b60.slice/crio-881b893a280c893d129bf1af2ae98af88e5bb9046c66f5dee975bbca0f636fd1 WatchSource:0}: Error finding container 881b893a280c893d129bf1af2ae98af88e5bb9046c66f5dee975bbca0f636fd1: Status 404 returned error can't find the container with id 881b893a280c893d129bf1af2ae98af88e5bb9046c66f5dee975bbca0f636fd1 Jul 22 12:44:42.942154 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.942120 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb" event={"ID":"65954e92-f21b-40dc-8987-3255ee324b60","Type":"ContainerStarted","Data":"881b893a280c893d129bf1af2ae98af88e5bb9046c66f5dee975bbca0f636fd1"} Jul 22 12:44:42.943099 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:42.943069 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d746d875b-rwtms" event={"ID":"1c2b80c7-8895-4ea3-9e27-ffcd86b33c52","Type":"ContainerStarted","Data":"aa2371288e7b0176f14a5c52be7ca659bc4eeeb40ab999561254956fa47d1cb8"} Jul 22 12:44:46.964947 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:46.964887 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d746d875b-rwtms" event={"ID":"1c2b80c7-8895-4ea3-9e27-ffcd86b33c52","Type":"ContainerStarted","Data":"2ac0789b19929756fbb98b98959c649def754cc29c84cf1c880d659b0e958336"} Jul 22 12:44:46.966817 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:46.966794 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb" event={"ID":"65954e92-f21b-40dc-8987-3255ee324b60","Type":"ContainerStarted","Data":"c046b39cb19ba07467ce2e13396bad9a27ac12953606c04c58803008184a6682"} Jul 22 12:44:46.966881 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:46.966832 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb" event={"ID":"65954e92-f21b-40dc-8987-3255ee324b60","Type":"ContainerStarted","Data":"4f998b667b29a69239a22f610226d3558f3523571110cb335af46c4693966e77"} Jul 22 12:44:46.966881 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:46.966850 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb" event={"ID":"65954e92-f21b-40dc-8987-3255ee324b60","Type":"ContainerStarted","Data":"207361c8f0452cd65f44b05345f7e34366283a13fa388a871e7919d9b3765de3"} Jul 22 12:44:46.982144 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:46.982101 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6d746d875b-rwtms" podStartSLOduration=1.059317526 podStartE2EDuration="4.982086918s" totalImagesPullingTime="3.922769392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:44:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:44:46.981257457 +0000 UTC m=+235.382488389" watchObservedRunningTime="2026-07-22 12:44:46.982086918 +0000 UTC m=+235.383317837" Jul 22 12:44:47.000132 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:44:47.000091 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7b6464dbd5-p5lsb" podStartSLOduration=1.16195914 podStartE2EDuration="5.000076021s" totalImagesPullingTime="3.838116881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:44:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:44:46.999163654 +0000 UTC m=+235.400394576" watchObservedRunningTime="2026-07-22 12:44:47.000076021 +0000 UTC m=+235.401306961" Jul 22 12:45:51.908947 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:45:51.908904 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ddj6_b28e0ca4-8512-45da-91ed-2cde1491b2e2/ovn-acl-logging/0.log" Jul 22 12:45:51.915596 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:45:51.915573 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ddj6_b28e0ca4-8512-45da-91ed-2cde1491b2e2/ovn-acl-logging/0.log" Jul 22 12:45:51.915735 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:45:51.915684 2579 kubelet.go:1648] "Image garbage collection succeeded" Jul 22 12:46:17.594916 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:17.594884 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64bbff868f-rt2tl"] Jul 22 12:46:17.598100 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:17.598085 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-rt2tl" Jul 22 12:46:17.600619 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:17.600599 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-nqhtb\"" Jul 22 12:46:17.601068 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:17.601053 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 22 12:46:17.603030 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:17.603006 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 22 12:46:17.603122 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:17.603108 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 22 12:46:17.603187 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:17.603144 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 22 12:46:17.614569 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:17.614548 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64bbff868f-rt2tl"] Jul 22 12:46:17.760314 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:17.760275 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e9c73605-9677-46a0-a88e-25e22d8428b6-manifests\") pod \"opendatahub-operator-controller-manager-64bbff868f-rt2tl\" (UID: \"e9c73605-9677-46a0-a88e-25e22d8428b6\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-rt2tl" Jul 22 12:46:17.760479 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:17.760354 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e9c73605-9677-46a0-a88e-25e22d8428b6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64bbff868f-rt2tl\" (UID: \"e9c73605-9677-46a0-a88e-25e22d8428b6\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-rt2tl" Jul 22 12:46:17.760479 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:17.760375 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9c73605-9677-46a0-a88e-25e22d8428b6-webhook-cert\") pod \"opendatahub-operator-controller-manager-64bbff868f-rt2tl\" (UID: \"e9c73605-9677-46a0-a88e-25e22d8428b6\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-rt2tl" Jul 22 12:46:17.760479 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:17.760397 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nbd5\" (UniqueName: \"kubernetes.io/projected/e9c73605-9677-46a0-a88e-25e22d8428b6-kube-api-access-8nbd5\") pod \"opendatahub-operator-controller-manager-64bbff868f-rt2tl\" (UID: \"e9c73605-9677-46a0-a88e-25e22d8428b6\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-rt2tl" Jul 22 12:46:17.760479 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:17.760445 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9c73605-9677-46a0-a88e-25e22d8428b6-tmp\") pod \"opendatahub-operator-controller-manager-64bbff868f-rt2tl\" (UID: \"e9c73605-9677-46a0-a88e-25e22d8428b6\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-rt2tl" Jul 22 12:46:17.861516 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:17.861423 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e9c73605-9677-46a0-a88e-25e22d8428b6-manifests\") pod \"opendatahub-operator-controller-manager-64bbff868f-rt2tl\" (UID: \"e9c73605-9677-46a0-a88e-25e22d8428b6\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-rt2tl" Jul 22 12:46:17.861516 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:17.861491 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e9c73605-9677-46a0-a88e-25e22d8428b6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64bbff868f-rt2tl\" (UID: \"e9c73605-9677-46a0-a88e-25e22d8428b6\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-rt2tl" Jul 22 12:46:17.861999 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:17.861536 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9c73605-9677-46a0-a88e-25e22d8428b6-webhook-cert\") pod \"opendatahub-operator-controller-manager-64bbff868f-rt2tl\" (UID: \"e9c73605-9677-46a0-a88e-25e22d8428b6\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-rt2tl" Jul 22 12:46:17.861999 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:17.861567 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nbd5\" (UniqueName: \"kubernetes.io/projected/e9c73605-9677-46a0-a88e-25e22d8428b6-kube-api-access-8nbd5\") pod \"opendatahub-operator-controller-manager-64bbff868f-rt2tl\" (UID: \"e9c73605-9677-46a0-a88e-25e22d8428b6\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-rt2tl" Jul 22 12:46:17.861999 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:17.861595 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9c73605-9677-46a0-a88e-25e22d8428b6-tmp\") pod \"opendatahub-operator-controller-manager-64bbff868f-rt2tl\" (UID: \"e9c73605-9677-46a0-a88e-25e22d8428b6\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-rt2tl" Jul 22 12:46:17.861999 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:17.861902 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e9c73605-9677-46a0-a88e-25e22d8428b6-manifests\") pod \"opendatahub-operator-controller-manager-64bbff868f-rt2tl\" (UID: \"e9c73605-9677-46a0-a88e-25e22d8428b6\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-rt2tl" Jul 22 12:46:17.864046 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:17.864005 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e9c73605-9677-46a0-a88e-25e22d8428b6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64bbff868f-rt2tl\" (UID: \"e9c73605-9677-46a0-a88e-25e22d8428b6\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-rt2tl" Jul 22 12:46:17.864180 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:17.864149 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9c73605-9677-46a0-a88e-25e22d8428b6-tmp\") pod \"opendatahub-operator-controller-manager-64bbff868f-rt2tl\" (UID: \"e9c73605-9677-46a0-a88e-25e22d8428b6\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-rt2tl" Jul 22 12:46:17.864483 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:17.864460 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9c73605-9677-46a0-a88e-25e22d8428b6-webhook-cert\") pod \"opendatahub-operator-controller-manager-64bbff868f-rt2tl\" (UID: \"e9c73605-9677-46a0-a88e-25e22d8428b6\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-rt2tl" Jul 22 12:46:17.870392 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:17.870374 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nbd5\" (UniqueName: \"kubernetes.io/projected/e9c73605-9677-46a0-a88e-25e22d8428b6-kube-api-access-8nbd5\") pod \"opendatahub-operator-controller-manager-64bbff868f-rt2tl\" (UID: \"e9c73605-9677-46a0-a88e-25e22d8428b6\") " pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-rt2tl" Jul 22 12:46:17.908402 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:17.908378 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-rt2tl" Jul 22 12:46:18.056900 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:18.056733 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64bbff868f-rt2tl"] Jul 22 12:46:18.059485 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:46:18.059449 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c73605_9677_46a0_a88e_25e22d8428b6.slice/crio-afff84fefbffd219ea3ed05e03e12d52cc119c0b4b84f74bc5f029ad0c935abf WatchSource:0}: Error finding container afff84fefbffd219ea3ed05e03e12d52cc119c0b4b84f74bc5f029ad0c935abf: Status 404 returned error can't find the container with id afff84fefbffd219ea3ed05e03e12d52cc119c0b4b84f74bc5f029ad0c935abf Jul 22 12:46:18.061293 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:18.061276 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 12:46:18.316712 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:18.316683 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-rt2tl" event={"ID":"e9c73605-9677-46a0-a88e-25e22d8428b6","Type":"ContainerStarted","Data":"afff84fefbffd219ea3ed05e03e12d52cc119c0b4b84f74bc5f029ad0c935abf"} Jul 22 12:46:21.336298 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:21.336256 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-rt2tl" event={"ID":"e9c73605-9677-46a0-a88e-25e22d8428b6","Type":"ContainerStarted","Data":"e09f424ec721a6320c6d56bffe5362459ec6961b8563a09e1573d6896277773d"} Jul 22 12:46:22.341627 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:22.341543 2579 generic.go:356] "Generic (PLEG): container finished" podID="e9c73605-9677-46a0-a88e-25e22d8428b6" containerID="e09f424ec721a6320c6d56bffe5362459ec6961b8563a09e1573d6896277773d" exitCode=0 Jul 22 12:46:22.341627 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:22.341601 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-rt2tl" event={"ID":"e9c73605-9677-46a0-a88e-25e22d8428b6","Type":"ContainerDied","Data":"e09f424ec721a6320c6d56bffe5362459ec6961b8563a09e1573d6896277773d"} Jul 22 12:46:23.347449 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:23.347416 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-rt2tl" event={"ID":"e9c73605-9677-46a0-a88e-25e22d8428b6","Type":"ContainerStarted","Data":"8b75b216bd436609216d68df9bf46a2e8041d1de09db934c5ffc43361b5f8b49"} Jul 22 12:46:23.347984 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:23.347528 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-rt2tl" Jul 22 12:46:23.382450 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:23.382397 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-rt2tl" podStartSLOduration=1.952312788 podStartE2EDuration="6.38238297s" totalImagesPullingTime="3.39813911s" totalInitContainerRuntime="1.031931072s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:46:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:46:23.380440668 +0000 UTC m=+331.781671589" watchObservedRunningTime="2026-07-22 12:46:23.38238297 +0000 UTC m=+331.783613890" Jul 22 12:46:34.354066 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:34.354033 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-64bbff868f-rt2tl" Jul 22 12:46:39.516729 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.516692 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-lf29h"] Jul 22 12:46:39.526442 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.526421 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" Jul 22 12:46:39.536959 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.536935 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 22 12:46:39.537054 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.536935 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 22 12:46:39.537093 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.537062 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 22 12:46:39.537211 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.537198 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-m6lkm\"" Jul 22 12:46:39.537334 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.537321 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 22 12:46:39.537373 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.537343 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 22 12:46:39.537408 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.537384 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 22 12:46:39.544676 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.544647 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-lf29h"] Jul 22 12:46:39.648812 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.648784 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-cacerts\") pod \"istiod-openshift-gateway-7574459656-lf29h\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" Jul 22 12:46:39.648986 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.648834 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-lf29h\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" Jul 22 12:46:39.648986 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.648861 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsjvt\" (UniqueName: \"kubernetes.io/projected/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-kube-api-access-jsjvt\") pod \"istiod-openshift-gateway-7574459656-lf29h\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" Jul 22 12:46:39.648986 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.648879 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-local-certs\") pod \"istiod-openshift-gateway-7574459656-lf29h\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" Jul 22 12:46:39.648986 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.648942 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-lf29h\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" Jul 22 12:46:39.649131 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.649004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-lf29h\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" Jul 22 12:46:39.649131 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.649041 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-istio-token\") pod \"istiod-openshift-gateway-7574459656-lf29h\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" Jul 22 12:46:39.749812 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.749775 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-istio-token\") pod \"istiod-openshift-gateway-7574459656-lf29h\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" Jul 22 12:46:39.750002 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.749831 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-cacerts\") pod \"istiod-openshift-gateway-7574459656-lf29h\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" Jul 22 12:46:39.750002 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.749865 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-lf29h\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" Jul 22 12:46:39.750002 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.749901 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsjvt\" (UniqueName: \"kubernetes.io/projected/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-kube-api-access-jsjvt\") pod \"istiod-openshift-gateway-7574459656-lf29h\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" Jul 22 12:46:39.750002 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.749918 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-local-certs\") pod \"istiod-openshift-gateway-7574459656-lf29h\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" Jul 22 12:46:39.750002 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.749961 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-lf29h\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" Jul 22 12:46:39.750295 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.750014 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-lf29h\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" Jul 22 12:46:39.750687 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.750665 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-lf29h\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" Jul 22 12:46:39.752275 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.752252 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-local-certs\") pod \"istiod-openshift-gateway-7574459656-lf29h\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" Jul 22 12:46:39.752380 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.752367 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-cacerts\") pod \"istiod-openshift-gateway-7574459656-lf29h\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" Jul 22 12:46:39.752754 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.752735 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-lf29h\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" Jul 22 12:46:39.752822 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.752737 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-lf29h\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" Jul 22 12:46:39.762034 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.762005 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsjvt\" (UniqueName: \"kubernetes.io/projected/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-kube-api-access-jsjvt\") pod \"istiod-openshift-gateway-7574459656-lf29h\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" Jul 22 12:46:39.762190 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.762170 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-istio-token\") pod \"istiod-openshift-gateway-7574459656-lf29h\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" Jul 22 12:46:39.834800 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.834771 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" Jul 22 12:46:39.985392 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:39.985359 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-lf29h"] Jul 22 12:46:39.988839 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:46:39.988814 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565a97c4_c2e6_4a0b_b58c_2e40fb4b9084.slice/crio-b7cffad371ee55761c270f04740161ecc220aeb0d95b00d55eacae713c55a845 WatchSource:0}: Error finding container b7cffad371ee55761c270f04740161ecc220aeb0d95b00d55eacae713c55a845: Status 404 returned error can't find the container with id b7cffad371ee55761c270f04740161ecc220aeb0d95b00d55eacae713c55a845 Jul 22 12:46:40.417549 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:40.417509 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" event={"ID":"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084","Type":"ContainerStarted","Data":"b7cffad371ee55761c270f04740161ecc220aeb0d95b00d55eacae713c55a845"} Jul 22 12:46:42.716732 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:42.716681 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 12:46:42.717026 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:42.716779 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 22 12:46:43.440907 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:43.440865 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" event={"ID":"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084","Type":"ContainerStarted","Data":"62a0e0a385cd656ca17275d12048c21b1324f29b98aa1e72afbe578c1ba2e662"} Jul 22 12:46:43.441132 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:43.441064 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" Jul 22 12:46:43.442846 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:43.442817 2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-lf29h container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 12:46:43.442994 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:43.442873 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" podUID="565a97c4-c2e6-4a0b-b58c-2e40fb4b9084" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 12:46:43.478823 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:43.478767 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" podStartSLOduration=1.753103582 podStartE2EDuration="4.478751282s" totalImagesPullingTime="2.7256477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:46:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:46:43.47724137 +0000 UTC m=+351.878472291" watchObservedRunningTime="2026-07-22 12:46:43.478751282 +0000 UTC m=+351.879982201" Jul 22 12:46:44.445325 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:44.445236 2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-lf29h container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 22 12:46:44.445325 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:44.445288 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" podUID="565a97c4-c2e6-4a0b-b58c-2e40fb4b9084" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 22 12:46:45.568418 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:45.568385 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8b6495d78-grmzq"] Jul 22 12:46:45.571609 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:45.571591 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8b6495d78-grmzq" Jul 22 12:46:45.574838 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:46:45.574810 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"metrics-server-cert\" is forbidden: User \"system:node:ip-10-0-129-34.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-lws-operator\": no relationship found between node 'ip-10-0-129-34.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" type="*v1.Secret" Jul 22 12:46:45.576371 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:46:45.576341 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"lws-manager-config\" is forbidden: User \"system:node:ip-10-0-129-34.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-lws-operator\": no relationship found between node 'ip-10-0-129-34.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" type="*v1.ConfigMap" Jul 22 12:46:45.576371 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:46:45.576341 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"webhook-server-cert\" is forbidden: User \"system:node:ip-10-0-129-34.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-lws-operator\": no relationship found between node 'ip-10-0-129-34.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" type="*v1.Secret" Jul 22 12:46:45.576512 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:46:45.576379 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-129-34.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-lws-operator\": no relationship found between node 'ip-10-0-129-34.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 22 12:46:45.576599 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:46:45.576578 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"lws-controller-manager-8b6495d78-grmzq\" is forbidden: User \"system:node:ip-10-0-129-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-lws-operator\": no relationship found between node 'ip-10-0-129-34.ec2.internal' and this object" podUID="e04e2273-7cc5-46b0-9bb5-0506c6c5af46" pod="openshift-lws-operator/lws-controller-manager-8b6495d78-grmzq" Jul 22 12:46:45.577466 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:46:45.577441 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-129-34.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-lws-operator\": no relationship found between node 'ip-10-0-129-34.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 22 12:46:45.578787 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:46:45.578760 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"lws-controller-manager-dockercfg-gvbqk\" is forbidden: User \"system:node:ip-10-0-129-34.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-lws-operator\": no relationship found between node 'ip-10-0-129-34.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-gvbqk\"" type="*v1.Secret" Jul 22 12:46:45.593773 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:45.593748 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8b6495d78-grmzq"] Jul 22 12:46:45.704847 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:45.704817 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e04e2273-7cc5-46b0-9bb5-0506c6c5af46-metrics-cert\") pod \"lws-controller-manager-8b6495d78-grmzq\" (UID: \"e04e2273-7cc5-46b0-9bb5-0506c6c5af46\") " pod="openshift-lws-operator/lws-controller-manager-8b6495d78-grmzq" Jul 22 12:46:45.704847 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:45.704848 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fpxc\" (UniqueName: \"kubernetes.io/projected/e04e2273-7cc5-46b0-9bb5-0506c6c5af46-kube-api-access-4fpxc\") pod \"lws-controller-manager-8b6495d78-grmzq\" (UID: \"e04e2273-7cc5-46b0-9bb5-0506c6c5af46\") " pod="openshift-lws-operator/lws-controller-manager-8b6495d78-grmzq" Jul 22 12:46:45.705073 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:45.704868 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e04e2273-7cc5-46b0-9bb5-0506c6c5af46-cert\") pod \"lws-controller-manager-8b6495d78-grmzq\" (UID: \"e04e2273-7cc5-46b0-9bb5-0506c6c5af46\") " pod="openshift-lws-operator/lws-controller-manager-8b6495d78-grmzq" Jul 22 12:46:45.705073 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:45.704993 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e04e2273-7cc5-46b0-9bb5-0506c6c5af46-manager-config\") pod \"lws-controller-manager-8b6495d78-grmzq\" (UID: \"e04e2273-7cc5-46b0-9bb5-0506c6c5af46\") " pod="openshift-lws-operator/lws-controller-manager-8b6495d78-grmzq" Jul 22 12:46:45.806197 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:45.806161 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e04e2273-7cc5-46b0-9bb5-0506c6c5af46-manager-config\") pod \"lws-controller-manager-8b6495d78-grmzq\" (UID: \"e04e2273-7cc5-46b0-9bb5-0506c6c5af46\") " pod="openshift-lws-operator/lws-controller-manager-8b6495d78-grmzq" Jul 22 12:46:45.806385 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:45.806220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e04e2273-7cc5-46b0-9bb5-0506c6c5af46-metrics-cert\") pod \"lws-controller-manager-8b6495d78-grmzq\" (UID: \"e04e2273-7cc5-46b0-9bb5-0506c6c5af46\") " pod="openshift-lws-operator/lws-controller-manager-8b6495d78-grmzq" Jul 22 12:46:45.806385 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:45.806237 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fpxc\" (UniqueName: \"kubernetes.io/projected/e04e2273-7cc5-46b0-9bb5-0506c6c5af46-kube-api-access-4fpxc\") pod \"lws-controller-manager-8b6495d78-grmzq\" (UID: \"e04e2273-7cc5-46b0-9bb5-0506c6c5af46\") " pod="openshift-lws-operator/lws-controller-manager-8b6495d78-grmzq" Jul 22 12:46:45.806385 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:45.806260 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e04e2273-7cc5-46b0-9bb5-0506c6c5af46-cert\") pod \"lws-controller-manager-8b6495d78-grmzq\" (UID: \"e04e2273-7cc5-46b0-9bb5-0506c6c5af46\") " pod="openshift-lws-operator/lws-controller-manager-8b6495d78-grmzq" Jul 22 12:46:46.594616 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:46.594579 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 22 12:46:46.599067 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:46.599040 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e04e2273-7cc5-46b0-9bb5-0506c6c5af46-metrics-cert\") pod \"lws-controller-manager-8b6495d78-grmzq\" (UID: \"e04e2273-7cc5-46b0-9bb5-0506c6c5af46\") " pod="openshift-lws-operator/lws-controller-manager-8b6495d78-grmzq" Jul 22 12:46:46.748357 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:46.748325 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 22 12:46:46.806611 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:46:46.806573 2579 configmap.go:193] Couldn't get configMap openshift-lws-operator/lws-manager-config: failed to sync configmap cache: timed out waiting for the condition Jul 22 12:46:46.806767 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:46:46.806678 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e04e2273-7cc5-46b0-9bb5-0506c6c5af46-manager-config podName:e04e2273-7cc5-46b0-9bb5-0506c6c5af46 nodeName:}" failed. No retries permitted until 2026-07-22 12:46:47.306656712 +0000 UTC m=+355.707887624 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "manager-config" (UniqueName: "kubernetes.io/configmap/e04e2273-7cc5-46b0-9bb5-0506c6c5af46-manager-config") pod "lws-controller-manager-8b6495d78-grmzq" (UID: "e04e2273-7cc5-46b0-9bb5-0506c6c5af46") : failed to sync configmap cache: timed out waiting for the condition Jul 22 12:46:46.806767 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:46:46.806589 2579 secret.go:189] Couldn't get secret openshift-lws-operator/webhook-server-cert: failed to sync secret cache: timed out waiting for the condition Jul 22 12:46:46.806767 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:46:46.806714 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e04e2273-7cc5-46b0-9bb5-0506c6c5af46-cert podName:e04e2273-7cc5-46b0-9bb5-0506c6c5af46 nodeName:}" failed. No retries permitted until 2026-07-22 12:46:47.306707733 +0000 UTC m=+355.707938631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e04e2273-7cc5-46b0-9bb5-0506c6c5af46-cert") pod "lws-controller-manager-8b6495d78-grmzq" (UID: "e04e2273-7cc5-46b0-9bb5-0506c6c5af46") : failed to sync secret cache: timed out waiting for the condition Jul 22 12:46:46.821613 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:46:46.821579 2579 projected.go:291] Couldn't get configMap openshift-lws-operator/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition Jul 22 12:46:46.821762 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:46:46.821621 2579 projected.go:196] Error preparing data for projected volume kube-api-access-4fpxc for pod openshift-lws-operator/lws-controller-manager-8b6495d78-grmzq: failed to sync configmap cache: timed out waiting for the condition Jul 22 12:46:46.821762 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:46:46.821700 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e04e2273-7cc5-46b0-9bb5-0506c6c5af46-kube-api-access-4fpxc podName:e04e2273-7cc5-46b0-9bb5-0506c6c5af46 nodeName:}" failed. No retries permitted until 2026-07-22 12:46:47.321679068 +0000 UTC m=+355.722909966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4fpxc" (UniqueName: "kubernetes.io/projected/e04e2273-7cc5-46b0-9bb5-0506c6c5af46-kube-api-access-4fpxc") pod "lws-controller-manager-8b6495d78-grmzq" (UID: "e04e2273-7cc5-46b0-9bb5-0506c6c5af46") : failed to sync configmap cache: timed out waiting for the condition Jul 22 12:46:46.839555 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:46.839524 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-gvbqk\"" Jul 22 12:46:46.858708 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:46.858635 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 22 12:46:46.871380 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:46.871352 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 22 12:46:46.932023 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:46.931990 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 22 12:46:47.322646 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:47.322619 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fpxc\" (UniqueName: \"kubernetes.io/projected/e04e2273-7cc5-46b0-9bb5-0506c6c5af46-kube-api-access-4fpxc\") pod \"lws-controller-manager-8b6495d78-grmzq\" (UID: \"e04e2273-7cc5-46b0-9bb5-0506c6c5af46\") " pod="openshift-lws-operator/lws-controller-manager-8b6495d78-grmzq" Jul 22 12:46:47.322801 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:47.322654 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e04e2273-7cc5-46b0-9bb5-0506c6c5af46-cert\") pod \"lws-controller-manager-8b6495d78-grmzq\" (UID: \"e04e2273-7cc5-46b0-9bb5-0506c6c5af46\") " pod="openshift-lws-operator/lws-controller-manager-8b6495d78-grmzq" Jul 22 12:46:47.322801 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:47.322707 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e04e2273-7cc5-46b0-9bb5-0506c6c5af46-manager-config\") pod \"lws-controller-manager-8b6495d78-grmzq\" (UID: \"e04e2273-7cc5-46b0-9bb5-0506c6c5af46\") " pod="openshift-lws-operator/lws-controller-manager-8b6495d78-grmzq" Jul 22 12:46:47.323285 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:47.323264 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e04e2273-7cc5-46b0-9bb5-0506c6c5af46-manager-config\") pod \"lws-controller-manager-8b6495d78-grmzq\" (UID: \"e04e2273-7cc5-46b0-9bb5-0506c6c5af46\") " pod="openshift-lws-operator/lws-controller-manager-8b6495d78-grmzq" Jul 22 12:46:47.325097 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:47.325066 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fpxc\" (UniqueName: \"kubernetes.io/projected/e04e2273-7cc5-46b0-9bb5-0506c6c5af46-kube-api-access-4fpxc\") pod \"lws-controller-manager-8b6495d78-grmzq\" (UID: \"e04e2273-7cc5-46b0-9bb5-0506c6c5af46\") " pod="openshift-lws-operator/lws-controller-manager-8b6495d78-grmzq" Jul 22 12:46:47.325097 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:47.325078 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e04e2273-7cc5-46b0-9bb5-0506c6c5af46-cert\") pod \"lws-controller-manager-8b6495d78-grmzq\" (UID: \"e04e2273-7cc5-46b0-9bb5-0506c6c5af46\") " pod="openshift-lws-operator/lws-controller-manager-8b6495d78-grmzq" Jul 22 12:46:47.380516 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:47.380482 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8b6495d78-grmzq" Jul 22 12:46:47.445776 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:47.445747 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" Jul 22 12:46:47.574666 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:47.574605 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8b6495d78-grmzq"] Jul 22 12:46:47.577374 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:46:47.577352 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04e2273_7cc5_46b0_9bb5_0506c6c5af46.slice/crio-6f7fe5a3e73831831f8522acea671aa4c67bd23f43d1b990a8178597d17a14ec WatchSource:0}: Error finding container 6f7fe5a3e73831831f8522acea671aa4c67bd23f43d1b990a8178597d17a14ec: Status 404 returned error can't find the container with id 6f7fe5a3e73831831f8522acea671aa4c67bd23f43d1b990a8178597d17a14ec Jul 22 12:46:48.464804 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:48.464728 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8b6495d78-grmzq" event={"ID":"e04e2273-7cc5-46b0-9bb5-0506c6c5af46","Type":"ContainerStarted","Data":"6f7fe5a3e73831831f8522acea671aa4c67bd23f43d1b990a8178597d17a14ec"} Jul 22 12:46:50.476266 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:50.476186 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8b6495d78-grmzq" event={"ID":"e04e2273-7cc5-46b0-9bb5-0506c6c5af46","Type":"ContainerStarted","Data":"d88108f63996cc77ac0edcca5dab2aa9a629613c3fc65648363ba2a70e03c3b3"} Jul 22 12:46:50.476626 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:50.476307 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8b6495d78-grmzq" Jul 22 12:46:50.534599 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:50.534549 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8b6495d78-grmzq" podStartSLOduration=2.976015481 podStartE2EDuration="5.534533454s" totalImagesPullingTime="2.558517973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:46:50.534531464 +0000 UTC m=+358.935762384" watchObservedRunningTime="2026-07-22 12:46:50.534533454 +0000 UTC m=+358.935764374" Jul 22 12:46:56.383322 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.383284 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-lf29h"] Jul 22 12:46:56.383855 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.383510 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" podUID="565a97c4-c2e6-4a0b-b58c-2e40fb4b9084" containerName="discovery" containerID="cri-o://62a0e0a385cd656ca17275d12048c21b1324f29b98aa1e72afbe578c1ba2e662" gracePeriod=30 Jul 22 12:46:56.445991 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.445959 2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-lf29h container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.25:8080/ready\": dial tcp 10.133.0.25:8080: connect: connection refused" start-of-body= Jul 22 12:46:56.446121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.446025 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" podUID="565a97c4-c2e6-4a0b-b58c-2e40fb4b9084" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.25:8080/ready\": dial tcp 10.133.0.25:8080: connect: connection refused" Jul 22 12:46:56.502522 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.502491 2579 generic.go:356] "Generic (PLEG): container finished" podID="565a97c4-c2e6-4a0b-b58c-2e40fb4b9084" containerID="62a0e0a385cd656ca17275d12048c21b1324f29b98aa1e72afbe578c1ba2e662" exitCode=0 Jul 22 12:46:56.502654 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.502532 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" event={"ID":"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084","Type":"ContainerDied","Data":"62a0e0a385cd656ca17275d12048c21b1324f29b98aa1e72afbe578c1ba2e662"} Jul 22 12:46:56.620916 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.620894 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" Jul 22 12:46:56.706850 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.706763 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-istio-csr-dns-cert\") pod \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " Jul 22 12:46:56.706850 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.706822 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsjvt\" (UniqueName: \"kubernetes.io/projected/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-kube-api-access-jsjvt\") pod \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " Jul 22 12:46:56.706850 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.706848 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-istio-token\") pod \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " Jul 22 12:46:56.707156 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.706870 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-cacerts\") pod \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " Jul 22 12:46:56.707156 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.706891 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-istio-kubeconfig\") pod \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " Jul 22 12:46:56.707156 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.706948 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-istio-csr-ca-configmap\") pod \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " Jul 22 12:46:56.707156 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.706986 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-local-certs\") pod \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\" (UID: \"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084\") " Jul 22 12:46:56.707395 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.707365 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "565a97c4-c2e6-4a0b-b58c-2e40fb4b9084" (UID: "565a97c4-c2e6-4a0b-b58c-2e40fb4b9084"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:46:56.709243 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.709216 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "565a97c4-c2e6-4a0b-b58c-2e40fb4b9084" (UID: "565a97c4-c2e6-4a0b-b58c-2e40fb4b9084"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:46:56.709363 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.709257 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "565a97c4-c2e6-4a0b-b58c-2e40fb4b9084" (UID: "565a97c4-c2e6-4a0b-b58c-2e40fb4b9084"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:46:56.709508 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.709481 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-kube-api-access-jsjvt" (OuterVolumeSpecName: "kube-api-access-jsjvt") pod "565a97c4-c2e6-4a0b-b58c-2e40fb4b9084" (UID: "565a97c4-c2e6-4a0b-b58c-2e40fb4b9084"). InnerVolumeSpecName "kube-api-access-jsjvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:46:56.709600 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.709510 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-cacerts" (OuterVolumeSpecName: "cacerts") pod "565a97c4-c2e6-4a0b-b58c-2e40fb4b9084" (UID: "565a97c4-c2e6-4a0b-b58c-2e40fb4b9084"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:46:56.709600 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.709549 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-istio-token" (OuterVolumeSpecName: "istio-token") pod "565a97c4-c2e6-4a0b-b58c-2e40fb4b9084" (UID: "565a97c4-c2e6-4a0b-b58c-2e40fb4b9084"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:46:56.709600 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.709586 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-local-certs" (OuterVolumeSpecName: "local-certs") pod "565a97c4-c2e6-4a0b-b58c-2e40fb4b9084" (UID: "565a97c4-c2e6-4a0b-b58c-2e40fb4b9084"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:46:56.807908 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.807877 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsjvt\" (UniqueName: \"kubernetes.io/projected/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-kube-api-access-jsjvt\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:46:56.807908 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.807902 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-istio-token\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:46:56.807908 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.807912 2579 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-cacerts\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:46:56.807908 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.807920 2579 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-istio-kubeconfig\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:46:56.808168 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.807948 2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-istio-csr-ca-configmap\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:46:56.808168 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.807956 2579 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-local-certs\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:46:56.808168 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:56.807964 2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084-istio-csr-dns-cert\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:46:57.508052 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:57.508020 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" Jul 22 12:46:57.508501 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:57.508063 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lf29h" event={"ID":"565a97c4-c2e6-4a0b-b58c-2e40fb4b9084","Type":"ContainerDied","Data":"b7cffad371ee55761c270f04740161ecc220aeb0d95b00d55eacae713c55a845"} Jul 22 12:46:57.508501 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:57.508131 2579 scope.go:117] "RemoveContainer" containerID="62a0e0a385cd656ca17275d12048c21b1324f29b98aa1e72afbe578c1ba2e662" Jul 22 12:46:57.536977 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:57.536917 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-lf29h"] Jul 22 12:46:57.540979 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:57.540954 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-lf29h"] Jul 22 12:46:58.067366 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:46:58.067336 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="565a97c4-c2e6-4a0b-b58c-2e40fb4b9084" path="/var/lib/kubelet/pods/565a97c4-c2e6-4a0b-b58c-2e40fb4b9084/volumes" Jul 22 12:47:01.482689 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:01.482654 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8b6495d78-grmzq" Jul 22 12:47:36.104010 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.103974 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-565f69cdff-6z46k"] Jul 22 12:47:36.104443 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.104288 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="565a97c4-c2e6-4a0b-b58c-2e40fb4b9084" containerName="discovery" Jul 22 12:47:36.104443 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.104300 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="565a97c4-c2e6-4a0b-b58c-2e40fb4b9084" containerName="discovery" Jul 22 12:47:36.104443 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.104365 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="565a97c4-c2e6-4a0b-b58c-2e40fb4b9084" containerName="discovery" Jul 22 12:47:36.113171 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.113146 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-565f69cdff-6z46k" Jul 22 12:47:36.124170 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.124148 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-565f69cdff-6z46k"] Jul 22 12:47:36.243802 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.243768 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e15933ab-6768-4e85-b079-d1860634462a-console-config\") pod \"console-565f69cdff-6z46k\" (UID: \"e15933ab-6768-4e85-b079-d1860634462a\") " pod="openshift-console/console-565f69cdff-6z46k" Jul 22 12:47:36.243802 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.243803 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e15933ab-6768-4e85-b079-d1860634462a-console-serving-cert\") pod \"console-565f69cdff-6z46k\" (UID: \"e15933ab-6768-4e85-b079-d1860634462a\") " pod="openshift-console/console-565f69cdff-6z46k" Jul 22 12:47:36.244064 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.243823 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e15933ab-6768-4e85-b079-d1860634462a-oauth-serving-cert\") pod \"console-565f69cdff-6z46k\" (UID: \"e15933ab-6768-4e85-b079-d1860634462a\") " pod="openshift-console/console-565f69cdff-6z46k" Jul 22 12:47:36.244064 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.243916 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e15933ab-6768-4e85-b079-d1860634462a-service-ca\") pod \"console-565f69cdff-6z46k\" (UID: \"e15933ab-6768-4e85-b079-d1860634462a\") " pod="openshift-console/console-565f69cdff-6z46k" Jul 22 12:47:36.244064 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.243974 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e15933ab-6768-4e85-b079-d1860634462a-console-oauth-config\") pod \"console-565f69cdff-6z46k\" (UID: \"e15933ab-6768-4e85-b079-d1860634462a\") " pod="openshift-console/console-565f69cdff-6z46k" Jul 22 12:47:36.244064 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.244055 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e15933ab-6768-4e85-b079-d1860634462a-trusted-ca-bundle\") pod \"console-565f69cdff-6z46k\" (UID: \"e15933ab-6768-4e85-b079-d1860634462a\") " pod="openshift-console/console-565f69cdff-6z46k" Jul 22 12:47:36.244260 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.244112 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67qcb\" (UniqueName: \"kubernetes.io/projected/e15933ab-6768-4e85-b079-d1860634462a-kube-api-access-67qcb\") pod \"console-565f69cdff-6z46k\" (UID: \"e15933ab-6768-4e85-b079-d1860634462a\") " pod="openshift-console/console-565f69cdff-6z46k" Jul 22 12:47:36.345007 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.344975 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67qcb\" (UniqueName: \"kubernetes.io/projected/e15933ab-6768-4e85-b079-d1860634462a-kube-api-access-67qcb\") pod \"console-565f69cdff-6z46k\" (UID: \"e15933ab-6768-4e85-b079-d1860634462a\") " pod="openshift-console/console-565f69cdff-6z46k" Jul 22 12:47:36.345175 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.345021 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e15933ab-6768-4e85-b079-d1860634462a-console-config\") pod \"console-565f69cdff-6z46k\" (UID: \"e15933ab-6768-4e85-b079-d1860634462a\") " pod="openshift-console/console-565f69cdff-6z46k" Jul 22 12:47:36.345175 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.345038 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e15933ab-6768-4e85-b079-d1860634462a-console-serving-cert\") pod \"console-565f69cdff-6z46k\" (UID: \"e15933ab-6768-4e85-b079-d1860634462a\") " pod="openshift-console/console-565f69cdff-6z46k" Jul 22 12:47:36.345175 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.345056 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e15933ab-6768-4e85-b079-d1860634462a-oauth-serving-cert\") pod \"console-565f69cdff-6z46k\" (UID: \"e15933ab-6768-4e85-b079-d1860634462a\") " pod="openshift-console/console-565f69cdff-6z46k" Jul 22 12:47:36.345175 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.345089 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e15933ab-6768-4e85-b079-d1860634462a-service-ca\") pod \"console-565f69cdff-6z46k\" (UID: \"e15933ab-6768-4e85-b079-d1860634462a\") " pod="openshift-console/console-565f69cdff-6z46k" Jul 22 12:47:36.345175 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.345107 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e15933ab-6768-4e85-b079-d1860634462a-console-oauth-config\") pod \"console-565f69cdff-6z46k\" (UID: \"e15933ab-6768-4e85-b079-d1860634462a\") " pod="openshift-console/console-565f69cdff-6z46k" Jul 22 12:47:36.345175 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.345144 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e15933ab-6768-4e85-b079-d1860634462a-trusted-ca-bundle\") pod \"console-565f69cdff-6z46k\" (UID: \"e15933ab-6768-4e85-b079-d1860634462a\") " pod="openshift-console/console-565f69cdff-6z46k" Jul 22 12:47:36.345895 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.345869 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e15933ab-6768-4e85-b079-d1860634462a-service-ca\") pod \"console-565f69cdff-6z46k\" (UID: \"e15933ab-6768-4e85-b079-d1860634462a\") " pod="openshift-console/console-565f69cdff-6z46k" Jul 22 12:47:36.346021 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.345869 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e15933ab-6768-4e85-b079-d1860634462a-oauth-serving-cert\") pod \"console-565f69cdff-6z46k\" (UID: \"e15933ab-6768-4e85-b079-d1860634462a\") " pod="openshift-console/console-565f69cdff-6z46k" Jul 22 12:47:36.346021 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.345916 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e15933ab-6768-4e85-b079-d1860634462a-console-config\") pod \"console-565f69cdff-6z46k\" (UID: \"e15933ab-6768-4e85-b079-d1860634462a\") " pod="openshift-console/console-565f69cdff-6z46k" Jul 22 12:47:36.346021 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.345975 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e15933ab-6768-4e85-b079-d1860634462a-trusted-ca-bundle\") pod \"console-565f69cdff-6z46k\" (UID: \"e15933ab-6768-4e85-b079-d1860634462a\") " pod="openshift-console/console-565f69cdff-6z46k" Jul 22 12:47:36.347517 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.347489 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e15933ab-6768-4e85-b079-d1860634462a-console-oauth-config\") pod \"console-565f69cdff-6z46k\" (UID: \"e15933ab-6768-4e85-b079-d1860634462a\") " pod="openshift-console/console-565f69cdff-6z46k" Jul 22 12:47:36.347611 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.347555 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e15933ab-6768-4e85-b079-d1860634462a-console-serving-cert\") pod \"console-565f69cdff-6z46k\" (UID: \"e15933ab-6768-4e85-b079-d1860634462a\") " pod="openshift-console/console-565f69cdff-6z46k" Jul 22 12:47:36.354736 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.354688 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67qcb\" (UniqueName: \"kubernetes.io/projected/e15933ab-6768-4e85-b079-d1860634462a-kube-api-access-67qcb\") pod \"console-565f69cdff-6z46k\" (UID: \"e15933ab-6768-4e85-b079-d1860634462a\") " pod="openshift-console/console-565f69cdff-6z46k" Jul 22 12:47:36.422795 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.422753 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-565f69cdff-6z46k" Jul 22 12:47:36.556344 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.556310 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-565f69cdff-6z46k"] Jul 22 12:47:36.558969 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:47:36.558919 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode15933ab_6768_4e85_b079_d1860634462a.slice/crio-8642380c5fd1ee6007167df9ee8ff0e158e5288c9606fe210d6cee4c10cfaca8 WatchSource:0}: Error finding container 8642380c5fd1ee6007167df9ee8ff0e158e5288c9606fe210d6cee4c10cfaca8: Status 404 returned error can't find the container with id 8642380c5fd1ee6007167df9ee8ff0e158e5288c9606fe210d6cee4c10cfaca8 Jul 22 12:47:36.680297 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.680264 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-565f69cdff-6z46k" event={"ID":"e15933ab-6768-4e85-b079-d1860634462a","Type":"ContainerStarted","Data":"5ecc34ac22a50e7c9c3c486f2d1b21d08a31d47a150c64e3c955d6e29710d2a1"} Jul 22 12:47:36.680443 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.680323 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-565f69cdff-6z46k" event={"ID":"e15933ab-6768-4e85-b079-d1860634462a","Type":"ContainerStarted","Data":"8642380c5fd1ee6007167df9ee8ff0e158e5288c9606fe210d6cee4c10cfaca8"} Jul 22 12:47:36.701705 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:36.701652 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-565f69cdff-6z46k" podStartSLOduration=0.701634885 podStartE2EDuration="701.634885ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:47:36.701224442 +0000 UTC m=+405.102455362" watchObservedRunningTime="2026-07-22 12:47:36.701634885 +0000 UTC m=+405.102865816" Jul 22 12:47:38.389207 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:38.389164 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dcv85"] Jul 22 12:47:38.392451 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:38.392436 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dcv85" Jul 22 12:47:38.396994 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:38.396979 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 22 12:47:38.398181 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:38.398164 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-8hl6m\"" Jul 22 12:47:38.398268 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:38.398166 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 22 12:47:38.409973 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:38.409948 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dcv85"] Jul 22 12:47:38.462860 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:38.462833 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8pwx\" (UniqueName: \"kubernetes.io/projected/cbb87f9d-69aa-4766-bca0-1d568e04b8bb-kube-api-access-k8pwx\") pod \"kuadrant-operator-catalog-dcv85\" (UID: \"cbb87f9d-69aa-4766-bca0-1d568e04b8bb\") " pod="kuadrant-system/kuadrant-operator-catalog-dcv85" Jul 22 12:47:38.564206 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:38.564167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8pwx\" (UniqueName: \"kubernetes.io/projected/cbb87f9d-69aa-4766-bca0-1d568e04b8bb-kube-api-access-k8pwx\") pod \"kuadrant-operator-catalog-dcv85\" (UID: \"cbb87f9d-69aa-4766-bca0-1d568e04b8bb\") " pod="kuadrant-system/kuadrant-operator-catalog-dcv85" Jul 22 12:47:38.585826 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:38.585791 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8pwx\" (UniqueName: \"kubernetes.io/projected/cbb87f9d-69aa-4766-bca0-1d568e04b8bb-kube-api-access-k8pwx\") pod \"kuadrant-operator-catalog-dcv85\" (UID: \"cbb87f9d-69aa-4766-bca0-1d568e04b8bb\") " pod="kuadrant-system/kuadrant-operator-catalog-dcv85" Jul 22 12:47:38.701711 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:38.701616 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dcv85" Jul 22 12:47:38.751086 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:38.751056 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dcv85"] Jul 22 12:47:38.857476 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:38.855723 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dcv85"] Jul 22 12:47:38.969032 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:38.968950 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tpbmj"] Jul 22 12:47:38.973540 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:38.973525 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tpbmj" Jul 22 12:47:38.987244 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:38.987218 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tpbmj"] Jul 22 12:47:39.068159 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:39.068129 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhpzv\" (UniqueName: \"kubernetes.io/projected/50f01f4e-5b13-4a21-bf0e-6c2631057dfd-kube-api-access-vhpzv\") pod \"kuadrant-operator-catalog-tpbmj\" (UID: \"50f01f4e-5b13-4a21-bf0e-6c2631057dfd\") " pod="kuadrant-system/kuadrant-operator-catalog-tpbmj" Jul 22 12:47:39.169555 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:39.169520 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhpzv\" (UniqueName: \"kubernetes.io/projected/50f01f4e-5b13-4a21-bf0e-6c2631057dfd-kube-api-access-vhpzv\") pod \"kuadrant-operator-catalog-tpbmj\" (UID: \"50f01f4e-5b13-4a21-bf0e-6c2631057dfd\") " pod="kuadrant-system/kuadrant-operator-catalog-tpbmj" Jul 22 12:47:39.179677 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:39.179646 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhpzv\" (UniqueName: \"kubernetes.io/projected/50f01f4e-5b13-4a21-bf0e-6c2631057dfd-kube-api-access-vhpzv\") pod \"kuadrant-operator-catalog-tpbmj\" (UID: \"50f01f4e-5b13-4a21-bf0e-6c2631057dfd\") " pod="kuadrant-system/kuadrant-operator-catalog-tpbmj" Jul 22 12:47:39.282660 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:39.282581 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-tpbmj" Jul 22 12:47:39.425093 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:39.425059 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-tpbmj"] Jul 22 12:47:39.432190 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:47:39.432167 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f01f4e_5b13_4a21_bf0e_6c2631057dfd.slice/crio-539d4f0a0bbe9313248188f5a3eba7b033fb4756d1f1ec6099438764d8fe9a9e WatchSource:0}: Error finding container 539d4f0a0bbe9313248188f5a3eba7b033fb4756d1f1ec6099438764d8fe9a9e: Status 404 returned error can't find the container with id 539d4f0a0bbe9313248188f5a3eba7b033fb4756d1f1ec6099438764d8fe9a9e Jul 22 12:47:39.694842 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:39.694808 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tpbmj" event={"ID":"50f01f4e-5b13-4a21-bf0e-6c2631057dfd","Type":"ContainerStarted","Data":"539d4f0a0bbe9313248188f5a3eba7b033fb4756d1f1ec6099438764d8fe9a9e"} Jul 22 12:47:39.695883 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:39.695859 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dcv85" event={"ID":"cbb87f9d-69aa-4766-bca0-1d568e04b8bb","Type":"ContainerStarted","Data":"98a2aed82c5a65d5bc47d543747e648efbe0323b3e6fc7d46cdc912ca1dbf6c9"} Jul 22 12:47:41.708202 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:41.708170 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-tpbmj" event={"ID":"50f01f4e-5b13-4a21-bf0e-6c2631057dfd","Type":"ContainerStarted","Data":"e936d0bdab23e1872aca381b8cf9847f146e90bfadb9eb68e80a62388fa4e0ac"} Jul 22 12:47:41.709685 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:41.709658 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dcv85" event={"ID":"cbb87f9d-69aa-4766-bca0-1d568e04b8bb","Type":"ContainerStarted","Data":"4964e841db1fd8ffb820ae5d49df544cbc66b2b64bcdbf3a32a2663ab2a32e7f"} Jul 22 12:47:41.709803 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:41.709716 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-dcv85" podUID="cbb87f9d-69aa-4766-bca0-1d568e04b8bb" containerName="registry-server" containerID="cri-o://4964e841db1fd8ffb820ae5d49df544cbc66b2b64bcdbf3a32a2663ab2a32e7f" gracePeriod=2 Jul 22 12:47:41.730004 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:41.729959 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-tpbmj" podStartSLOduration=1.907554134 podStartE2EDuration="3.729944042s" totalImagesPullingTime="1.822389908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:47:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:47:41.727839912 +0000 UTC m=+410.129070833" watchObservedRunningTime="2026-07-22 12:47:41.729944042 +0000 UTC m=+410.131174954" Jul 22 12:47:41.751226 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:41.751175 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-dcv85" podStartSLOduration=1.357102843 podStartE2EDuration="3.751161134s" totalImagesPullingTime="2.394058291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:47:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:47:41.750888596 +0000 UTC m=+410.152119518" watchObservedRunningTime="2026-07-22 12:47:41.751161134 +0000 UTC m=+410.152392053" Jul 22 12:47:41.947306 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:41.947284 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dcv85" Jul 22 12:47:42.095756 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:42.095727 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8pwx\" (UniqueName: \"kubernetes.io/projected/cbb87f9d-69aa-4766-bca0-1d568e04b8bb-kube-api-access-k8pwx\") pod \"cbb87f9d-69aa-4766-bca0-1d568e04b8bb\" (UID: \"cbb87f9d-69aa-4766-bca0-1d568e04b8bb\") " Jul 22 12:47:42.097909 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:42.097880 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb87f9d-69aa-4766-bca0-1d568e04b8bb-kube-api-access-k8pwx" (OuterVolumeSpecName: "kube-api-access-k8pwx") pod "cbb87f9d-69aa-4766-bca0-1d568e04b8bb" (UID: "cbb87f9d-69aa-4766-bca0-1d568e04b8bb"). InnerVolumeSpecName "kube-api-access-k8pwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:47:42.197393 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:42.197358 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8pwx\" (UniqueName: \"kubernetes.io/projected/cbb87f9d-69aa-4766-bca0-1d568e04b8bb-kube-api-access-k8pwx\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:47:42.715519 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:42.715486 2579 generic.go:356] "Generic (PLEG): container finished" podID="cbb87f9d-69aa-4766-bca0-1d568e04b8bb" containerID="4964e841db1fd8ffb820ae5d49df544cbc66b2b64bcdbf3a32a2663ab2a32e7f" exitCode=0 Jul 22 12:47:42.715902 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:42.715529 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dcv85" Jul 22 12:47:42.715902 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:42.715571 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dcv85" event={"ID":"cbb87f9d-69aa-4766-bca0-1d568e04b8bb","Type":"ContainerDied","Data":"4964e841db1fd8ffb820ae5d49df544cbc66b2b64bcdbf3a32a2663ab2a32e7f"} Jul 22 12:47:42.715902 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:42.715622 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dcv85" event={"ID":"cbb87f9d-69aa-4766-bca0-1d568e04b8bb","Type":"ContainerDied","Data":"98a2aed82c5a65d5bc47d543747e648efbe0323b3e6fc7d46cdc912ca1dbf6c9"} Jul 22 12:47:42.715902 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:42.715646 2579 scope.go:117] "RemoveContainer" containerID="4964e841db1fd8ffb820ae5d49df544cbc66b2b64bcdbf3a32a2663ab2a32e7f" Jul 22 12:47:42.724323 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:42.724307 2579 scope.go:117] "RemoveContainer" containerID="4964e841db1fd8ffb820ae5d49df544cbc66b2b64bcdbf3a32a2663ab2a32e7f" Jul 22 12:47:42.724551 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:47:42.724533 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4964e841db1fd8ffb820ae5d49df544cbc66b2b64bcdbf3a32a2663ab2a32e7f\": container with ID starting with 4964e841db1fd8ffb820ae5d49df544cbc66b2b64bcdbf3a32a2663ab2a32e7f not found: ID does not exist" containerID="4964e841db1fd8ffb820ae5d49df544cbc66b2b64bcdbf3a32a2663ab2a32e7f" Jul 22 12:47:42.724609 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:42.724563 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4964e841db1fd8ffb820ae5d49df544cbc66b2b64bcdbf3a32a2663ab2a32e7f"} err="failed to get container status \"4964e841db1fd8ffb820ae5d49df544cbc66b2b64bcdbf3a32a2663ab2a32e7f\": rpc error: code = NotFound desc = could not find container \"4964e841db1fd8ffb820ae5d49df544cbc66b2b64bcdbf3a32a2663ab2a32e7f\": container with ID starting with 4964e841db1fd8ffb820ae5d49df544cbc66b2b64bcdbf3a32a2663ab2a32e7f not found: ID does not exist" Jul 22 12:47:42.737088 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:42.737062 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dcv85"] Jul 22 12:47:42.743955 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:42.743916 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dcv85"] Jul 22 12:47:44.067275 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:44.067241 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbb87f9d-69aa-4766-bca0-1d568e04b8bb" path="/var/lib/kubelet/pods/cbb87f9d-69aa-4766-bca0-1d568e04b8bb/volumes" Jul 22 12:47:46.423889 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:46.423832 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-565f69cdff-6z46k" Jul 22 12:47:46.424284 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:46.423945 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-565f69cdff-6z46k" Jul 22 12:47:46.429683 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:46.429657 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-565f69cdff-6z46k" Jul 22 12:47:46.737504 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:46.737434 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-565f69cdff-6z46k" Jul 22 12:47:46.790294 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:46.790261 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-579f767c9c-btscm"] Jul 22 12:47:49.283543 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:49.283509 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-tpbmj" Jul 22 12:47:49.284052 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:49.283593 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-tpbmj" Jul 22 12:47:49.306071 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:49.306049 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-tpbmj" Jul 22 12:47:49.767578 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:47:49.767500 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-tpbmj" Jul 22 12:48:05.201814 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:05.201778 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-stb8r"] Jul 22 12:48:05.202233 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:05.202159 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbb87f9d-69aa-4766-bca0-1d568e04b8bb" containerName="registry-server" Jul 22 12:48:05.202233 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:05.202172 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbb87f9d-69aa-4766-bca0-1d568e04b8bb" containerName="registry-server" Jul 22 12:48:05.202233 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:05.202231 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbb87f9d-69aa-4766-bca0-1d568e04b8bb" containerName="registry-server" Jul 22 12:48:05.206551 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:05.206532 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-stb8r" Jul 22 12:48:05.210356 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:05.210334 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-485cp\"" Jul 22 12:48:05.215836 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:05.215816 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 22 12:48:05.238556 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:05.238529 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-stb8r"] Jul 22 12:48:05.297193 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:05.297164 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7rs7\" (UniqueName: \"kubernetes.io/projected/5c4a4f48-2186-459f-aa7c-81054b32cc31-kube-api-access-r7rs7\") pod \"dns-operator-controller-manager-864dd78c4c-stb8r\" (UID: \"5c4a4f48-2186-459f-aa7c-81054b32cc31\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-stb8r" Jul 22 12:48:05.397573 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:05.397540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7rs7\" (UniqueName: \"kubernetes.io/projected/5c4a4f48-2186-459f-aa7c-81054b32cc31-kube-api-access-r7rs7\") pod \"dns-operator-controller-manager-864dd78c4c-stb8r\" (UID: \"5c4a4f48-2186-459f-aa7c-81054b32cc31\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-stb8r" Jul 22 12:48:05.412099 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:05.412071 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7rs7\" (UniqueName: \"kubernetes.io/projected/5c4a4f48-2186-459f-aa7c-81054b32cc31-kube-api-access-r7rs7\") pod \"dns-operator-controller-manager-864dd78c4c-stb8r\" (UID: \"5c4a4f48-2186-459f-aa7c-81054b32cc31\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-stb8r" Jul 22 12:48:05.516516 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:05.516433 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-stb8r" Jul 22 12:48:05.650858 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:05.650832 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-stb8r"] Jul 22 12:48:05.653353 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:48:05.653323 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4a4f48_2186_459f_aa7c_81054b32cc31.slice/crio-ac8e1be62ee14ed877b44509920fb31775e9b52d03d14bf36ae30dfd4f16cf41 WatchSource:0}: Error finding container ac8e1be62ee14ed877b44509920fb31775e9b52d03d14bf36ae30dfd4f16cf41: Status 404 returned error can't find the container with id ac8e1be62ee14ed877b44509920fb31775e9b52d03d14bf36ae30dfd4f16cf41 Jul 22 12:48:05.816809 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:05.816773 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-stb8r" event={"ID":"5c4a4f48-2186-459f-aa7c-81054b32cc31","Type":"ContainerStarted","Data":"ac8e1be62ee14ed877b44509920fb31775e9b52d03d14bf36ae30dfd4f16cf41"} Jul 22 12:48:08.830718 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:08.830682 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-stb8r" event={"ID":"5c4a4f48-2186-459f-aa7c-81054b32cc31","Type":"ContainerStarted","Data":"f08b56bfb6cc87e09eeebd63ae4bd4a0bb9ba48b16a26300453ad608e4d2428b"} Jul 22 12:48:08.831138 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:08.830822 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-stb8r" Jul 22 12:48:08.860329 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:08.860271 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-stb8r" podStartSLOduration=1.746720981 podStartE2EDuration="3.860255841s" totalImagesPullingTime="2.11353486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:48:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:48:08.859170927 +0000 UTC m=+437.260401846" watchObservedRunningTime="2026-07-22 12:48:08.860255841 +0000 UTC m=+437.261486761" Jul 22 12:48:09.166201 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:09.166119 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-t2zg5"] Jul 22 12:48:09.169564 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:09.169546 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-t2zg5" Jul 22 12:48:09.172376 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:09.172356 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-fbbwp\"" Jul 22 12:48:09.182893 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:09.182869 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-t2zg5"] Jul 22 12:48:09.234015 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:09.233979 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m69t\" (UniqueName: \"kubernetes.io/projected/b07a2a52-8017-414c-a7bc-2c1dd6c7cf9b-kube-api-access-7m69t\") pod \"authorino-operator-75cc8d5cb7-t2zg5\" (UID: \"b07a2a52-8017-414c-a7bc-2c1dd6c7cf9b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-t2zg5" Jul 22 12:48:09.335168 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:09.335132 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m69t\" (UniqueName: \"kubernetes.io/projected/b07a2a52-8017-414c-a7bc-2c1dd6c7cf9b-kube-api-access-7m69t\") pod \"authorino-operator-75cc8d5cb7-t2zg5\" (UID: \"b07a2a52-8017-414c-a7bc-2c1dd6c7cf9b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-t2zg5" Jul 22 12:48:09.349021 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:09.349000 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m69t\" (UniqueName: \"kubernetes.io/projected/b07a2a52-8017-414c-a7bc-2c1dd6c7cf9b-kube-api-access-7m69t\") pod \"authorino-operator-75cc8d5cb7-t2zg5\" (UID: \"b07a2a52-8017-414c-a7bc-2c1dd6c7cf9b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-t2zg5" Jul 22 12:48:09.479236 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:09.479152 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-t2zg5" Jul 22 12:48:09.631093 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:09.631069 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-t2zg5"] Jul 22 12:48:09.633484 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:48:09.633461 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb07a2a52_8017_414c_a7bc_2c1dd6c7cf9b.slice/crio-ff89b5eab8c8e62b74e4fcb27cf350dbe150240df463b271c8693ba8d076a67c WatchSource:0}: Error finding container ff89b5eab8c8e62b74e4fcb27cf350dbe150240df463b271c8693ba8d076a67c: Status 404 returned error can't find the container with id ff89b5eab8c8e62b74e4fcb27cf350dbe150240df463b271c8693ba8d076a67c Jul 22 12:48:09.836369 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:09.836335 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-t2zg5" event={"ID":"b07a2a52-8017-414c-a7bc-2c1dd6c7cf9b","Type":"ContainerStarted","Data":"ff89b5eab8c8e62b74e4fcb27cf350dbe150240df463b271c8693ba8d076a67c"} Jul 22 12:48:11.810394 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:11.810328 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-579f767c9c-btscm" podUID="ab830ca0-eab6-498f-a8d7-5743e1f84ecd" containerName="console" containerID="cri-o://51376a26d5b17f62654db0d58ce25b104fa096b85c4e262b76c9da60ee32ab29" gracePeriod=15 Jul 22 12:48:11.851081 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:11.851051 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-t2zg5" event={"ID":"b07a2a52-8017-414c-a7bc-2c1dd6c7cf9b","Type":"ContainerStarted","Data":"59668dda948f588c33959ab07f3742e19125c74e297042fffbd0d689c45f1129"} Jul 22 12:48:11.851231 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:11.851139 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-t2zg5" Jul 22 12:48:11.872237 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:11.872197 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-t2zg5" podStartSLOduration=0.748074074 podStartE2EDuration="2.872182497s" totalImagesPullingTime="2.124108423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:48:11.870263749 +0000 UTC m=+440.271494674" watchObservedRunningTime="2026-07-22 12:48:11.872182497 +0000 UTC m=+440.273413416" Jul 22 12:48:12.051990 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.051962 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:48:12.162597 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.162564 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb8tf\" (UniqueName: \"kubernetes.io/projected/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-kube-api-access-gb8tf\") pod \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " Jul 22 12:48:12.162759 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.162628 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-console-config\") pod \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " Jul 22 12:48:12.162759 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.162646 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-trusted-ca-bundle\") pod \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " Jul 22 12:48:12.162759 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.162661 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-oauth-serving-cert\") pod \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " Jul 22 12:48:12.162759 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.162695 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-console-oauth-config\") pod \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " Jul 22 12:48:12.162966 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.162763 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-service-ca\") pod \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " Jul 22 12:48:12.162966 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.162802 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-console-serving-cert\") pod \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\" (UID: \"ab830ca0-eab6-498f-a8d7-5743e1f84ecd\") " Jul 22 12:48:12.163164 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.163139 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ab830ca0-eab6-498f-a8d7-5743e1f84ecd" (UID: "ab830ca0-eab6-498f-a8d7-5743e1f84ecd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:48:12.163249 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.163190 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ab830ca0-eab6-498f-a8d7-5743e1f84ecd" (UID: "ab830ca0-eab6-498f-a8d7-5743e1f84ecd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:48:12.163249 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.163199 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-console-config" (OuterVolumeSpecName: "console-config") pod "ab830ca0-eab6-498f-a8d7-5743e1f84ecd" (UID: "ab830ca0-eab6-498f-a8d7-5743e1f84ecd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:48:12.163352 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.163241 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-service-ca" (OuterVolumeSpecName: "service-ca") pod "ab830ca0-eab6-498f-a8d7-5743e1f84ecd" (UID: "ab830ca0-eab6-498f-a8d7-5743e1f84ecd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 12:48:12.164874 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.164851 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ab830ca0-eab6-498f-a8d7-5743e1f84ecd" (UID: "ab830ca0-eab6-498f-a8d7-5743e1f84ecd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:48:12.164976 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.164855 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-kube-api-access-gb8tf" (OuterVolumeSpecName: "kube-api-access-gb8tf") pod "ab830ca0-eab6-498f-a8d7-5743e1f84ecd" (UID: "ab830ca0-eab6-498f-a8d7-5743e1f84ecd"). InnerVolumeSpecName "kube-api-access-gb8tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:48:12.164976 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.164876 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ab830ca0-eab6-498f-a8d7-5743e1f84ecd" (UID: "ab830ca0-eab6-498f-a8d7-5743e1f84ecd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 12:48:12.263984 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.263951 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-console-serving-cert\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:48:12.263984 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.263980 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb8tf\" (UniqueName: \"kubernetes.io/projected/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-kube-api-access-gb8tf\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:48:12.263984 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.263989 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-console-config\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:48:12.264200 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.263998 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-trusted-ca-bundle\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:48:12.264200 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.264008 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-oauth-serving-cert\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:48:12.264200 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.264018 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-console-oauth-config\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:48:12.264200 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.264026 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab830ca0-eab6-498f-a8d7-5743e1f84ecd-service-ca\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:48:12.629751 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.629719 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vbpgh"] Jul 22 12:48:12.630168 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.630151 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab830ca0-eab6-498f-a8d7-5743e1f84ecd" containerName="console" Jul 22 12:48:12.630168 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.630170 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab830ca0-eab6-498f-a8d7-5743e1f84ecd" containerName="console" Jul 22 12:48:12.630289 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.630238 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab830ca0-eab6-498f-a8d7-5743e1f84ecd" containerName="console" Jul 22 12:48:12.634233 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.634215 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vbpgh" Jul 22 12:48:12.639493 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.639468 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-62jkf\"" Jul 22 12:48:12.659042 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.659018 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vbpgh"] Jul 22 12:48:12.767863 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.767833 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsgzn\" (UniqueName: \"kubernetes.io/projected/80e30068-c33d-49fd-924d-cfe3aa66bb23-kube-api-access-lsgzn\") pod \"limitador-operator-controller-manager-859d7b78bd-vbpgh\" (UID: \"80e30068-c33d-49fd-924d-cfe3aa66bb23\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vbpgh" Jul 22 12:48:12.859835 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.859798 2579 generic.go:356] "Generic (PLEG): container finished" podID="ab830ca0-eab6-498f-a8d7-5743e1f84ecd" containerID="51376a26d5b17f62654db0d58ce25b104fa096b85c4e262b76c9da60ee32ab29" exitCode=0 Jul 22 12:48:12.860231 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.859848 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579f767c9c-btscm" Jul 22 12:48:12.860231 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.859852 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579f767c9c-btscm" event={"ID":"ab830ca0-eab6-498f-a8d7-5743e1f84ecd","Type":"ContainerDied","Data":"51376a26d5b17f62654db0d58ce25b104fa096b85c4e262b76c9da60ee32ab29"} Jul 22 12:48:12.860231 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.859908 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579f767c9c-btscm" event={"ID":"ab830ca0-eab6-498f-a8d7-5743e1f84ecd","Type":"ContainerDied","Data":"56fa0fd1fe5d2b11f9ee5f00298e497297338ba85e7aa19176c0c0ef1f4321db"} Jul 22 12:48:12.860231 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.859948 2579 scope.go:117] "RemoveContainer" containerID="51376a26d5b17f62654db0d58ce25b104fa096b85c4e262b76c9da60ee32ab29" Jul 22 12:48:12.868467 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.868448 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsgzn\" (UniqueName: \"kubernetes.io/projected/80e30068-c33d-49fd-924d-cfe3aa66bb23-kube-api-access-lsgzn\") pod \"limitador-operator-controller-manager-859d7b78bd-vbpgh\" (UID: \"80e30068-c33d-49fd-924d-cfe3aa66bb23\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vbpgh" Jul 22 12:48:12.868546 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.868450 2579 scope.go:117] "RemoveContainer" containerID="51376a26d5b17f62654db0d58ce25b104fa096b85c4e262b76c9da60ee32ab29" Jul 22 12:48:12.868751 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:48:12.868730 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51376a26d5b17f62654db0d58ce25b104fa096b85c4e262b76c9da60ee32ab29\": container with ID starting with 51376a26d5b17f62654db0d58ce25b104fa096b85c4e262b76c9da60ee32ab29 not found: ID does not exist" containerID="51376a26d5b17f62654db0d58ce25b104fa096b85c4e262b76c9da60ee32ab29" Jul 22 12:48:12.868805 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.868760 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51376a26d5b17f62654db0d58ce25b104fa096b85c4e262b76c9da60ee32ab29"} err="failed to get container status \"51376a26d5b17f62654db0d58ce25b104fa096b85c4e262b76c9da60ee32ab29\": rpc error: code = NotFound desc = could not find container \"51376a26d5b17f62654db0d58ce25b104fa096b85c4e262b76c9da60ee32ab29\": container with ID starting with 51376a26d5b17f62654db0d58ce25b104fa096b85c4e262b76c9da60ee32ab29 not found: ID does not exist" Jul 22 12:48:12.878973 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.878948 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsgzn\" (UniqueName: \"kubernetes.io/projected/80e30068-c33d-49fd-924d-cfe3aa66bb23-kube-api-access-lsgzn\") pod \"limitador-operator-controller-manager-859d7b78bd-vbpgh\" (UID: \"80e30068-c33d-49fd-924d-cfe3aa66bb23\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vbpgh" Jul 22 12:48:12.887392 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.887345 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-579f767c9c-btscm"] Jul 22 12:48:12.900330 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.900304 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-579f767c9c-btscm"] Jul 22 12:48:12.944676 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:12.944655 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vbpgh" Jul 22 12:48:13.090057 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:13.090014 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vbpgh"] Jul 22 12:48:13.092682 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:48:13.092654 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e30068_c33d_49fd_924d_cfe3aa66bb23.slice/crio-b9934daece7830fb6ca2e59e885fc830ec7b59ed8ff32a823a69f1c2fe74241b WatchSource:0}: Error finding container b9934daece7830fb6ca2e59e885fc830ec7b59ed8ff32a823a69f1c2fe74241b: Status 404 returned error can't find the container with id b9934daece7830fb6ca2e59e885fc830ec7b59ed8ff32a823a69f1c2fe74241b Jul 22 12:48:13.867142 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:13.867109 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vbpgh" event={"ID":"80e30068-c33d-49fd-924d-cfe3aa66bb23","Type":"ContainerStarted","Data":"b9934daece7830fb6ca2e59e885fc830ec7b59ed8ff32a823a69f1c2fe74241b"} Jul 22 12:48:14.071590 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:14.071549 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab830ca0-eab6-498f-a8d7-5743e1f84ecd" path="/var/lib/kubelet/pods/ab830ca0-eab6-498f-a8d7-5743e1f84ecd/volumes" Jul 22 12:48:15.877781 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:15.877743 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vbpgh" event={"ID":"80e30068-c33d-49fd-924d-cfe3aa66bb23","Type":"ContainerStarted","Data":"9f486b4dcc6c895aedef427519c98952025823dca5b9f4d29f22b937862205ee"} Jul 22 12:48:15.878199 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:15.877805 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vbpgh" Jul 22 12:48:19.839246 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:19.839219 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-stb8r" Jul 22 12:48:19.859983 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:19.859918 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vbpgh" podStartSLOduration=6.06029285 podStartE2EDuration="7.859902542s" totalImagesPullingTime="1.799609692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:48:15.922423909 +0000 UTC m=+444.323654828" watchObservedRunningTime="2026-07-22 12:48:19.859902542 +0000 UTC m=+448.261133465" Jul 22 12:48:22.862043 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:22.862011 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-t2zg5" Jul 22 12:48:26.884918 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:26.884882 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vbpgh" Jul 22 12:48:36.083460 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:36.083424 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vbpgh"] Jul 22 12:48:36.083908 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:36.083657 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vbpgh" podUID="80e30068-c33d-49fd-924d-cfe3aa66bb23" containerName="manager" containerID="cri-o://9f486b4dcc6c895aedef427519c98952025823dca5b9f4d29f22b937862205ee" gracePeriod=2 Jul 22 12:48:36.089812 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:36.089778 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vbpgh"] Jul 22 12:48:36.130017 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:36.129989 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x8bq9"] Jul 22 12:48:36.130344 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:36.130330 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80e30068-c33d-49fd-924d-cfe3aa66bb23" containerName="manager" Jul 22 12:48:36.130393 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:36.130345 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="80e30068-c33d-49fd-924d-cfe3aa66bb23" containerName="manager" Jul 22 12:48:36.130429 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:36.130424 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="80e30068-c33d-49fd-924d-cfe3aa66bb23" containerName="manager" Jul 22 12:48:36.133448 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:36.133430 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x8bq9" Jul 22 12:48:36.137898 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:48:36.137865 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vbpgh\" is forbidden: User \"system:node:ip-10-0-129-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-34.ec2.internal' and this object" podUID="80e30068-c33d-49fd-924d-cfe3aa66bb23" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vbpgh" Jul 22 12:48:36.153567 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:36.153542 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x8bq9"] Jul 22 12:48:36.272536 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:36.272506 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqxcp\" (UniqueName: \"kubernetes.io/projected/69a15b20-8ebf-4716-8c1b-9d577fa96224-kube-api-access-jqxcp\") pod \"limitador-operator-controller-manager-859d7b78bd-x8bq9\" (UID: \"69a15b20-8ebf-4716-8c1b-9d577fa96224\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x8bq9" Jul 22 12:48:36.322346 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:36.322327 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vbpgh" Jul 22 12:48:36.325101 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:48:36.325074 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vbpgh\" is forbidden: User \"system:node:ip-10-0-129-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-34.ec2.internal' and this object" podUID="80e30068-c33d-49fd-924d-cfe3aa66bb23" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vbpgh" Jul 22 12:48:36.374083 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:36.374023 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqxcp\" (UniqueName: \"kubernetes.io/projected/69a15b20-8ebf-4716-8c1b-9d577fa96224-kube-api-access-jqxcp\") pod \"limitador-operator-controller-manager-859d7b78bd-x8bq9\" (UID: \"69a15b20-8ebf-4716-8c1b-9d577fa96224\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x8bq9" Jul 22 12:48:36.387017 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:36.386995 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqxcp\" (UniqueName: \"kubernetes.io/projected/69a15b20-8ebf-4716-8c1b-9d577fa96224-kube-api-access-jqxcp\") pod \"limitador-operator-controller-manager-859d7b78bd-x8bq9\" (UID: \"69a15b20-8ebf-4716-8c1b-9d577fa96224\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x8bq9" Jul 22 12:48:36.471249 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:36.471223 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x8bq9" Jul 22 12:48:36.475191 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:36.475168 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsgzn\" (UniqueName: \"kubernetes.io/projected/80e30068-c33d-49fd-924d-cfe3aa66bb23-kube-api-access-lsgzn\") pod \"80e30068-c33d-49fd-924d-cfe3aa66bb23\" (UID: \"80e30068-c33d-49fd-924d-cfe3aa66bb23\") " Jul 22 12:48:36.477189 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:36.477160 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e30068-c33d-49fd-924d-cfe3aa66bb23-kube-api-access-lsgzn" (OuterVolumeSpecName: "kube-api-access-lsgzn") pod "80e30068-c33d-49fd-924d-cfe3aa66bb23" (UID: "80e30068-c33d-49fd-924d-cfe3aa66bb23"). InnerVolumeSpecName "kube-api-access-lsgzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:48:36.576539 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:36.576512 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsgzn\" (UniqueName: \"kubernetes.io/projected/80e30068-c33d-49fd-924d-cfe3aa66bb23-kube-api-access-lsgzn\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:48:36.627253 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:36.627227 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x8bq9"] Jul 22 12:48:36.629683 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:48:36.629652 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a15b20_8ebf_4716_8c1b_9d577fa96224.slice/crio-3ae279983bff324966c452ee7518bdb3c598a47378418c5f08027b8b883fce78 WatchSource:0}: Error finding container 3ae279983bff324966c452ee7518bdb3c598a47378418c5f08027b8b883fce78: Status 404 returned error can't find the container with id 3ae279983bff324966c452ee7518bdb3c598a47378418c5f08027b8b883fce78 Jul 22 12:48:36.971210 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:36.971107 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x8bq9" event={"ID":"69a15b20-8ebf-4716-8c1b-9d577fa96224","Type":"ContainerStarted","Data":"218bcc891a1c1ed126e836709ed3861b98f81926cee5d4767be31351f694f3f7"} Jul 22 12:48:36.971210 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:36.971177 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x8bq9" event={"ID":"69a15b20-8ebf-4716-8c1b-9d577fa96224","Type":"ContainerStarted","Data":"3ae279983bff324966c452ee7518bdb3c598a47378418c5f08027b8b883fce78"} Jul 22 12:48:36.971210 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:36.971210 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x8bq9" Jul 22 12:48:36.972264 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:36.972242 2579 generic.go:356] "Generic (PLEG): container finished" podID="80e30068-c33d-49fd-924d-cfe3aa66bb23" containerID="9f486b4dcc6c895aedef427519c98952025823dca5b9f4d29f22b937862205ee" exitCode=0 Jul 22 12:48:36.972356 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:36.972277 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vbpgh" Jul 22 12:48:36.972356 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:36.972305 2579 scope.go:117] "RemoveContainer" containerID="9f486b4dcc6c895aedef427519c98952025823dca5b9f4d29f22b937862205ee" Jul 22 12:48:36.974393 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:48:36.974372 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vbpgh\" is forbidden: User \"system:node:ip-10-0-129-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-34.ec2.internal' and this object" podUID="80e30068-c33d-49fd-924d-cfe3aa66bb23" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vbpgh" Jul 22 12:48:36.980657 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:36.980642 2579 scope.go:117] "RemoveContainer" containerID="9f486b4dcc6c895aedef427519c98952025823dca5b9f4d29f22b937862205ee" Jul 22 12:48:36.980897 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:48:36.980881 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f486b4dcc6c895aedef427519c98952025823dca5b9f4d29f22b937862205ee\": container with ID starting with 9f486b4dcc6c895aedef427519c98952025823dca5b9f4d29f22b937862205ee not found: ID does not exist" containerID="9f486b4dcc6c895aedef427519c98952025823dca5b9f4d29f22b937862205ee" Jul 22 12:48:36.980973 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:36.980903 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f486b4dcc6c895aedef427519c98952025823dca5b9f4d29f22b937862205ee"} err="failed to get container status \"9f486b4dcc6c895aedef427519c98952025823dca5b9f4d29f22b937862205ee\": rpc error: code = NotFound desc = could not find container \"9f486b4dcc6c895aedef427519c98952025823dca5b9f4d29f22b937862205ee\": container with ID starting with 9f486b4dcc6c895aedef427519c98952025823dca5b9f4d29f22b937862205ee not found: ID does not exist" Jul 22 12:48:37.000105 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:36.999959 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x8bq9" podStartSLOduration=0.999943628 podStartE2EDuration="999.943628ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:48:36.997397006 +0000 UTC m=+465.398627937" watchObservedRunningTime="2026-07-22 12:48:36.999943628 +0000 UTC m=+465.401174547" Jul 22 12:48:37.000213 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:48:37.000109 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vbpgh\" is forbidden: User \"system:node:ip-10-0-129-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-34.ec2.internal' and this object" podUID="80e30068-c33d-49fd-924d-cfe3aa66bb23" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vbpgh" Jul 22 12:48:37.117888 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:37.117852 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85w2k"] Jul 22 12:48:37.122442 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:37.122427 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85w2k" Jul 22 12:48:37.126280 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:48:37.126251 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-vbpgh\" is forbidden: User \"system:node:ip-10-0-129-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-34.ec2.internal' and this object" podUID="80e30068-c33d-49fd-924d-cfe3aa66bb23" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-vbpgh" Jul 22 12:48:37.130074 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:37.130056 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hv48z\"" Jul 22 12:48:37.159165 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:37.159140 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85w2k"] Jul 22 12:48:37.283049 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:37.282961 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c11584f-1e2d-464b-96a9-ed0f48c0f930-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-85w2k\" (UID: \"3c11584f-1e2d-464b-96a9-ed0f48c0f930\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85w2k" Jul 22 12:48:37.283196 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:37.283103 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hskqg\" (UniqueName: \"kubernetes.io/projected/3c11584f-1e2d-464b-96a9-ed0f48c0f930-kube-api-access-hskqg\") pod \"kuadrant-operator-controller-manager-86878c769f-85w2k\" (UID: \"3c11584f-1e2d-464b-96a9-ed0f48c0f930\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85w2k" Jul 22 12:48:37.384209 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:37.384176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hskqg\" (UniqueName: \"kubernetes.io/projected/3c11584f-1e2d-464b-96a9-ed0f48c0f930-kube-api-access-hskqg\") pod \"kuadrant-operator-controller-manager-86878c769f-85w2k\" (UID: \"3c11584f-1e2d-464b-96a9-ed0f48c0f930\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85w2k" Jul 22 12:48:37.384384 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:37.384224 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c11584f-1e2d-464b-96a9-ed0f48c0f930-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-85w2k\" (UID: \"3c11584f-1e2d-464b-96a9-ed0f48c0f930\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85w2k" Jul 22 12:48:37.384533 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:37.384515 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c11584f-1e2d-464b-96a9-ed0f48c0f930-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-85w2k\" (UID: \"3c11584f-1e2d-464b-96a9-ed0f48c0f930\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85w2k" Jul 22 12:48:37.395634 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:37.395603 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hskqg\" (UniqueName: \"kubernetes.io/projected/3c11584f-1e2d-464b-96a9-ed0f48c0f930-kube-api-access-hskqg\") pod \"kuadrant-operator-controller-manager-86878c769f-85w2k\" (UID: \"3c11584f-1e2d-464b-96a9-ed0f48c0f930\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85w2k" Jul 22 12:48:37.433061 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:37.433031 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85w2k" Jul 22 12:48:37.588288 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:37.588257 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85w2k"] Jul 22 12:48:37.592087 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:48:37.592059 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c11584f_1e2d_464b_96a9_ed0f48c0f930.slice/crio-08e5e5dadcdd05c79022cdfedffb3ce52c31b929267cbe967bb71f59100fb53b WatchSource:0}: Error finding container 08e5e5dadcdd05c79022cdfedffb3ce52c31b929267cbe967bb71f59100fb53b: Status 404 returned error can't find the container with id 08e5e5dadcdd05c79022cdfedffb3ce52c31b929267cbe967bb71f59100fb53b Jul 22 12:48:37.978333 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:37.978256 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85w2k" event={"ID":"3c11584f-1e2d-464b-96a9-ed0f48c0f930","Type":"ContainerStarted","Data":"08e5e5dadcdd05c79022cdfedffb3ce52c31b929267cbe967bb71f59100fb53b"} Jul 22 12:48:38.066644 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:38.066609 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80e30068-c33d-49fd-924d-cfe3aa66bb23" path="/var/lib/kubelet/pods/80e30068-c33d-49fd-924d-cfe3aa66bb23/volumes" Jul 22 12:48:42.002881 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:42.002848 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85w2k" event={"ID":"3c11584f-1e2d-464b-96a9-ed0f48c0f930","Type":"ContainerStarted","Data":"728399d1272623f6c319dd01cbb35095611c7a30300f75fdfb0e78f3900e90ae"} Jul 22 12:48:42.003247 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:42.002919 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85w2k" Jul 22 12:48:42.029411 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:42.029364 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85w2k" podStartSLOduration=0.75796984 podStartE2EDuration="5.029349609s" totalImagesPullingTime="4.271379769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:48:42.027254293 +0000 UTC m=+470.428485212" watchObservedRunningTime="2026-07-22 12:48:42.029349609 +0000 UTC m=+470.430580528" Jul 22 12:48:47.981697 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:47.981666 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-x8bq9" Jul 22 12:48:53.010072 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:48:53.010037 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85w2k" Jul 22 12:49:12.718504 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:12.718469 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-g26qz"] Jul 22 12:49:12.721440 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:12.721424 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-g26qz" Jul 22 12:49:12.724355 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:12.724335 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-q5zrv\"" Jul 22 12:49:12.732201 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:12.732176 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-g26qz"] Jul 22 12:49:12.802510 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:12.802476 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85jwj\" (UniqueName: \"kubernetes.io/projected/6b7ab0ca-cab5-4cf4-9631-82e22e68653b-kube-api-access-85jwj\") pod \"authorino-75d556bd77-g26qz\" (UID: \"6b7ab0ca-cab5-4cf4-9631-82e22e68653b\") " pod="kuadrant-system/authorino-75d556bd77-g26qz" Jul 22 12:49:12.903479 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:12.903447 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85jwj\" (UniqueName: \"kubernetes.io/projected/6b7ab0ca-cab5-4cf4-9631-82e22e68653b-kube-api-access-85jwj\") pod \"authorino-75d556bd77-g26qz\" (UID: \"6b7ab0ca-cab5-4cf4-9631-82e22e68653b\") " pod="kuadrant-system/authorino-75d556bd77-g26qz" Jul 22 12:49:12.911673 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:12.911647 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85jwj\" (UniqueName: \"kubernetes.io/projected/6b7ab0ca-cab5-4cf4-9631-82e22e68653b-kube-api-access-85jwj\") pod \"authorino-75d556bd77-g26qz\" (UID: \"6b7ab0ca-cab5-4cf4-9631-82e22e68653b\") " pod="kuadrant-system/authorino-75d556bd77-g26qz" Jul 22 12:49:13.030446 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:13.030359 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-g26qz" Jul 22 12:49:13.161601 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:13.161575 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-g26qz"] Jul 22 12:49:13.163649 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:49:13.163621 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b7ab0ca_cab5_4cf4_9631_82e22e68653b.slice/crio-6c71d6b479f12bb6a0ee89593cd1ef5226488170119c74e65ea7a9cffadb8e23 WatchSource:0}: Error finding container 6c71d6b479f12bb6a0ee89593cd1ef5226488170119c74e65ea7a9cffadb8e23: Status 404 returned error can't find the container with id 6c71d6b479f12bb6a0ee89593cd1ef5226488170119c74e65ea7a9cffadb8e23 Jul 22 12:49:14.158514 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:14.158474 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-g26qz" event={"ID":"6b7ab0ca-cab5-4cf4-9631-82e22e68653b","Type":"ContainerStarted","Data":"6c71d6b479f12bb6a0ee89593cd1ef5226488170119c74e65ea7a9cffadb8e23"} Jul 22 12:49:17.175426 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:17.175393 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-g26qz" event={"ID":"6b7ab0ca-cab5-4cf4-9631-82e22e68653b","Type":"ContainerStarted","Data":"f98bdf3097818e910826eeb03e097b542a1689b285137a3a6157e7998e6363df"} Jul 22 12:49:17.196435 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:17.196386 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-g26qz" podStartSLOduration=2.164215818 podStartE2EDuration="5.196370153s" totalImagesPullingTime="3.032154335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:49:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:49:17.196129644 +0000 UTC m=+505.597360563" watchObservedRunningTime="2026-07-22 12:49:17.196370153 +0000 UTC m=+505.597601073" Jul 22 12:49:33.452876 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:33.452843 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-gvxdk"] Jul 22 12:49:33.456499 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:33.456478 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gvxdk" Jul 22 12:49:33.459670 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:33.459651 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 22 12:49:33.460652 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:33.460633 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-5z799\"" Jul 22 12:49:33.460741 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:33.460659 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 22 12:49:33.460741 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:33.460673 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 22 12:49:33.466121 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:33.466102 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-gvxdk"] Jul 22 12:49:33.490332 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:33.490299 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/88665a68-3afd-450d-8321-834747980c41-data\") pod \"postgres-69fb4594cf-gvxdk\" (UID: \"88665a68-3afd-450d-8321-834747980c41\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gvxdk" Jul 22 12:49:33.490446 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:33.490361 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b758l\" (UniqueName: \"kubernetes.io/projected/88665a68-3afd-450d-8321-834747980c41-kube-api-access-b758l\") pod \"postgres-69fb4594cf-gvxdk\" (UID: \"88665a68-3afd-450d-8321-834747980c41\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gvxdk" Jul 22 12:49:33.591659 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:33.591621 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/88665a68-3afd-450d-8321-834747980c41-data\") pod \"postgres-69fb4594cf-gvxdk\" (UID: \"88665a68-3afd-450d-8321-834747980c41\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gvxdk" Jul 22 12:49:33.591814 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:33.591682 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b758l\" (UniqueName: \"kubernetes.io/projected/88665a68-3afd-450d-8321-834747980c41-kube-api-access-b758l\") pod \"postgres-69fb4594cf-gvxdk\" (UID: \"88665a68-3afd-450d-8321-834747980c41\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gvxdk" Jul 22 12:49:33.592033 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:33.592013 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/88665a68-3afd-450d-8321-834747980c41-data\") pod \"postgres-69fb4594cf-gvxdk\" (UID: \"88665a68-3afd-450d-8321-834747980c41\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gvxdk" Jul 22 12:49:33.601989 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:33.601970 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b758l\" (UniqueName: \"kubernetes.io/projected/88665a68-3afd-450d-8321-834747980c41-kube-api-access-b758l\") pod \"postgres-69fb4594cf-gvxdk\" (UID: \"88665a68-3afd-450d-8321-834747980c41\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gvxdk" Jul 22 12:49:33.768396 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:33.768313 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gvxdk" Jul 22 12:49:33.905032 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:33.904883 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-gvxdk"] Jul 22 12:49:33.907803 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:49:33.907775 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88665a68_3afd_450d_8321_834747980c41.slice/crio-8ff2e6d7060074a098ed0eb749dcd2c8735665aed01f8de5edb9202133874d6f WatchSource:0}: Error finding container 8ff2e6d7060074a098ed0eb749dcd2c8735665aed01f8de5edb9202133874d6f: Status 404 returned error can't find the container with id 8ff2e6d7060074a098ed0eb749dcd2c8735665aed01f8de5edb9202133874d6f Jul 22 12:49:34.252266 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:34.252230 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gvxdk" event={"ID":"88665a68-3afd-450d-8321-834747980c41","Type":"ContainerStarted","Data":"8ff2e6d7060074a098ed0eb749dcd2c8735665aed01f8de5edb9202133874d6f"} Jul 22 12:49:39.280669 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:39.280635 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gvxdk" event={"ID":"88665a68-3afd-450d-8321-834747980c41","Type":"ContainerStarted","Data":"3bf232337e2475e7a2fc79f0ead19c4ed3501376ebf140461732bfbcaa419926"} Jul 22 12:49:39.281085 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:39.280780 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gvxdk" Jul 22 12:49:39.303332 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:39.303289 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gvxdk" podStartSLOduration=1.6297530340000002 podStartE2EDuration="6.303270675s" totalImagesPullingTime="4.673517641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:49:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:49:39.301186746 +0000 UTC m=+527.702417689" watchObservedRunningTime="2026-07-22 12:49:39.303270675 +0000 UTC m=+527.704501597" Jul 22 12:49:45.314944 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:45.314858 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gvxdk" Jul 22 12:49:47.783569 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:47.783536 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-g26qz"] Jul 22 12:49:47.784050 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:47.783760 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-g26qz" podUID="6b7ab0ca-cab5-4cf4-9631-82e22e68653b" containerName="authorino" containerID="cri-o://f98bdf3097818e910826eeb03e097b542a1689b285137a3a6157e7998e6363df" gracePeriod=30 Jul 22 12:49:48.035052 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:48.034996 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-g26qz" Jul 22 12:49:48.124111 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:48.124079 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85jwj\" (UniqueName: \"kubernetes.io/projected/6b7ab0ca-cab5-4cf4-9631-82e22e68653b-kube-api-access-85jwj\") pod \"6b7ab0ca-cab5-4cf4-9631-82e22e68653b\" (UID: \"6b7ab0ca-cab5-4cf4-9631-82e22e68653b\") " Jul 22 12:49:48.126139 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:48.126111 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b7ab0ca-cab5-4cf4-9631-82e22e68653b-kube-api-access-85jwj" (OuterVolumeSpecName: "kube-api-access-85jwj") pod "6b7ab0ca-cab5-4cf4-9631-82e22e68653b" (UID: "6b7ab0ca-cab5-4cf4-9631-82e22e68653b"). InnerVolumeSpecName "kube-api-access-85jwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:49:48.224920 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:48.224887 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85jwj\" (UniqueName: \"kubernetes.io/projected/6b7ab0ca-cab5-4cf4-9631-82e22e68653b-kube-api-access-85jwj\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:49:48.321058 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:48.320981 2579 generic.go:356] "Generic (PLEG): container finished" podID="6b7ab0ca-cab5-4cf4-9631-82e22e68653b" containerID="f98bdf3097818e910826eeb03e097b542a1689b285137a3a6157e7998e6363df" exitCode=0 Jul 22 12:49:48.321058 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:48.321023 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-g26qz" Jul 22 12:49:48.321277 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:48.321070 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-g26qz" event={"ID":"6b7ab0ca-cab5-4cf4-9631-82e22e68653b","Type":"ContainerDied","Data":"f98bdf3097818e910826eeb03e097b542a1689b285137a3a6157e7998e6363df"} Jul 22 12:49:48.321277 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:48.321116 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-g26qz" event={"ID":"6b7ab0ca-cab5-4cf4-9631-82e22e68653b","Type":"ContainerDied","Data":"6c71d6b479f12bb6a0ee89593cd1ef5226488170119c74e65ea7a9cffadb8e23"} Jul 22 12:49:48.321277 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:48.321137 2579 scope.go:117] "RemoveContainer" containerID="f98bdf3097818e910826eeb03e097b542a1689b285137a3a6157e7998e6363df" Jul 22 12:49:48.329730 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:48.329713 2579 scope.go:117] "RemoveContainer" containerID="f98bdf3097818e910826eeb03e097b542a1689b285137a3a6157e7998e6363df" Jul 22 12:49:48.329997 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:49:48.329976 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f98bdf3097818e910826eeb03e097b542a1689b285137a3a6157e7998e6363df\": container with ID starting with f98bdf3097818e910826eeb03e097b542a1689b285137a3a6157e7998e6363df not found: ID does not exist" containerID="f98bdf3097818e910826eeb03e097b542a1689b285137a3a6157e7998e6363df" Jul 22 12:49:48.330075 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:48.330007 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f98bdf3097818e910826eeb03e097b542a1689b285137a3a6157e7998e6363df"} err="failed to get container status \"f98bdf3097818e910826eeb03e097b542a1689b285137a3a6157e7998e6363df\": rpc error: code = NotFound desc = could not find container \"f98bdf3097818e910826eeb03e097b542a1689b285137a3a6157e7998e6363df\": container with ID starting with f98bdf3097818e910826eeb03e097b542a1689b285137a3a6157e7998e6363df not found: ID does not exist" Jul 22 12:49:48.341880 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:48.341858 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-g26qz"] Jul 22 12:49:48.345298 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:48.345276 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-g26qz"] Jul 22 12:49:50.068067 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:49:50.068031 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b7ab0ca-cab5-4cf4-9631-82e22e68653b" path="/var/lib/kubelet/pods/6b7ab0ca-cab5-4cf4-9631-82e22e68653b/volumes" Jul 22 12:50:51.940136 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:51.940110 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ddj6_b28e0ca4-8512-45da-91ed-2cde1491b2e2/ovn-acl-logging/0.log" Jul 22 12:50:51.943299 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:51.943276 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ddj6_b28e0ca4-8512-45da-91ed-2cde1491b2e2/ovn-acl-logging/0.log" Jul 22 12:50:52.260103 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.260030 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-47494"] Jul 22 12:50:52.260398 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.260386 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b7ab0ca-cab5-4cf4-9631-82e22e68653b" containerName="authorino" Jul 22 12:50:52.260447 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.260400 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b7ab0ca-cab5-4cf4-9631-82e22e68653b" containerName="authorino" Jul 22 12:50:52.260537 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.260525 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b7ab0ca-cab5-4cf4-9631-82e22e68653b" containerName="authorino" Jul 22 12:50:52.263741 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.263721 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-47494" Jul 22 12:50:52.268672 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.268643 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 22 12:50:52.268672 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.268645 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 22 12:50:52.268672 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.268645 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-nqcsn\"" Jul 22 12:50:52.268948 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.268770 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 22 12:50:52.281301 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.281279 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-47494"] Jul 22 12:50:52.379761 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.379727 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dshgl\" (UniqueName: \"kubernetes.io/projected/8598cee8-c42f-4502-8a1f-5b7f20e39400-kube-api-access-dshgl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-47494\" (UID: \"8598cee8-c42f-4502-8a1f-5b7f20e39400\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-47494" Jul 22 12:50:52.379917 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.379814 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8598cee8-c42f-4502-8a1f-5b7f20e39400-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-47494\" (UID: \"8598cee8-c42f-4502-8a1f-5b7f20e39400\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-47494" Jul 22 12:50:52.379917 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.379843 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8598cee8-c42f-4502-8a1f-5b7f20e39400-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-47494\" (UID: \"8598cee8-c42f-4502-8a1f-5b7f20e39400\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-47494" Jul 22 12:50:52.379917 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.379894 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8598cee8-c42f-4502-8a1f-5b7f20e39400-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-47494\" (UID: \"8598cee8-c42f-4502-8a1f-5b7f20e39400\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-47494" Jul 22 12:50:52.380038 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.379919 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8598cee8-c42f-4502-8a1f-5b7f20e39400-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-47494\" (UID: \"8598cee8-c42f-4502-8a1f-5b7f20e39400\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-47494" Jul 22 12:50:52.380038 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.379982 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8598cee8-c42f-4502-8a1f-5b7f20e39400-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-47494\" (UID: \"8598cee8-c42f-4502-8a1f-5b7f20e39400\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-47494" Jul 22 12:50:52.481168 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.481138 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8598cee8-c42f-4502-8a1f-5b7f20e39400-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-47494\" (UID: \"8598cee8-c42f-4502-8a1f-5b7f20e39400\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-47494" Jul 22 12:50:52.481315 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.481185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dshgl\" (UniqueName: \"kubernetes.io/projected/8598cee8-c42f-4502-8a1f-5b7f20e39400-kube-api-access-dshgl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-47494\" (UID: \"8598cee8-c42f-4502-8a1f-5b7f20e39400\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-47494" Jul 22 12:50:52.481361 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.481304 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8598cee8-c42f-4502-8a1f-5b7f20e39400-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-47494\" (UID: \"8598cee8-c42f-4502-8a1f-5b7f20e39400\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-47494" Jul 22 12:50:52.481361 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.481346 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8598cee8-c42f-4502-8a1f-5b7f20e39400-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-47494\" (UID: \"8598cee8-c42f-4502-8a1f-5b7f20e39400\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-47494" Jul 22 12:50:52.481426 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.481409 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8598cee8-c42f-4502-8a1f-5b7f20e39400-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-47494\" (UID: \"8598cee8-c42f-4502-8a1f-5b7f20e39400\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-47494" Jul 22 12:50:52.481459 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.481442 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8598cee8-c42f-4502-8a1f-5b7f20e39400-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-47494\" (UID: \"8598cee8-c42f-4502-8a1f-5b7f20e39400\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-47494" Jul 22 12:50:52.481600 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.481582 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8598cee8-c42f-4502-8a1f-5b7f20e39400-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-47494\" (UID: \"8598cee8-c42f-4502-8a1f-5b7f20e39400\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-47494" Jul 22 12:50:52.481641 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.481620 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8598cee8-c42f-4502-8a1f-5b7f20e39400-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-47494\" (UID: \"8598cee8-c42f-4502-8a1f-5b7f20e39400\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-47494" Jul 22 12:50:52.481786 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.481760 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8598cee8-c42f-4502-8a1f-5b7f20e39400-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-47494\" (UID: \"8598cee8-c42f-4502-8a1f-5b7f20e39400\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-47494" Jul 22 12:50:52.483626 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.483601 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8598cee8-c42f-4502-8a1f-5b7f20e39400-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-47494\" (UID: \"8598cee8-c42f-4502-8a1f-5b7f20e39400\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-47494" Jul 22 12:50:52.483746 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.483728 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8598cee8-c42f-4502-8a1f-5b7f20e39400-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-47494\" (UID: \"8598cee8-c42f-4502-8a1f-5b7f20e39400\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-47494" Jul 22 12:50:52.502826 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.502803 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dshgl\" (UniqueName: \"kubernetes.io/projected/8598cee8-c42f-4502-8a1f-5b7f20e39400-kube-api-access-dshgl\") pod \"e2e-distinct-simulated-kserve-59975d69f9-47494\" (UID: \"8598cee8-c42f-4502-8a1f-5b7f20e39400\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-47494" Jul 22 12:50:52.574418 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.574381 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-47494" Jul 22 12:50:52.711645 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:52.711445 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-47494"] Jul 22 12:50:52.713842 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:50:52.713816 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8598cee8_c42f_4502_8a1f_5b7f20e39400.slice/crio-7a78ee60d1a2486cdcb6c289efcded2b8c0b57c29a06022a7a5f4ab8753551b1 WatchSource:0}: Error finding container 7a78ee60d1a2486cdcb6c289efcded2b8c0b57c29a06022a7a5f4ab8753551b1: Status 404 returned error can't find the container with id 7a78ee60d1a2486cdcb6c289efcded2b8c0b57c29a06022a7a5f4ab8753551b1 Jul 22 12:50:53.616775 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:53.616724 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-47494" event={"ID":"8598cee8-c42f-4502-8a1f-5b7f20e39400","Type":"ContainerStarted","Data":"7a78ee60d1a2486cdcb6c289efcded2b8c0b57c29a06022a7a5f4ab8753551b1"} Jul 22 12:50:54.623404 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:54.623372 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-47494" event={"ID":"8598cee8-c42f-4502-8a1f-5b7f20e39400","Type":"ContainerStarted","Data":"b563575acbac6eceaa8d8c774ca1b937fd4bad4e85a679ba735890744cf4edb0"} Jul 22 12:50:54.643863 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:50:54.643812 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-47494" podStartSLOduration=1.508649305 podStartE2EDuration="2.64379735s" totalImagesPullingTime="1.135148045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:50:54.642612296 +0000 UTC m=+603.043843229" watchObservedRunningTime="2026-07-22 12:50:54.64379735 +0000 UTC m=+603.045028271" Jul 22 12:51:02.574872 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:02.574816 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-47494" Jul 22 12:51:02.574872 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:02.574889 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-47494" Jul 22 12:51:02.587588 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:02.587558 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-47494" Jul 22 12:51:02.672980 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:02.672955 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-47494" Jul 22 12:51:35.877486 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:35.877454 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8"] Jul 22 12:51:35.881065 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:35.881037 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8" Jul 22 12:51:35.884330 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:35.884297 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 22 12:51:35.893197 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:35.893174 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8"] Jul 22 12:51:35.964678 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:35.964646 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4b45501-1153-49f4-a9a7-753f617969ef-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8\" (UID: \"c4b45501-1153-49f4-a9a7-753f617969ef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8" Jul 22 12:51:35.964678 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:35.964679 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4b45501-1153-49f4-a9a7-753f617969ef-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8\" (UID: \"c4b45501-1153-49f4-a9a7-753f617969ef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8" Jul 22 12:51:35.964880 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:35.964697 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4b45501-1153-49f4-a9a7-753f617969ef-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8\" (UID: \"c4b45501-1153-49f4-a9a7-753f617969ef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8" Jul 22 12:51:35.964880 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:35.964717 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8dw2\" (UniqueName: \"kubernetes.io/projected/c4b45501-1153-49f4-a9a7-753f617969ef-kube-api-access-f8dw2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8\" (UID: \"c4b45501-1153-49f4-a9a7-753f617969ef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8" Jul 22 12:51:35.964880 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:35.964792 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4b45501-1153-49f4-a9a7-753f617969ef-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8\" (UID: \"c4b45501-1153-49f4-a9a7-753f617969ef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8" Jul 22 12:51:35.964880 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:35.964869 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b45501-1153-49f4-a9a7-753f617969ef-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8\" (UID: \"c4b45501-1153-49f4-a9a7-753f617969ef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8" Jul 22 12:51:36.066118 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:36.066090 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4b45501-1153-49f4-a9a7-753f617969ef-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8\" (UID: \"c4b45501-1153-49f4-a9a7-753f617969ef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8" Jul 22 12:51:36.066118 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:36.066122 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4b45501-1153-49f4-a9a7-753f617969ef-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8\" (UID: \"c4b45501-1153-49f4-a9a7-753f617969ef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8" Jul 22 12:51:36.066320 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:36.066144 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4b45501-1153-49f4-a9a7-753f617969ef-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8\" (UID: \"c4b45501-1153-49f4-a9a7-753f617969ef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8" Jul 22 12:51:36.066320 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:36.066263 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8dw2\" (UniqueName: \"kubernetes.io/projected/c4b45501-1153-49f4-a9a7-753f617969ef-kube-api-access-f8dw2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8\" (UID: \"c4b45501-1153-49f4-a9a7-753f617969ef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8" Jul 22 12:51:36.066419 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:36.066323 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4b45501-1153-49f4-a9a7-753f617969ef-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8\" (UID: \"c4b45501-1153-49f4-a9a7-753f617969ef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8" Jul 22 12:51:36.066419 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:36.066399 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b45501-1153-49f4-a9a7-753f617969ef-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8\" (UID: \"c4b45501-1153-49f4-a9a7-753f617969ef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8" Jul 22 12:51:36.066515 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:36.066445 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4b45501-1153-49f4-a9a7-753f617969ef-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8\" (UID: \"c4b45501-1153-49f4-a9a7-753f617969ef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8" Jul 22 12:51:36.066515 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:36.066493 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4b45501-1153-49f4-a9a7-753f617969ef-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8\" (UID: \"c4b45501-1153-49f4-a9a7-753f617969ef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8" Jul 22 12:51:36.066612 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:36.066546 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4b45501-1153-49f4-a9a7-753f617969ef-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8\" (UID: \"c4b45501-1153-49f4-a9a7-753f617969ef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8" Jul 22 12:51:36.068433 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:36.068407 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4b45501-1153-49f4-a9a7-753f617969ef-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8\" (UID: \"c4b45501-1153-49f4-a9a7-753f617969ef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8" Jul 22 12:51:36.068744 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:36.068727 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4b45501-1153-49f4-a9a7-753f617969ef-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8\" (UID: \"c4b45501-1153-49f4-a9a7-753f617969ef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8" Jul 22 12:51:36.076161 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:36.076134 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8dw2\" (UniqueName: \"kubernetes.io/projected/c4b45501-1153-49f4-a9a7-753f617969ef-kube-api-access-f8dw2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8\" (UID: \"c4b45501-1153-49f4-a9a7-753f617969ef\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8" Jul 22 12:51:36.191330 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:36.191245 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8" Jul 22 12:51:36.334199 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:36.334168 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8"] Jul 22 12:51:36.336633 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:51:36.336607 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b45501_1153_49f4_a9a7_753f617969ef.slice/crio-2c4c618873037ea739f6c6dd898f0600a30ed81096a215396ee74a5be7477376 WatchSource:0}: Error finding container 2c4c618873037ea739f6c6dd898f0600a30ed81096a215396ee74a5be7477376: Status 404 returned error can't find the container with id 2c4c618873037ea739f6c6dd898f0600a30ed81096a215396ee74a5be7477376 Jul 22 12:51:36.338012 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:36.337993 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 12:51:36.818564 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:36.818532 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8" event={"ID":"c4b45501-1153-49f4-a9a7-753f617969ef","Type":"ContainerStarted","Data":"2c4c618873037ea739f6c6dd898f0600a30ed81096a215396ee74a5be7477376"} Jul 22 12:51:37.825053 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:37.825019 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8" event={"ID":"c4b45501-1153-49f4-a9a7-753f617969ef","Type":"ContainerStarted","Data":"62542a8e3bed99572c85e4f0f560d95c17e9a66b4870927db565e11c5efd3fdd"} Jul 22 12:51:37.857499 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:37.857447 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8" podStartSLOduration=1.713709461 podStartE2EDuration="2.857430539s" totalImagesPullingTime="1.143721078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:51:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:51:37.857057537 +0000 UTC m=+646.258288457" watchObservedRunningTime="2026-07-22 12:51:37.857430539 +0000 UTC m=+646.258661458" Jul 22 12:51:46.192297 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:46.192265 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8" Jul 22 12:51:46.192658 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:46.192316 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8" Jul 22 12:51:46.204870 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:46.204835 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8" Jul 22 12:51:46.877378 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:46.877349 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8" Jul 22 12:51:47.298816 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:47.298777 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5"] Jul 22 12:51:47.302427 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:47.302405 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5" Jul 22 12:51:47.307389 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:47.307353 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 22 12:51:47.314252 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:47.314229 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5"] Jul 22 12:51:47.365101 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:47.365060 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbd6c631-6153-4e93-94f4-b3f1e756ce5e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5\" (UID: \"bbd6c631-6153-4e93-94f4-b3f1e756ce5e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5" Jul 22 12:51:47.365240 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:47.365154 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv4kz\" (UniqueName: \"kubernetes.io/projected/bbd6c631-6153-4e93-94f4-b3f1e756ce5e-kube-api-access-lv4kz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5\" (UID: \"bbd6c631-6153-4e93-94f4-b3f1e756ce5e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5" Jul 22 12:51:47.365240 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:47.365201 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbd6c631-6153-4e93-94f4-b3f1e756ce5e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5\" (UID: \"bbd6c631-6153-4e93-94f4-b3f1e756ce5e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5" Jul 22 12:51:47.365240 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:47.365225 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbd6c631-6153-4e93-94f4-b3f1e756ce5e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5\" (UID: \"bbd6c631-6153-4e93-94f4-b3f1e756ce5e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5" Jul 22 12:51:47.365356 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:47.365260 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbd6c631-6153-4e93-94f4-b3f1e756ce5e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5\" (UID: \"bbd6c631-6153-4e93-94f4-b3f1e756ce5e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5" Jul 22 12:51:47.365356 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:47.365281 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbd6c631-6153-4e93-94f4-b3f1e756ce5e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5\" (UID: \"bbd6c631-6153-4e93-94f4-b3f1e756ce5e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5" Jul 22 12:51:47.466574 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:47.466535 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbd6c631-6153-4e93-94f4-b3f1e756ce5e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5\" (UID: \"bbd6c631-6153-4e93-94f4-b3f1e756ce5e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5" Jul 22 12:51:47.466574 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:47.466574 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbd6c631-6153-4e93-94f4-b3f1e756ce5e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5\" (UID: \"bbd6c631-6153-4e93-94f4-b3f1e756ce5e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5" Jul 22 12:51:47.466773 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:47.466604 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbd6c631-6153-4e93-94f4-b3f1e756ce5e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5\" (UID: \"bbd6c631-6153-4e93-94f4-b3f1e756ce5e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5" Jul 22 12:51:47.466773 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:47.466675 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv4kz\" (UniqueName: \"kubernetes.io/projected/bbd6c631-6153-4e93-94f4-b3f1e756ce5e-kube-api-access-lv4kz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5\" (UID: \"bbd6c631-6153-4e93-94f4-b3f1e756ce5e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5" Jul 22 12:51:47.466773 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:47.466719 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbd6c631-6153-4e93-94f4-b3f1e756ce5e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5\" (UID: \"bbd6c631-6153-4e93-94f4-b3f1e756ce5e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5" Jul 22 12:51:47.466773 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:47.466746 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbd6c631-6153-4e93-94f4-b3f1e756ce5e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5\" (UID: \"bbd6c631-6153-4e93-94f4-b3f1e756ce5e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5" Jul 22 12:51:47.467090 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:47.467059 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbd6c631-6153-4e93-94f4-b3f1e756ce5e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5\" (UID: \"bbd6c631-6153-4e93-94f4-b3f1e756ce5e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5" Jul 22 12:51:47.467251 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:47.467141 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbd6c631-6153-4e93-94f4-b3f1e756ce5e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5\" (UID: \"bbd6c631-6153-4e93-94f4-b3f1e756ce5e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5" Jul 22 12:51:47.467368 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:47.467337 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbd6c631-6153-4e93-94f4-b3f1e756ce5e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5\" (UID: \"bbd6c631-6153-4e93-94f4-b3f1e756ce5e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5" Jul 22 12:51:47.469288 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:47.469264 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbd6c631-6153-4e93-94f4-b3f1e756ce5e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5\" (UID: \"bbd6c631-6153-4e93-94f4-b3f1e756ce5e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5" Jul 22 12:51:47.469556 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:47.469539 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbd6c631-6153-4e93-94f4-b3f1e756ce5e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5\" (UID: \"bbd6c631-6153-4e93-94f4-b3f1e756ce5e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5" Jul 22 12:51:47.475791 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:47.475768 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv4kz\" (UniqueName: \"kubernetes.io/projected/bbd6c631-6153-4e93-94f4-b3f1e756ce5e-kube-api-access-lv4kz\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5\" (UID: \"bbd6c631-6153-4e93-94f4-b3f1e756ce5e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5" Jul 22 12:51:47.611550 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:47.611469 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5" Jul 22 12:51:47.747340 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:47.747317 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5"] Jul 22 12:51:47.749470 ip-10-0-129-34 kubenswrapper[2579]: W0722 12:51:47.749440 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbd6c631_6153_4e93_94f4_b3f1e756ce5e.slice/crio-cf248161cbc03c53e5daf50d35dd2e0653df569c9266d25dde33ff7896db3242 WatchSource:0}: Error finding container cf248161cbc03c53e5daf50d35dd2e0653df569c9266d25dde33ff7896db3242: Status 404 returned error can't find the container with id cf248161cbc03c53e5daf50d35dd2e0653df569c9266d25dde33ff7896db3242 Jul 22 12:51:47.871896 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:47.871821 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5" event={"ID":"bbd6c631-6153-4e93-94f4-b3f1e756ce5e","Type":"ContainerStarted","Data":"cf248161cbc03c53e5daf50d35dd2e0653df569c9266d25dde33ff7896db3242"} Jul 22 12:51:51.893624 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:51.893587 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5" event={"ID":"bbd6c631-6153-4e93-94f4-b3f1e756ce5e","Type":"ContainerStarted","Data":"edfe19aa332fc5e7c42d390ab6b2b8984ef8690329afb605dcaefd75377febe2"} Jul 22 12:51:51.914528 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:51.914475 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5" podStartSLOduration=1.225718297 podStartE2EDuration="4.914459322s" totalImagesPullingTime="3.688741025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 12:51:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 12:51:51.9133956 +0000 UTC m=+660.314626521" watchObservedRunningTime="2026-07-22 12:51:51.914459322 +0000 UTC m=+660.315690241" Jul 22 12:51:57.612329 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:57.612283 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5" Jul 22 12:51:57.612727 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:57.612343 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5" Jul 22 12:51:57.624595 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:57.624572 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5" Jul 22 12:51:57.935414 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:51:57.935331 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5" Jul 22 12:55:51.978250 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:55:51.978166 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ddj6_b28e0ca4-8512-45da-91ed-2cde1491b2e2/ovn-acl-logging/0.log" Jul 22 12:55:51.979368 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:55:51.979344 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ddj6_b28e0ca4-8512-45da-91ed-2cde1491b2e2/ovn-acl-logging/0.log" Jul 22 12:59:34.797422 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:59:34.797387 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85w2k"] Jul 22 12:59:34.797918 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:59:34.797580 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85w2k" podUID="3c11584f-1e2d-464b-96a9-ed0f48c0f930" containerName="manager" containerID="cri-o://728399d1272623f6c319dd01cbb35095611c7a30300f75fdfb0e78f3900e90ae" gracePeriod=10 Jul 22 12:59:38.050088 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:59:38.050062 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85w2k" Jul 22 12:59:38.127391 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:59:38.127316 2579 generic.go:356] "Generic (PLEG): container finished" podID="3c11584f-1e2d-464b-96a9-ed0f48c0f930" containerID="728399d1272623f6c319dd01cbb35095611c7a30300f75fdfb0e78f3900e90ae" exitCode=0 Jul 22 12:59:38.127391 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:59:38.127361 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85w2k" Jul 22 12:59:38.127557 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:59:38.127396 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85w2k" event={"ID":"3c11584f-1e2d-464b-96a9-ed0f48c0f930","Type":"ContainerDied","Data":"728399d1272623f6c319dd01cbb35095611c7a30300f75fdfb0e78f3900e90ae"} Jul 22 12:59:38.127557 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:59:38.127452 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85w2k" event={"ID":"3c11584f-1e2d-464b-96a9-ed0f48c0f930","Type":"ContainerDied","Data":"08e5e5dadcdd05c79022cdfedffb3ce52c31b929267cbe967bb71f59100fb53b"} Jul 22 12:59:38.127557 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:59:38.127482 2579 scope.go:117] "RemoveContainer" containerID="728399d1272623f6c319dd01cbb35095611c7a30300f75fdfb0e78f3900e90ae" Jul 22 12:59:38.136147 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:59:38.136130 2579 scope.go:117] "RemoveContainer" containerID="728399d1272623f6c319dd01cbb35095611c7a30300f75fdfb0e78f3900e90ae" Jul 22 12:59:38.136390 ip-10-0-129-34 kubenswrapper[2579]: E0722 12:59:38.136372 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"728399d1272623f6c319dd01cbb35095611c7a30300f75fdfb0e78f3900e90ae\": container with ID starting with 728399d1272623f6c319dd01cbb35095611c7a30300f75fdfb0e78f3900e90ae not found: ID does not exist" containerID="728399d1272623f6c319dd01cbb35095611c7a30300f75fdfb0e78f3900e90ae" Jul 22 12:59:38.136447 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:59:38.136398 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"728399d1272623f6c319dd01cbb35095611c7a30300f75fdfb0e78f3900e90ae"} err="failed to get container status \"728399d1272623f6c319dd01cbb35095611c7a30300f75fdfb0e78f3900e90ae\": rpc error: code = NotFound desc = could not find container \"728399d1272623f6c319dd01cbb35095611c7a30300f75fdfb0e78f3900e90ae\": container with ID starting with 728399d1272623f6c319dd01cbb35095611c7a30300f75fdfb0e78f3900e90ae not found: ID does not exist" Jul 22 12:59:38.202373 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:59:38.202335 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c11584f-1e2d-464b-96a9-ed0f48c0f930-extensions-socket-volume\") pod \"3c11584f-1e2d-464b-96a9-ed0f48c0f930\" (UID: \"3c11584f-1e2d-464b-96a9-ed0f48c0f930\") " Jul 22 12:59:38.202533 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:59:38.202422 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hskqg\" (UniqueName: \"kubernetes.io/projected/3c11584f-1e2d-464b-96a9-ed0f48c0f930-kube-api-access-hskqg\") pod \"3c11584f-1e2d-464b-96a9-ed0f48c0f930\" (UID: \"3c11584f-1e2d-464b-96a9-ed0f48c0f930\") " Jul 22 12:59:38.202719 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:59:38.202697 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c11584f-1e2d-464b-96a9-ed0f48c0f930-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3c11584f-1e2d-464b-96a9-ed0f48c0f930" (UID: "3c11584f-1e2d-464b-96a9-ed0f48c0f930"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 12:59:38.204462 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:59:38.204438 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c11584f-1e2d-464b-96a9-ed0f48c0f930-kube-api-access-hskqg" (OuterVolumeSpecName: "kube-api-access-hskqg") pod "3c11584f-1e2d-464b-96a9-ed0f48c0f930" (UID: "3c11584f-1e2d-464b-96a9-ed0f48c0f930"). InnerVolumeSpecName "kube-api-access-hskqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 12:59:38.303595 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:59:38.303568 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3c11584f-1e2d-464b-96a9-ed0f48c0f930-extensions-socket-volume\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:59:38.303595 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:59:38.303592 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hskqg\" (UniqueName: \"kubernetes.io/projected/3c11584f-1e2d-464b-96a9-ed0f48c0f930-kube-api-access-hskqg\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 12:59:38.450686 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:59:38.450660 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85w2k"] Jul 22 12:59:38.454379 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:59:38.454353 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-85w2k"] Jul 22 12:59:40.067918 ip-10-0-129-34 kubenswrapper[2579]: I0722 12:59:40.067882 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c11584f-1e2d-464b-96a9-ed0f48c0f930" path="/var/lib/kubelet/pods/3c11584f-1e2d-464b-96a9-ed0f48c0f930/volumes" Jul 22 13:00:00.148943 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:00.148893 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745420-skfnw"] Jul 22 13:00:00.149374 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:00.149278 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c11584f-1e2d-464b-96a9-ed0f48c0f930" containerName="manager" Jul 22 13:00:00.149374 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:00.149289 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c11584f-1e2d-464b-96a9-ed0f48c0f930" containerName="manager" Jul 22 13:00:00.149374 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:00.149363 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c11584f-1e2d-464b-96a9-ed0f48c0f930" containerName="manager" Jul 22 13:00:00.153564 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:00.153546 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745420-skfnw" Jul 22 13:00:00.156418 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:00.156395 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-85t5x\"" Jul 22 13:00:00.160230 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:00.160207 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745420-skfnw"] Jul 22 13:00:00.164388 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:00.164371 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745420-skfnw" Jul 22 13:00:00.294417 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:00.294387 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29745420-skfnw"] Jul 22 13:00:00.296039 ip-10-0-129-34 kubenswrapper[2579]: W0722 13:00:00.296015 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69b0193_ea31_438c_a92f_c1b866c1ed68.slice/crio-74d33249581e2aa286cd4f09f2d97bbe009e8bf7405a901b648c01335fc822d9 WatchSource:0}: Error finding container 74d33249581e2aa286cd4f09f2d97bbe009e8bf7405a901b648c01335fc822d9: Status 404 returned error can't find the container with id 74d33249581e2aa286cd4f09f2d97bbe009e8bf7405a901b648c01335fc822d9 Jul 22 13:00:00.298309 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:00.298293 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 22 13:00:01.238588 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:01.238550 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745420-skfnw" event={"ID":"b69b0193-ea31-438c-a92f-c1b866c1ed68","Type":"ContainerStarted","Data":"74d33249581e2aa286cd4f09f2d97bbe009e8bf7405a901b648c01335fc822d9"} Jul 22 13:00:03.256415 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:03.256376 2579 generic.go:356] "Generic (PLEG): container finished" podID="b69b0193-ea31-438c-a92f-c1b866c1ed68" containerID="54a02134be796ef2f9424a139cd517c16ec99143fc966f21c0d68a0d0f32419f" exitCode=0 Jul 22 13:00:03.256815 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:03.256422 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745420-skfnw" event={"ID":"b69b0193-ea31-438c-a92f-c1b866c1ed68","Type":"ContainerDied","Data":"54a02134be796ef2f9424a139cd517c16ec99143fc966f21c0d68a0d0f32419f"} Jul 22 13:00:04.393423 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:04.393398 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745420-skfnw" Jul 22 13:00:05.267266 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:05.267239 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745420-skfnw" Jul 22 13:00:05.267433 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:05.267286 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29745420-skfnw" event={"ID":"b69b0193-ea31-438c-a92f-c1b866c1ed68","Type":"ContainerDied","Data":"74d33249581e2aa286cd4f09f2d97bbe009e8bf7405a901b648c01335fc822d9"} Jul 22 13:00:05.267433 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:05.267327 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74d33249581e2aa286cd4f09f2d97bbe009e8bf7405a901b648c01335fc822d9" Jul 22 13:00:36.015115 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:36.015041 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76ks9"] Jul 22 13:00:36.015582 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:36.015508 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b69b0193-ea31-438c-a92f-c1b866c1ed68" containerName="cleanup" Jul 22 13:00:36.015582 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:36.015523 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b69b0193-ea31-438c-a92f-c1b866c1ed68" containerName="cleanup" Jul 22 13:00:36.015701 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:36.015587 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b69b0193-ea31-438c-a92f-c1b866c1ed68" containerName="cleanup" Jul 22 13:00:36.018735 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:36.018720 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76ks9" Jul 22 13:00:36.021622 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:36.021602 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hv48z\"" Jul 22 13:00:36.033839 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:36.033818 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76ks9"] Jul 22 13:00:36.122015 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:36.121977 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2cb8077a-631e-4ada-b4eb-2be7ce4afc27-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-76ks9\" (UID: \"2cb8077a-631e-4ada-b4eb-2be7ce4afc27\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76ks9" Jul 22 13:00:36.122185 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:36.122029 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjbzj\" (UniqueName: \"kubernetes.io/projected/2cb8077a-631e-4ada-b4eb-2be7ce4afc27-kube-api-access-jjbzj\") pod \"kuadrant-operator-controller-manager-86878c769f-76ks9\" (UID: \"2cb8077a-631e-4ada-b4eb-2be7ce4afc27\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76ks9" Jul 22 13:00:36.223286 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:36.223252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2cb8077a-631e-4ada-b4eb-2be7ce4afc27-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-76ks9\" (UID: \"2cb8077a-631e-4ada-b4eb-2be7ce4afc27\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76ks9" Jul 22 13:00:36.223456 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:36.223296 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjbzj\" (UniqueName: \"kubernetes.io/projected/2cb8077a-631e-4ada-b4eb-2be7ce4afc27-kube-api-access-jjbzj\") pod \"kuadrant-operator-controller-manager-86878c769f-76ks9\" (UID: \"2cb8077a-631e-4ada-b4eb-2be7ce4afc27\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76ks9" Jul 22 13:00:36.223682 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:36.223660 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2cb8077a-631e-4ada-b4eb-2be7ce4afc27-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-76ks9\" (UID: \"2cb8077a-631e-4ada-b4eb-2be7ce4afc27\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76ks9" Jul 22 13:00:36.234129 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:36.234104 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjbzj\" (UniqueName: \"kubernetes.io/projected/2cb8077a-631e-4ada-b4eb-2be7ce4afc27-kube-api-access-jjbzj\") pod \"kuadrant-operator-controller-manager-86878c769f-76ks9\" (UID: \"2cb8077a-631e-4ada-b4eb-2be7ce4afc27\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76ks9" Jul 22 13:00:36.328717 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:36.328686 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76ks9" Jul 22 13:00:36.488769 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:36.488733 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76ks9"] Jul 22 13:00:36.492624 ip-10-0-129-34 kubenswrapper[2579]: W0722 13:00:36.492594 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb8077a_631e_4ada_b4eb_2be7ce4afc27.slice/crio-56a151fd695e98e1745e3f30b4d00155f639340d7226c39292dbb6f4cf69334e WatchSource:0}: Error finding container 56a151fd695e98e1745e3f30b4d00155f639340d7226c39292dbb6f4cf69334e: Status 404 returned error can't find the container with id 56a151fd695e98e1745e3f30b4d00155f639340d7226c39292dbb6f4cf69334e Jul 22 13:00:37.424603 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:37.424569 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76ks9" event={"ID":"2cb8077a-631e-4ada-b4eb-2be7ce4afc27","Type":"ContainerStarted","Data":"3bf20a07c2ea7b47e9b973061e1c78539adc50609c3e91e27dd4026f8d7ee03b"} Jul 22 13:00:37.424986 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:37.424623 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76ks9" event={"ID":"2cb8077a-631e-4ada-b4eb-2be7ce4afc27","Type":"ContainerStarted","Data":"56a151fd695e98e1745e3f30b4d00155f639340d7226c39292dbb6f4cf69334e"} Jul 22 13:00:37.424986 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:37.424669 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76ks9" Jul 22 13:00:37.448332 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:37.448280 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76ks9" podStartSLOduration=2.448264054 podStartE2EDuration="2.448264054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:00:37.445488609 +0000 UTC m=+1185.846719527" watchObservedRunningTime="2026-07-22 13:00:37.448264054 +0000 UTC m=+1185.849494973" Jul 22 13:00:48.432106 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:48.432072 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-76ks9" Jul 22 13:00:52.016676 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:52.016648 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ddj6_b28e0ca4-8512-45da-91ed-2cde1491b2e2/ovn-acl-logging/0.log" Jul 22 13:00:52.017975 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:00:52.017902 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ddj6_b28e0ca4-8512-45da-91ed-2cde1491b2e2/ovn-acl-logging/0.log" Jul 22 13:03:37.495539 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:37.495502 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8"] Jul 22 13:03:37.499018 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:37.498999 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" Jul 22 13:03:37.502315 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:37.502288 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-20f81251-kserve-self-signed-certs\"" Jul 22 13:03:37.512016 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:37.511994 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8"] Jul 22 13:03:37.560229 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:37.560192 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/739c45d9-9c58-403d-a5a3-77879afac22b-model-cache\") pod \"e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8\" (UID: \"739c45d9-9c58-403d-a5a3-77879afac22b\") " pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" Jul 22 13:03:37.560406 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:37.560238 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/739c45d9-9c58-403d-a5a3-77879afac22b-tmp-dir\") pod \"e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8\" (UID: \"739c45d9-9c58-403d-a5a3-77879afac22b\") " pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" Jul 22 13:03:37.560406 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:37.560350 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/739c45d9-9c58-403d-a5a3-77879afac22b-dshm\") pod \"e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8\" (UID: \"739c45d9-9c58-403d-a5a3-77879afac22b\") " pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" Jul 22 13:03:37.560406 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:37.560391 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/739c45d9-9c58-403d-a5a3-77879afac22b-home\") pod \"e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8\" (UID: \"739c45d9-9c58-403d-a5a3-77879afac22b\") " pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" Jul 22 13:03:37.560539 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:37.560457 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk7mg\" (UniqueName: \"kubernetes.io/projected/739c45d9-9c58-403d-a5a3-77879afac22b-kube-api-access-pk7mg\") pod \"e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8\" (UID: \"739c45d9-9c58-403d-a5a3-77879afac22b\") " pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" Jul 22 13:03:37.560539 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:37.560498 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/739c45d9-9c58-403d-a5a3-77879afac22b-tls-certs\") pod \"e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8\" (UID: \"739c45d9-9c58-403d-a5a3-77879afac22b\") " pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" Jul 22 13:03:37.661893 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:37.661855 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/739c45d9-9c58-403d-a5a3-77879afac22b-dshm\") pod \"e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8\" (UID: \"739c45d9-9c58-403d-a5a3-77879afac22b\") " pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" Jul 22 13:03:37.661893 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:37.661894 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/739c45d9-9c58-403d-a5a3-77879afac22b-home\") pod \"e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8\" (UID: \"739c45d9-9c58-403d-a5a3-77879afac22b\") " pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" Jul 22 13:03:37.662154 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:37.661918 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk7mg\" (UniqueName: \"kubernetes.io/projected/739c45d9-9c58-403d-a5a3-77879afac22b-kube-api-access-pk7mg\") pod \"e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8\" (UID: \"739c45d9-9c58-403d-a5a3-77879afac22b\") " pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" Jul 22 13:03:37.662154 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:37.661982 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/739c45d9-9c58-403d-a5a3-77879afac22b-tls-certs\") pod \"e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8\" (UID: \"739c45d9-9c58-403d-a5a3-77879afac22b\") " pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" Jul 22 13:03:37.662154 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:37.662010 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/739c45d9-9c58-403d-a5a3-77879afac22b-model-cache\") pod \"e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8\" (UID: \"739c45d9-9c58-403d-a5a3-77879afac22b\") " pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" Jul 22 13:03:37.662154 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:37.662050 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/739c45d9-9c58-403d-a5a3-77879afac22b-tmp-dir\") pod \"e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8\" (UID: \"739c45d9-9c58-403d-a5a3-77879afac22b\") " pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" Jul 22 13:03:37.662395 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:37.662377 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/739c45d9-9c58-403d-a5a3-77879afac22b-home\") pod \"e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8\" (UID: \"739c45d9-9c58-403d-a5a3-77879afac22b\") " pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" Jul 22 13:03:37.662474 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:37.662428 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/739c45d9-9c58-403d-a5a3-77879afac22b-model-cache\") pod \"e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8\" (UID: \"739c45d9-9c58-403d-a5a3-77879afac22b\") " pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" Jul 22 13:03:37.662474 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:37.662454 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/739c45d9-9c58-403d-a5a3-77879afac22b-tmp-dir\") pod \"e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8\" (UID: \"739c45d9-9c58-403d-a5a3-77879afac22b\") " pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" Jul 22 13:03:37.664181 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:37.664157 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/739c45d9-9c58-403d-a5a3-77879afac22b-dshm\") pod \"e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8\" (UID: \"739c45d9-9c58-403d-a5a3-77879afac22b\") " pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" Jul 22 13:03:37.664595 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:37.664573 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/739c45d9-9c58-403d-a5a3-77879afac22b-tls-certs\") pod \"e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8\" (UID: \"739c45d9-9c58-403d-a5a3-77879afac22b\") " pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" Jul 22 13:03:37.670911 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:37.670887 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk7mg\" (UniqueName: \"kubernetes.io/projected/739c45d9-9c58-403d-a5a3-77879afac22b-kube-api-access-pk7mg\") pod \"e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8\" (UID: \"739c45d9-9c58-403d-a5a3-77879afac22b\") " pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" Jul 22 13:03:37.808163 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:37.808129 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" Jul 22 13:03:37.943007 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:37.942981 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8"] Jul 22 13:03:37.945737 ip-10-0-129-34 kubenswrapper[2579]: W0722 13:03:37.945711 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod739c45d9_9c58_403d_a5a3_77879afac22b.slice/crio-2da3c63fcb30ecd746ec2a4a10660380113339b28da85e9f5b0fd7f98b2c0672 WatchSource:0}: Error finding container 2da3c63fcb30ecd746ec2a4a10660380113339b28da85e9f5b0fd7f98b2c0672: Status 404 returned error can't find the container with id 2da3c63fcb30ecd746ec2a4a10660380113339b28da85e9f5b0fd7f98b2c0672 Jul 22 13:03:38.304498 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:38.304461 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" event={"ID":"739c45d9-9c58-403d-a5a3-77879afac22b","Type":"ContainerStarted","Data":"cd402856dea802c0ad9aa5f655ae9579783e91374ea182172f93568384476f38"} Jul 22 13:03:38.304498 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:38.304504 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" event={"ID":"739c45d9-9c58-403d-a5a3-77879afac22b","Type":"ContainerStarted","Data":"2da3c63fcb30ecd746ec2a4a10660380113339b28da85e9f5b0fd7f98b2c0672"} Jul 22 13:03:38.324702 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:38.324650 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" podStartSLOduration=1.324635008 podStartE2EDuration="1.324635008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:03:38.322868068 +0000 UTC m=+1366.724098989" watchObservedRunningTime="2026-07-22 13:03:38.324635008 +0000 UTC m=+1366.725865928" Jul 22 13:03:47.808281 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:47.808239 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" Jul 22 13:03:47.808685 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:47.808296 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" Jul 22 13:03:47.820723 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:47.820700 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" Jul 22 13:03:48.374690 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:03:48.374662 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" Jul 22 13:04:04.298526 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:04.298492 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8"] Jul 22 13:04:19.318436 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:19.318392 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" podUID="739c45d9-9c58-403d-a5a3-77879afac22b" containerName="main" containerID="cri-o://cd402856dea802c0ad9aa5f655ae9579783e91374ea182172f93568384476f38" gracePeriod=45 Jul 22 13:04:19.515878 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:19.515844 2579 generic.go:356] "Generic (PLEG): container finished" podID="739c45d9-9c58-403d-a5a3-77879afac22b" containerID="cd402856dea802c0ad9aa5f655ae9579783e91374ea182172f93568384476f38" exitCode=0 Jul 22 13:04:19.516071 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:19.515912 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" event={"ID":"739c45d9-9c58-403d-a5a3-77879afac22b","Type":"ContainerDied","Data":"cd402856dea802c0ad9aa5f655ae9579783e91374ea182172f93568384476f38"} Jul 22 13:04:19.566789 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:19.566766 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" Jul 22 13:04:19.642471 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:19.642387 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/739c45d9-9c58-403d-a5a3-77879afac22b-model-cache\") pod \"739c45d9-9c58-403d-a5a3-77879afac22b\" (UID: \"739c45d9-9c58-403d-a5a3-77879afac22b\") " Jul 22 13:04:19.642471 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:19.642426 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/739c45d9-9c58-403d-a5a3-77879afac22b-dshm\") pod \"739c45d9-9c58-403d-a5a3-77879afac22b\" (UID: \"739c45d9-9c58-403d-a5a3-77879afac22b\") " Jul 22 13:04:19.642471 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:19.642463 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pk7mg\" (UniqueName: \"kubernetes.io/projected/739c45d9-9c58-403d-a5a3-77879afac22b-kube-api-access-pk7mg\") pod \"739c45d9-9c58-403d-a5a3-77879afac22b\" (UID: \"739c45d9-9c58-403d-a5a3-77879afac22b\") " Jul 22 13:04:19.642732 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:19.642500 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/739c45d9-9c58-403d-a5a3-77879afac22b-tmp-dir\") pod \"739c45d9-9c58-403d-a5a3-77879afac22b\" (UID: \"739c45d9-9c58-403d-a5a3-77879afac22b\") " Jul 22 13:04:19.642732 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:19.642568 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/739c45d9-9c58-403d-a5a3-77879afac22b-home\") pod \"739c45d9-9c58-403d-a5a3-77879afac22b\" (UID: \"739c45d9-9c58-403d-a5a3-77879afac22b\") " Jul 22 13:04:19.642732 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:19.642616 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/739c45d9-9c58-403d-a5a3-77879afac22b-tls-certs\") pod \"739c45d9-9c58-403d-a5a3-77879afac22b\" (UID: \"739c45d9-9c58-403d-a5a3-77879afac22b\") " Jul 22 13:04:19.642732 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:19.642706 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/739c45d9-9c58-403d-a5a3-77879afac22b-model-cache" (OuterVolumeSpecName: "model-cache") pod "739c45d9-9c58-403d-a5a3-77879afac22b" (UID: "739c45d9-9c58-403d-a5a3-77879afac22b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:04:19.643022 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:19.642918 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/739c45d9-9c58-403d-a5a3-77879afac22b-model-cache\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 13:04:19.643090 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:19.643021 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/739c45d9-9c58-403d-a5a3-77879afac22b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "739c45d9-9c58-403d-a5a3-77879afac22b" (UID: "739c45d9-9c58-403d-a5a3-77879afac22b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:04:19.643090 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:19.643076 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/739c45d9-9c58-403d-a5a3-77879afac22b-home" (OuterVolumeSpecName: "home") pod "739c45d9-9c58-403d-a5a3-77879afac22b" (UID: "739c45d9-9c58-403d-a5a3-77879afac22b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:04:19.644758 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:19.644729 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/739c45d9-9c58-403d-a5a3-77879afac22b-kube-api-access-pk7mg" (OuterVolumeSpecName: "kube-api-access-pk7mg") pod "739c45d9-9c58-403d-a5a3-77879afac22b" (UID: "739c45d9-9c58-403d-a5a3-77879afac22b"). InnerVolumeSpecName "kube-api-access-pk7mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:04:19.644887 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:19.644758 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/739c45d9-9c58-403d-a5a3-77879afac22b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "739c45d9-9c58-403d-a5a3-77879afac22b" (UID: "739c45d9-9c58-403d-a5a3-77879afac22b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 13:04:19.644887 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:19.644764 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/739c45d9-9c58-403d-a5a3-77879afac22b-dshm" (OuterVolumeSpecName: "dshm") pod "739c45d9-9c58-403d-a5a3-77879afac22b" (UID: "739c45d9-9c58-403d-a5a3-77879afac22b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:04:19.743423 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:19.743392 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/739c45d9-9c58-403d-a5a3-77879afac22b-tmp-dir\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 13:04:19.743423 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:19.743417 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/739c45d9-9c58-403d-a5a3-77879afac22b-home\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 13:04:19.743423 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:19.743425 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/739c45d9-9c58-403d-a5a3-77879afac22b-tls-certs\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 13:04:19.743423 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:19.743434 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/739c45d9-9c58-403d-a5a3-77879afac22b-dshm\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 13:04:19.743751 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:19.743444 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk7mg\" (UniqueName: \"kubernetes.io/projected/739c45d9-9c58-403d-a5a3-77879afac22b-kube-api-access-pk7mg\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 13:04:20.522438 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:20.522363 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" event={"ID":"739c45d9-9c58-403d-a5a3-77879afac22b","Type":"ContainerDied","Data":"2da3c63fcb30ecd746ec2a4a10660380113339b28da85e9f5b0fd7f98b2c0672"} Jul 22 13:04:20.522438 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:20.522405 2579 scope.go:117] "RemoveContainer" containerID="cd402856dea802c0ad9aa5f655ae9579783e91374ea182172f93568384476f38" Jul 22 13:04:20.522850 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:20.522370 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8" Jul 22 13:04:20.541738 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:20.541712 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8"] Jul 22 13:04:20.545167 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:20.545139 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-20f81251-kserve-6f9cfcfb5f-gvnb8"] Jul 22 13:04:22.067796 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:04:22.067765 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="739c45d9-9c58-403d-a5a3-77879afac22b" path="/var/lib/kubelet/pods/739c45d9-9c58-403d-a5a3-77879afac22b/volumes" Jul 22 13:05:52.050864 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:05:52.050834 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ddj6_b28e0ca4-8512-45da-91ed-2cde1491b2e2/ovn-acl-logging/0.log" Jul 22 13:05:52.053224 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:05:52.053203 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ddj6_b28e0ca4-8512-45da-91ed-2cde1491b2e2/ovn-acl-logging/0.log" Jul 22 13:10:52.084345 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:10:52.084313 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ddj6_b28e0ca4-8512-45da-91ed-2cde1491b2e2/ovn-acl-logging/0.log" Jul 22 13:10:52.087059 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:10:52.087029 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ddj6_b28e0ca4-8512-45da-91ed-2cde1491b2e2/ovn-acl-logging/0.log" Jul 22 13:14:17.844463 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:14:17.844432 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7caca13d-5m8x7"] Jul 22 13:14:17.845088 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:14:17.845070 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="739c45d9-9c58-403d-a5a3-77879afac22b" containerName="main" Jul 22 13:14:17.845088 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:14:17.845090 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="739c45d9-9c58-403d-a5a3-77879afac22b" containerName="main" Jul 22 13:14:17.845222 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:14:17.845173 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="739c45d9-9c58-403d-a5a3-77879afac22b" containerName="main" Jul 22 13:14:17.848309 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:14:17.848288 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7caca13d-5m8x7" Jul 22 13:14:17.852388 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:14:17.852367 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-85t5x\"" Jul 22 13:14:17.863936 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:14:17.863893 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7caca13d-5m8x7"] Jul 22 13:14:17.941791 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:14:17.941758 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47cb8cd-01cd-41df-ae9d-547e90fe8e45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7caca13d-5m8x7\" (UID: \"a47cb8cd-01cd-41df-ae9d-547e90fe8e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7caca13d-5m8x7" Jul 22 13:14:18.042900 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:14:18.042869 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47cb8cd-01cd-41df-ae9d-547e90fe8e45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7caca13d-5m8x7\" (UID: \"a47cb8cd-01cd-41df-ae9d-547e90fe8e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7caca13d-5m8x7" Jul 22 13:14:18.043502 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:14:18.043474 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47cb8cd-01cd-41df-ae9d-547e90fe8e45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-7caca13d-5m8x7\" (UID: \"a47cb8cd-01cd-41df-ae9d-547e90fe8e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7caca13d-5m8x7" Jul 22 13:14:18.158808 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:14:18.158729 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7caca13d-5m8x7" Jul 22 13:14:18.496309 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:14:18.496285 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7caca13d-5m8x7"] Jul 22 13:14:18.498118 ip-10-0-129-34 kubenswrapper[2579]: W0722 13:14:18.498086 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47cb8cd_01cd_41df_ae9d_547e90fe8e45.slice/crio-8c7c08df18b3ce29c09c00e12337c04232357e594d2baea995b7b69dfa206aef WatchSource:0}: Error finding container 8c7c08df18b3ce29c09c00e12337c04232357e594d2baea995b7b69dfa206aef: Status 404 returned error can't find the container with id 8c7c08df18b3ce29c09c00e12337c04232357e594d2baea995b7b69dfa206aef Jul 22 13:14:19.405245 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:14:19.405211 2579 generic.go:356] "Generic (PLEG): container finished" podID="a47cb8cd-01cd-41df-ae9d-547e90fe8e45" containerID="19aa5d38c96cd56c7638f08b6a3cbfff2392aadfa3daaf41fc43eea84afe21f0" exitCode=0 Jul 22 13:14:19.405601 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:14:19.405270 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7caca13d-5m8x7" event={"ID":"a47cb8cd-01cd-41df-ae9d-547e90fe8e45","Type":"ContainerDied","Data":"19aa5d38c96cd56c7638f08b6a3cbfff2392aadfa3daaf41fc43eea84afe21f0"} Jul 22 13:14:19.405601 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:14:19.405322 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7caca13d-5m8x7" event={"ID":"a47cb8cd-01cd-41df-ae9d-547e90fe8e45","Type":"ContainerStarted","Data":"8c7c08df18b3ce29c09c00e12337c04232357e594d2baea995b7b69dfa206aef"} Jul 22 13:14:20.549226 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:14:20.549202 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7caca13d-5m8x7" Jul 22 13:14:20.668683 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:14:20.668594 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47cb8cd-01cd-41df-ae9d-547e90fe8e45-maas-api-service-ca\") pod \"a47cb8cd-01cd-41df-ae9d-547e90fe8e45\" (UID: \"a47cb8cd-01cd-41df-ae9d-547e90fe8e45\") " Jul 22 13:14:20.669032 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:14:20.668997 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a47cb8cd-01cd-41df-ae9d-547e90fe8e45-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a47cb8cd-01cd-41df-ae9d-547e90fe8e45" (UID: "a47cb8cd-01cd-41df-ae9d-547e90fe8e45"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 22 13:14:20.769781 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:14:20.769749 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47cb8cd-01cd-41df-ae9d-547e90fe8e45-maas-api-service-ca\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 13:14:21.416107 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:14:21.416074 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7caca13d-5m8x7" event={"ID":"a47cb8cd-01cd-41df-ae9d-547e90fe8e45","Type":"ContainerDied","Data":"8c7c08df18b3ce29c09c00e12337c04232357e594d2baea995b7b69dfa206aef"} Jul 22 13:14:21.416271 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:14:21.416124 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c7c08df18b3ce29c09c00e12337c04232357e594d2baea995b7b69dfa206aef" Jul 22 13:14:21.416271 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:14:21.416086 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7caca13d-5m8x7" Jul 22 13:14:37.931248 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:14:37.931207 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7caca13d-5m8x7"] Jul 22 13:14:37.935329 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:14:37.935304 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-7caca13d-5m8x7"] Jul 22 13:14:38.067602 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:14:38.067568 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a47cb8cd-01cd-41df-ae9d-547e90fe8e45" path="/var/lib/kubelet/pods/a47cb8cd-01cd-41df-ae9d-547e90fe8e45/volumes" Jul 22 13:15:52.114712 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:15:52.114686 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ddj6_b28e0ca4-8512-45da-91ed-2cde1491b2e2/ovn-acl-logging/0.log" Jul 22 13:15:52.118244 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:15:52.118221 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ddj6_b28e0ca4-8512-45da-91ed-2cde1491b2e2/ovn-acl-logging/0.log" Jul 22 13:16:54.554500 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.554424 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt"] Jul 22 13:16:54.554988 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.554848 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a47cb8cd-01cd-41df-ae9d-547e90fe8e45" containerName="revoke-keys" Jul 22 13:16:54.554988 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.554861 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a47cb8cd-01cd-41df-ae9d-547e90fe8e45" containerName="revoke-keys" Jul 22 13:16:54.554988 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.554954 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a47cb8cd-01cd-41df-ae9d-547e90fe8e45" containerName="revoke-keys" Jul 22 13:16:54.558093 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.558075 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" Jul 22 13:16:54.562058 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.562037 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-50f07d\"/\"default-dockercfg-txx4t\"" Jul 22 13:16:54.562174 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.562068 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-50f07d\"/\"kube-root-ca.crt\"" Jul 22 13:16:54.562174 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.562037 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-50f07d\"/\"test-model-50f07d-kserve-self-signed-certs\"" Jul 22 13:16:54.562174 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.562038 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-50f07d\"/\"openshift-service-ca.crt\"" Jul 22 13:16:54.568704 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.568682 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt"] Jul 22 13:16:54.666679 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.666638 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-model-cache\") pod \"test-model-50f07d-kserve-546cb4bd69-znhlt\" (UID: \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\") " pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" Jul 22 13:16:54.666851 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.666710 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-dshm\") pod \"test-model-50f07d-kserve-546cb4bd69-znhlt\" (UID: \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\") " pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" Jul 22 13:16:54.666851 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.666756 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-home\") pod \"test-model-50f07d-kserve-546cb4bd69-znhlt\" (UID: \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\") " pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" Jul 22 13:16:54.666851 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.666778 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-tls-certs\") pod \"test-model-50f07d-kserve-546cb4bd69-znhlt\" (UID: \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\") " pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" Jul 22 13:16:54.666851 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.666839 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-tmp-dir\") pod \"test-model-50f07d-kserve-546cb4bd69-znhlt\" (UID: \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\") " pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" Jul 22 13:16:54.667024 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.666858 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b458c\" (UniqueName: \"kubernetes.io/projected/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-kube-api-access-b458c\") pod \"test-model-50f07d-kserve-546cb4bd69-znhlt\" (UID: \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\") " pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" Jul 22 13:16:54.767769 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.767731 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-dshm\") pod \"test-model-50f07d-kserve-546cb4bd69-znhlt\" (UID: \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\") " pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" Jul 22 13:16:54.767984 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.767787 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-home\") pod \"test-model-50f07d-kserve-546cb4bd69-znhlt\" (UID: \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\") " pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" Jul 22 13:16:54.767984 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.767806 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-tls-certs\") pod \"test-model-50f07d-kserve-546cb4bd69-znhlt\" (UID: \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\") " pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" Jul 22 13:16:54.767984 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.767841 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-tmp-dir\") pod \"test-model-50f07d-kserve-546cb4bd69-znhlt\" (UID: \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\") " pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" Jul 22 13:16:54.767984 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.767861 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b458c\" (UniqueName: \"kubernetes.io/projected/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-kube-api-access-b458c\") pod \"test-model-50f07d-kserve-546cb4bd69-znhlt\" (UID: \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\") " pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" Jul 22 13:16:54.767984 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.767899 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-model-cache\") pod \"test-model-50f07d-kserve-546cb4bd69-znhlt\" (UID: \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\") " pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" Jul 22 13:16:54.768290 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.768254 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-home\") pod \"test-model-50f07d-kserve-546cb4bd69-znhlt\" (UID: \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\") " pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" Jul 22 13:16:54.768290 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.768280 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-tmp-dir\") pod \"test-model-50f07d-kserve-546cb4bd69-znhlt\" (UID: \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\") " pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" Jul 22 13:16:54.768434 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.768318 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-model-cache\") pod \"test-model-50f07d-kserve-546cb4bd69-znhlt\" (UID: \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\") " pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" Jul 22 13:16:54.770117 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.770099 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-dshm\") pod \"test-model-50f07d-kserve-546cb4bd69-znhlt\" (UID: \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\") " pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" Jul 22 13:16:54.770291 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.770275 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-tls-certs\") pod \"test-model-50f07d-kserve-546cb4bd69-znhlt\" (UID: \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\") " pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" Jul 22 13:16:54.777747 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.777722 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b458c\" (UniqueName: \"kubernetes.io/projected/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-kube-api-access-b458c\") pod \"test-model-50f07d-kserve-546cb4bd69-znhlt\" (UID: \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\") " pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" Jul 22 13:16:54.867894 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:54.867810 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" Jul 22 13:16:55.002382 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:55.002285 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt"] Jul 22 13:16:55.004976 ip-10-0-129-34 kubenswrapper[2579]: W0722 13:16:55.004948 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e2e0f76_9b5b_4d6a_9730_6db0f4ee909d.slice/crio-4b171c8a472e9e7527ba8461753eb7c7d0304895a9e487e1a8303d94fe32e517 WatchSource:0}: Error finding container 4b171c8a472e9e7527ba8461753eb7c7d0304895a9e487e1a8303d94fe32e517: Status 404 returned error can't find the container with id 4b171c8a472e9e7527ba8461753eb7c7d0304895a9e487e1a8303d94fe32e517 Jul 22 13:16:55.182853 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:55.182771 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" event={"ID":"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d","Type":"ContainerStarted","Data":"b458b955a7b872c4b5ed95d4db541a4332cb959df06cbeac1db66388347c7d2b"} Jul 22 13:16:55.182853 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:55.182812 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" event={"ID":"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d","Type":"ContainerStarted","Data":"4b171c8a472e9e7527ba8461753eb7c7d0304895a9e487e1a8303d94fe32e517"} Jul 22 13:16:55.203762 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:16:55.203714 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" podStartSLOduration=1.203699308 podStartE2EDuration="1.203699308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:16:55.202202687 +0000 UTC m=+2163.603433609" watchObservedRunningTime="2026-07-22 13:16:55.203699308 +0000 UTC m=+2163.604930230" Jul 22 13:17:04.868308 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:17:04.868269 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" Jul 22 13:17:04.868716 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:17:04.868332 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" Jul 22 13:17:04.880520 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:17:04.880496 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" Jul 22 13:17:05.242504 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:17:05.242430 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" Jul 22 13:17:58.363103 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:17:58.363065 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt"] Jul 22 13:18:13.383051 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:13.383007 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" podUID="6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d" containerName="main" containerID="cri-o://b458b955a7b872c4b5ed95d4db541a4332cb959df06cbeac1db66388347c7d2b" gracePeriod=45 Jul 22 13:18:13.573644 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:13.573575 2579 generic.go:356] "Generic (PLEG): container finished" podID="6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d" containerID="b458b955a7b872c4b5ed95d4db541a4332cb959df06cbeac1db66388347c7d2b" exitCode=0 Jul 22 13:18:13.573644 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:13.573625 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" event={"ID":"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d","Type":"ContainerDied","Data":"b458b955a7b872c4b5ed95d4db541a4332cb959df06cbeac1db66388347c7d2b"} Jul 22 13:18:13.639109 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:13.639046 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" Jul 22 13:18:13.690510 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:13.690475 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b458c\" (UniqueName: \"kubernetes.io/projected/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-kube-api-access-b458c\") pod \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\" (UID: \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\") " Jul 22 13:18:13.690689 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:13.690523 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-model-cache\") pod \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\" (UID: \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\") " Jul 22 13:18:13.690689 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:13.690546 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-tls-certs\") pod \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\" (UID: \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\") " Jul 22 13:18:13.690689 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:13.690561 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-tmp-dir\") pod \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\" (UID: \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\") " Jul 22 13:18:13.690689 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:13.690604 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-home\") pod \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\" (UID: \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\") " Jul 22 13:18:13.690689 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:13.690646 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-dshm\") pod \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\" (UID: \"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d\") " Jul 22 13:18:13.690961 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:13.690841 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-model-cache" (OuterVolumeSpecName: "model-cache") pod "6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d" (UID: "6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:18:13.690961 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:13.690912 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-model-cache\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 13:18:13.691048 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:13.690977 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-home" (OuterVolumeSpecName: "home") pod "6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d" (UID: "6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:18:13.691048 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:13.691002 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d" (UID: "6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:18:13.692838 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:13.692817 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d" (UID: "6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 13:18:13.692838 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:13.692827 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-dshm" (OuterVolumeSpecName: "dshm") pod "6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d" (UID: "6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:18:13.692838 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:13.692832 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-kube-api-access-b458c" (OuterVolumeSpecName: "kube-api-access-b458c") pod "6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d" (UID: "6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d"). InnerVolumeSpecName "kube-api-access-b458c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:18:13.791508 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:13.791471 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b458c\" (UniqueName: \"kubernetes.io/projected/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-kube-api-access-b458c\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 13:18:13.791508 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:13.791501 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-tls-certs\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 13:18:13.791508 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:13.791511 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-tmp-dir\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 13:18:13.791508 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:13.791519 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-home\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 13:18:13.791753 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:13.791526 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d-dshm\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 13:18:14.580083 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:14.580054 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" Jul 22 13:18:14.580537 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:14.580106 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt" event={"ID":"6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d","Type":"ContainerDied","Data":"4b171c8a472e9e7527ba8461753eb7c7d0304895a9e487e1a8303d94fe32e517"} Jul 22 13:18:14.580537 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:14.580172 2579 scope.go:117] "RemoveContainer" containerID="b458b955a7b872c4b5ed95d4db541a4332cb959df06cbeac1db66388347c7d2b" Jul 22 13:18:14.597261 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:14.597232 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt"] Jul 22 13:18:14.603813 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:14.603791 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-50f07d/test-model-50f07d-kserve-546cb4bd69-znhlt"] Jul 22 13:18:16.067789 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:18:16.067755 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d" path="/var/lib/kubelet/pods/6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d/volumes" Jul 22 13:20:52.151648 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:20:52.151537 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ddj6_b28e0ca4-8512-45da-91ed-2cde1491b2e2/ovn-acl-logging/0.log" Jul 22 13:20:52.155720 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:20:52.155392 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ddj6_b28e0ca4-8512-45da-91ed-2cde1491b2e2/ovn-acl-logging/0.log" Jul 22 13:20:52.205665 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:20:52.205638 2579 scope.go:117] "RemoveContainer" containerID="19aa5d38c96cd56c7638f08b6a3cbfff2392aadfa3daaf41fc43eea84afe21f0" Jul 22 13:23:26.028497 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.028462 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk"] Jul 22 13:23:26.028907 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.028899 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d" containerName="main" Jul 22 13:23:26.028974 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.028911 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d" containerName="main" Jul 22 13:23:26.029107 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.029011 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e2e0f76-9b5b-4d6a-9730-6db0f4ee909d" containerName="main" Jul 22 13:23:26.032127 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.032109 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" Jul 22 13:23:26.036096 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.036077 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d24327\"/\"kube-root-ca.crt\"" Jul 22 13:23:26.036206 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.036076 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d24327\"/\"default-dockercfg-nctkf\"" Jul 22 13:23:26.036575 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.036555 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d24327\"/\"openshift-service-ca.crt\"" Jul 22 13:23:26.036662 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.036644 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d24327\"/\"sub-test-model-d24327-kserve-self-signed-certs\"" Jul 22 13:23:26.051826 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.051796 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk"] Jul 22 13:23:26.145433 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.145405 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e60460c9-8a71-4877-a27f-05cefd5b4641-dshm\") pod \"sub-test-model-d24327-kserve-69775b45c5-dpfrk\" (UID: \"e60460c9-8a71-4877-a27f-05cefd5b4641\") " pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" Jul 22 13:23:26.145619 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.145441 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e60460c9-8a71-4877-a27f-05cefd5b4641-tls-certs\") pod \"sub-test-model-d24327-kserve-69775b45c5-dpfrk\" (UID: \"e60460c9-8a71-4877-a27f-05cefd5b4641\") " pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" Jul 22 13:23:26.145619 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.145460 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2mnp\" (UniqueName: \"kubernetes.io/projected/e60460c9-8a71-4877-a27f-05cefd5b4641-kube-api-access-s2mnp\") pod \"sub-test-model-d24327-kserve-69775b45c5-dpfrk\" (UID: \"e60460c9-8a71-4877-a27f-05cefd5b4641\") " pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" Jul 22 13:23:26.145619 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.145560 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e60460c9-8a71-4877-a27f-05cefd5b4641-tmp-dir\") pod \"sub-test-model-d24327-kserve-69775b45c5-dpfrk\" (UID: \"e60460c9-8a71-4877-a27f-05cefd5b4641\") " pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" Jul 22 13:23:26.145768 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.145700 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e60460c9-8a71-4877-a27f-05cefd5b4641-model-cache\") pod \"sub-test-model-d24327-kserve-69775b45c5-dpfrk\" (UID: \"e60460c9-8a71-4877-a27f-05cefd5b4641\") " pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" Jul 22 13:23:26.145768 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.145731 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e60460c9-8a71-4877-a27f-05cefd5b4641-home\") pod \"sub-test-model-d24327-kserve-69775b45c5-dpfrk\" (UID: \"e60460c9-8a71-4877-a27f-05cefd5b4641\") " pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" Jul 22 13:23:26.246971 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.246913 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e60460c9-8a71-4877-a27f-05cefd5b4641-tmp-dir\") pod \"sub-test-model-d24327-kserve-69775b45c5-dpfrk\" (UID: \"e60460c9-8a71-4877-a27f-05cefd5b4641\") " pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" Jul 22 13:23:26.247143 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.247078 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e60460c9-8a71-4877-a27f-05cefd5b4641-model-cache\") pod \"sub-test-model-d24327-kserve-69775b45c5-dpfrk\" (UID: \"e60460c9-8a71-4877-a27f-05cefd5b4641\") " pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" Jul 22 13:23:26.247143 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.247116 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e60460c9-8a71-4877-a27f-05cefd5b4641-home\") pod \"sub-test-model-d24327-kserve-69775b45c5-dpfrk\" (UID: \"e60460c9-8a71-4877-a27f-05cefd5b4641\") " pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" Jul 22 13:23:26.247255 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.247190 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e60460c9-8a71-4877-a27f-05cefd5b4641-dshm\") pod \"sub-test-model-d24327-kserve-69775b45c5-dpfrk\" (UID: \"e60460c9-8a71-4877-a27f-05cefd5b4641\") " pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" Jul 22 13:23:26.247255 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.247223 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e60460c9-8a71-4877-a27f-05cefd5b4641-tls-certs\") pod \"sub-test-model-d24327-kserve-69775b45c5-dpfrk\" (UID: \"e60460c9-8a71-4877-a27f-05cefd5b4641\") " pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" Jul 22 13:23:26.247255 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.247249 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2mnp\" (UniqueName: \"kubernetes.io/projected/e60460c9-8a71-4877-a27f-05cefd5b4641-kube-api-access-s2mnp\") pod \"sub-test-model-d24327-kserve-69775b45c5-dpfrk\" (UID: \"e60460c9-8a71-4877-a27f-05cefd5b4641\") " pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" Jul 22 13:23:26.247420 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.247299 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e60460c9-8a71-4877-a27f-05cefd5b4641-tmp-dir\") pod \"sub-test-model-d24327-kserve-69775b45c5-dpfrk\" (UID: \"e60460c9-8a71-4877-a27f-05cefd5b4641\") " pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" Jul 22 13:23:26.247477 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.247435 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e60460c9-8a71-4877-a27f-05cefd5b4641-model-cache\") pod \"sub-test-model-d24327-kserve-69775b45c5-dpfrk\" (UID: \"e60460c9-8a71-4877-a27f-05cefd5b4641\") " pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" Jul 22 13:23:26.247531 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.247511 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e60460c9-8a71-4877-a27f-05cefd5b4641-home\") pod \"sub-test-model-d24327-kserve-69775b45c5-dpfrk\" (UID: \"e60460c9-8a71-4877-a27f-05cefd5b4641\") " pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" Jul 22 13:23:26.249471 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.249438 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e60460c9-8a71-4877-a27f-05cefd5b4641-dshm\") pod \"sub-test-model-d24327-kserve-69775b45c5-dpfrk\" (UID: \"e60460c9-8a71-4877-a27f-05cefd5b4641\") " pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" Jul 22 13:23:26.249795 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.249779 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e60460c9-8a71-4877-a27f-05cefd5b4641-tls-certs\") pod \"sub-test-model-d24327-kserve-69775b45c5-dpfrk\" (UID: \"e60460c9-8a71-4877-a27f-05cefd5b4641\") " pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" Jul 22 13:23:26.260682 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.260659 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2mnp\" (UniqueName: \"kubernetes.io/projected/e60460c9-8a71-4877-a27f-05cefd5b4641-kube-api-access-s2mnp\") pod \"sub-test-model-d24327-kserve-69775b45c5-dpfrk\" (UID: \"e60460c9-8a71-4877-a27f-05cefd5b4641\") " pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" Jul 22 13:23:26.341099 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.341070 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" Jul 22 13:23:26.477779 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:26.477753 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk"] Jul 22 13:23:26.479488 ip-10-0-129-34 kubenswrapper[2579]: W0722 13:23:26.479456 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode60460c9_8a71_4877_a27f_05cefd5b4641.slice/crio-d156b43d44bd8331be85309c51eb22e3b9f37a7bca0f2297a64b23e4eb684e8f WatchSource:0}: Error finding container d156b43d44bd8331be85309c51eb22e3b9f37a7bca0f2297a64b23e4eb684e8f: Status 404 returned error can't find the container with id d156b43d44bd8331be85309c51eb22e3b9f37a7bca0f2297a64b23e4eb684e8f Jul 22 13:23:27.110990 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:27.110945 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" event={"ID":"e60460c9-8a71-4877-a27f-05cefd5b4641","Type":"ContainerStarted","Data":"03cdfd8b6e6771fb76f09ae48858c5c5a2b237708a96c3f0f08a85ff272aa258"} Jul 22 13:23:27.111460 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:27.111003 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" event={"ID":"e60460c9-8a71-4877-a27f-05cefd5b4641","Type":"ContainerStarted","Data":"d156b43d44bd8331be85309c51eb22e3b9f37a7bca0f2297a64b23e4eb684e8f"} Jul 22 13:23:27.134512 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:27.134447 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" podStartSLOduration=2.134427868 podStartE2EDuration="2.134427868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:23:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:23:27.132943106 +0000 UTC m=+2555.534174017" watchObservedRunningTime="2026-07-22 13:23:27.134427868 +0000 UTC m=+2555.535658789" Jul 22 13:23:36.341740 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:36.341704 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" Jul 22 13:23:36.342533 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:36.341770 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" Jul 22 13:23:36.354797 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:36.354774 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" Jul 22 13:23:37.174402 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:23:37.174374 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" Jul 22 13:24:38.341325 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:38.341286 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk"] Jul 22 13:24:53.361185 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:53.361126 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" podUID="e60460c9-8a71-4877-a27f-05cefd5b4641" containerName="main" containerID="cri-o://03cdfd8b6e6771fb76f09ae48858c5c5a2b237708a96c3f0f08a85ff272aa258" gracePeriod=45 Jul 22 13:24:53.536613 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:53.536580 2579 generic.go:356] "Generic (PLEG): container finished" podID="e60460c9-8a71-4877-a27f-05cefd5b4641" containerID="03cdfd8b6e6771fb76f09ae48858c5c5a2b237708a96c3f0f08a85ff272aa258" exitCode=0 Jul 22 13:24:53.536790 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:53.536656 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" event={"ID":"e60460c9-8a71-4877-a27f-05cefd5b4641","Type":"ContainerDied","Data":"03cdfd8b6e6771fb76f09ae48858c5c5a2b237708a96c3f0f08a85ff272aa258"} Jul 22 13:24:53.606719 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:53.606696 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" Jul 22 13:24:53.741000 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:53.740903 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2mnp\" (UniqueName: \"kubernetes.io/projected/e60460c9-8a71-4877-a27f-05cefd5b4641-kube-api-access-s2mnp\") pod \"e60460c9-8a71-4877-a27f-05cefd5b4641\" (UID: \"e60460c9-8a71-4877-a27f-05cefd5b4641\") " Jul 22 13:24:53.741000 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:53.740987 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e60460c9-8a71-4877-a27f-05cefd5b4641-tls-certs\") pod \"e60460c9-8a71-4877-a27f-05cefd5b4641\" (UID: \"e60460c9-8a71-4877-a27f-05cefd5b4641\") " Jul 22 13:24:53.741232 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:53.741018 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e60460c9-8a71-4877-a27f-05cefd5b4641-model-cache\") pod \"e60460c9-8a71-4877-a27f-05cefd5b4641\" (UID: \"e60460c9-8a71-4877-a27f-05cefd5b4641\") " Jul 22 13:24:53.741232 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:53.741034 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e60460c9-8a71-4877-a27f-05cefd5b4641-home\") pod \"e60460c9-8a71-4877-a27f-05cefd5b4641\" (UID: \"e60460c9-8a71-4877-a27f-05cefd5b4641\") " Jul 22 13:24:53.741232 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:53.741056 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e60460c9-8a71-4877-a27f-05cefd5b4641-tmp-dir\") pod \"e60460c9-8a71-4877-a27f-05cefd5b4641\" (UID: \"e60460c9-8a71-4877-a27f-05cefd5b4641\") " Jul 22 13:24:53.741398 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:53.741320 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60460c9-8a71-4877-a27f-05cefd5b4641-model-cache" (OuterVolumeSpecName: "model-cache") pod "e60460c9-8a71-4877-a27f-05cefd5b4641" (UID: "e60460c9-8a71-4877-a27f-05cefd5b4641"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:24:53.741398 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:53.741333 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60460c9-8a71-4877-a27f-05cefd5b4641-home" (OuterVolumeSpecName: "home") pod "e60460c9-8a71-4877-a27f-05cefd5b4641" (UID: "e60460c9-8a71-4877-a27f-05cefd5b4641"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:24:53.741398 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:53.741392 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60460c9-8a71-4877-a27f-05cefd5b4641-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e60460c9-8a71-4877-a27f-05cefd5b4641" (UID: "e60460c9-8a71-4877-a27f-05cefd5b4641"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:24:53.741557 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:53.741474 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e60460c9-8a71-4877-a27f-05cefd5b4641-dshm\") pod \"e60460c9-8a71-4877-a27f-05cefd5b4641\" (UID: \"e60460c9-8a71-4877-a27f-05cefd5b4641\") " Jul 22 13:24:53.741814 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:53.741794 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e60460c9-8a71-4877-a27f-05cefd5b4641-model-cache\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 13:24:53.741956 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:53.741815 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e60460c9-8a71-4877-a27f-05cefd5b4641-home\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 13:24:53.741956 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:53.741856 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e60460c9-8a71-4877-a27f-05cefd5b4641-tmp-dir\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 13:24:53.743337 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:53.743302 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e60460c9-8a71-4877-a27f-05cefd5b4641-dshm" (OuterVolumeSpecName: "dshm") pod "e60460c9-8a71-4877-a27f-05cefd5b4641" (UID: "e60460c9-8a71-4877-a27f-05cefd5b4641"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:24:53.743411 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:53.743395 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e60460c9-8a71-4877-a27f-05cefd5b4641-kube-api-access-s2mnp" (OuterVolumeSpecName: "kube-api-access-s2mnp") pod "e60460c9-8a71-4877-a27f-05cefd5b4641" (UID: "e60460c9-8a71-4877-a27f-05cefd5b4641"). InnerVolumeSpecName "kube-api-access-s2mnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:24:53.743449 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:53.743416 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e60460c9-8a71-4877-a27f-05cefd5b4641-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e60460c9-8a71-4877-a27f-05cefd5b4641" (UID: "e60460c9-8a71-4877-a27f-05cefd5b4641"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 13:24:53.842745 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:53.842699 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e60460c9-8a71-4877-a27f-05cefd5b4641-dshm\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 13:24:53.842745 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:53.842739 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2mnp\" (UniqueName: \"kubernetes.io/projected/e60460c9-8a71-4877-a27f-05cefd5b4641-kube-api-access-s2mnp\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 13:24:53.842960 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:53.842754 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e60460c9-8a71-4877-a27f-05cefd5b4641-tls-certs\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 13:24:54.543614 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:54.543585 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" Jul 22 13:24:54.544058 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:54.543607 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk" event={"ID":"e60460c9-8a71-4877-a27f-05cefd5b4641","Type":"ContainerDied","Data":"d156b43d44bd8331be85309c51eb22e3b9f37a7bca0f2297a64b23e4eb684e8f"} Jul 22 13:24:54.544058 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:54.543663 2579 scope.go:117] "RemoveContainer" containerID="03cdfd8b6e6771fb76f09ae48858c5c5a2b237708a96c3f0f08a85ff272aa258" Jul 22 13:24:54.564059 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:54.564030 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk"] Jul 22 13:24:54.566417 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:54.566396 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d24327/sub-test-model-d24327-kserve-69775b45c5-dpfrk"] Jul 22 13:24:56.067620 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:24:56.067590 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e60460c9-8a71-4877-a27f-05cefd5b4641" path="/var/lib/kubelet/pods/e60460c9-8a71-4877-a27f-05cefd5b4641/volumes" Jul 22 13:25:13.594827 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.594796 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw"] Jul 22 13:25:13.595275 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.595210 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e60460c9-8a71-4877-a27f-05cefd5b4641" containerName="main" Jul 22 13:25:13.595275 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.595225 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e60460c9-8a71-4877-a27f-05cefd5b4641" containerName="main" Jul 22 13:25:13.595346 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.595290 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e60460c9-8a71-4877-a27f-05cefd5b4641" containerName="main" Jul 22 13:25:13.598158 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.598142 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" Jul 22 13:25:13.602274 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.602245 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ffc1de\"/\"kube-root-ca.crt\"" Jul 22 13:25:13.602422 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.602289 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ffc1de\"/\"openshift-service-ca.crt\"" Jul 22 13:25:13.602422 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.602250 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ffc1de\"/\"default-dockercfg-zs8km\"" Jul 22 13:25:13.602422 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.602395 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ffc1de\"/\"rate-test-model-ffc1de-kserve-self-signed-certs\"" Jul 22 13:25:13.610300 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.610278 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw"] Jul 22 13:25:13.726891 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.726858 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd8jq\" (UniqueName: \"kubernetes.io/projected/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-kube-api-access-sd8jq\") pod \"rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw\" (UID: \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\") " pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" Jul 22 13:25:13.727104 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.726975 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-home\") pod \"rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw\" (UID: \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\") " pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" Jul 22 13:25:13.727104 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.726996 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-tmp-dir\") pod \"rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw\" (UID: \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\") " pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" Jul 22 13:25:13.727104 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.727012 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-dshm\") pod \"rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw\" (UID: \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\") " pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" Jul 22 13:25:13.727104 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.727098 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-model-cache\") pod \"rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw\" (UID: \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\") " pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" Jul 22 13:25:13.727269 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.727120 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-tls-certs\") pod \"rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw\" (UID: \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\") " pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" Jul 22 13:25:13.828462 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.828427 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-home\") pod \"rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw\" (UID: \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\") " pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" Jul 22 13:25:13.828462 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.828463 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-tmp-dir\") pod \"rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw\" (UID: \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\") " pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" Jul 22 13:25:13.828690 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.828481 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-dshm\") pod \"rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw\" (UID: \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\") " pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" Jul 22 13:25:13.828690 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.828602 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-model-cache\") pod \"rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw\" (UID: \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\") " pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" Jul 22 13:25:13.828690 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.828648 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-tls-certs\") pod \"rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw\" (UID: \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\") " pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" Jul 22 13:25:13.828690 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.828680 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd8jq\" (UniqueName: \"kubernetes.io/projected/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-kube-api-access-sd8jq\") pod \"rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw\" (UID: \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\") " pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" Jul 22 13:25:13.828965 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.828916 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-tmp-dir\") pod \"rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw\" (UID: \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\") " pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" Jul 22 13:25:13.828965 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.828957 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-home\") pod \"rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw\" (UID: \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\") " pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" Jul 22 13:25:13.829102 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.829004 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-model-cache\") pod \"rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw\" (UID: \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\") " pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" Jul 22 13:25:13.830732 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.830704 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-dshm\") pod \"rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw\" (UID: \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\") " pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" Jul 22 13:25:13.831002 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.830984 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-tls-certs\") pod \"rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw\" (UID: \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\") " pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" Jul 22 13:25:13.838401 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.838375 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd8jq\" (UniqueName: \"kubernetes.io/projected/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-kube-api-access-sd8jq\") pod \"rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw\" (UID: \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\") " pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" Jul 22 13:25:13.907326 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:13.907251 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" Jul 22 13:25:14.044047 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:14.044022 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw"] Jul 22 13:25:14.046565 ip-10-0-129-34 kubenswrapper[2579]: W0722 13:25:14.046540 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89c9900e_e0f4_4e9c_90f5_4c83c518fd6d.slice/crio-ff3303e334c9ff853cfba3a12fb521d47b2b748c0cb94a61acc321b9dbcd8468 WatchSource:0}: Error finding container ff3303e334c9ff853cfba3a12fb521d47b2b748c0cb94a61acc321b9dbcd8468: Status 404 returned error can't find the container with id ff3303e334c9ff853cfba3a12fb521d47b2b748c0cb94a61acc321b9dbcd8468 Jul 22 13:25:14.640808 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:14.640770 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" event={"ID":"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d","Type":"ContainerStarted","Data":"cce745adcde93e5e5db9d9ad23033869a634a55e05e4967dc9c165e9cee36139"} Jul 22 13:25:14.641195 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:14.640817 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" event={"ID":"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d","Type":"ContainerStarted","Data":"ff3303e334c9ff853cfba3a12fb521d47b2b748c0cb94a61acc321b9dbcd8468"} Jul 22 13:25:14.661788 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:14.661731 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" podStartSLOduration=1.661715364 podStartE2EDuration="1.661715364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-22 13:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:25:14.660788555 +0000 UTC m=+2663.062019475" watchObservedRunningTime="2026-07-22 13:25:14.661715364 +0000 UTC m=+2663.062946283" Jul 22 13:25:23.908463 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:23.908424 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" Jul 22 13:25:23.908962 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:23.908478 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" Jul 22 13:25:23.920940 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:23.920902 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" Jul 22 13:25:24.699577 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:24.699547 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" Jul 22 13:25:52.183735 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:52.183626 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ddj6_b28e0ca4-8512-45da-91ed-2cde1491b2e2/ovn-acl-logging/0.log" Jul 22 13:25:52.189246 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:52.189228 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ddj6_b28e0ca4-8512-45da-91ed-2cde1491b2e2/ovn-acl-logging/0.log" Jul 22 13:25:54.992005 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:25:54.991656 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw"] Jul 22 13:26:10.014003 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.013908 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" podUID="89c9900e-e0f4-4e9c-90f5-4c83c518fd6d" containerName="main" containerID="cri-o://cce745adcde93e5e5db9d9ad23033869a634a55e05e4967dc9c165e9cee36139" gracePeriod=45 Jul 22 13:26:10.266211 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.266154 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" Jul 22 13:26:10.338328 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.338290 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-tmp-dir\") pod \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\" (UID: \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\") " Jul 22 13:26:10.338328 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.338335 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-tls-certs\") pod \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\" (UID: \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\") " Jul 22 13:26:10.338583 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.338361 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-model-cache\") pod \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\" (UID: \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\") " Jul 22 13:26:10.338583 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.338387 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-home\") pod \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\" (UID: \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\") " Jul 22 13:26:10.338583 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.338450 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sd8jq\" (UniqueName: \"kubernetes.io/projected/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-kube-api-access-sd8jq\") pod \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\" (UID: \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\") " Jul 22 13:26:10.338583 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.338523 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-dshm\") pod \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\" (UID: \"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d\") " Jul 22 13:26:10.338818 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.338609 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "89c9900e-e0f4-4e9c-90f5-4c83c518fd6d" (UID: "89c9900e-e0f4-4e9c-90f5-4c83c518fd6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:26:10.338818 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.338624 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-model-cache" (OuterVolumeSpecName: "model-cache") pod "89c9900e-e0f4-4e9c-90f5-4c83c518fd6d" (UID: "89c9900e-e0f4-4e9c-90f5-4c83c518fd6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:26:10.338818 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.338776 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-home" (OuterVolumeSpecName: "home") pod "89c9900e-e0f4-4e9c-90f5-4c83c518fd6d" (UID: "89c9900e-e0f4-4e9c-90f5-4c83c518fd6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:26:10.338991 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.338903 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-tmp-dir\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 13:26:10.338991 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.338947 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-model-cache\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 13:26:10.338991 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.338962 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-home\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 13:26:10.340623 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.340600 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-dshm" (OuterVolumeSpecName: "dshm") pod "89c9900e-e0f4-4e9c-90f5-4c83c518fd6d" (UID: "89c9900e-e0f4-4e9c-90f5-4c83c518fd6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 22 13:26:10.340623 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.340602 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "89c9900e-e0f4-4e9c-90f5-4c83c518fd6d" (UID: "89c9900e-e0f4-4e9c-90f5-4c83c518fd6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 22 13:26:10.340913 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.340894 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-kube-api-access-sd8jq" (OuterVolumeSpecName: "kube-api-access-sd8jq") pod "89c9900e-e0f4-4e9c-90f5-4c83c518fd6d" (UID: "89c9900e-e0f4-4e9c-90f5-4c83c518fd6d"). InnerVolumeSpecName "kube-api-access-sd8jq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 22 13:26:10.440421 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.440373 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-tls-certs\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 13:26:10.440421 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.440416 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd8jq\" (UniqueName: \"kubernetes.io/projected/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-kube-api-access-sd8jq\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 13:26:10.440421 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.440426 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d-dshm\") on node \"ip-10-0-129-34.ec2.internal\" DevicePath \"\"" Jul 22 13:26:10.917999 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.917968 2579 generic.go:356] "Generic (PLEG): container finished" podID="89c9900e-e0f4-4e9c-90f5-4c83c518fd6d" containerID="cce745adcde93e5e5db9d9ad23033869a634a55e05e4967dc9c165e9cee36139" exitCode=0 Jul 22 13:26:10.918205 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.918019 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" Jul 22 13:26:10.918205 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.918032 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" event={"ID":"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d","Type":"ContainerDied","Data":"cce745adcde93e5e5db9d9ad23033869a634a55e05e4967dc9c165e9cee36139"} Jul 22 13:26:10.918205 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.918064 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw" event={"ID":"89c9900e-e0f4-4e9c-90f5-4c83c518fd6d","Type":"ContainerDied","Data":"ff3303e334c9ff853cfba3a12fb521d47b2b748c0cb94a61acc321b9dbcd8468"} Jul 22 13:26:10.918205 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.918081 2579 scope.go:117] "RemoveContainer" containerID="cce745adcde93e5e5db9d9ad23033869a634a55e05e4967dc9c165e9cee36139" Jul 22 13:26:10.926943 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.926901 2579 scope.go:117] "RemoveContainer" containerID="cce745adcde93e5e5db9d9ad23033869a634a55e05e4967dc9c165e9cee36139" Jul 22 13:26:10.927194 ip-10-0-129-34 kubenswrapper[2579]: E0722 13:26:10.927176 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce745adcde93e5e5db9d9ad23033869a634a55e05e4967dc9c165e9cee36139\": container with ID starting with cce745adcde93e5e5db9d9ad23033869a634a55e05e4967dc9c165e9cee36139 not found: ID does not exist" containerID="cce745adcde93e5e5db9d9ad23033869a634a55e05e4967dc9c165e9cee36139" Jul 22 13:26:10.927240 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.927200 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce745adcde93e5e5db9d9ad23033869a634a55e05e4967dc9c165e9cee36139"} err="failed to get container status \"cce745adcde93e5e5db9d9ad23033869a634a55e05e4967dc9c165e9cee36139\": rpc error: code = NotFound desc = could not find container \"cce745adcde93e5e5db9d9ad23033869a634a55e05e4967dc9c165e9cee36139\": container with ID starting with cce745adcde93e5e5db9d9ad23033869a634a55e05e4967dc9c165e9cee36139 not found: ID does not exist" Jul 22 13:26:10.939908 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.939881 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw"] Jul 22 13:26:10.945342 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:10.945317 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ffc1de/rate-test-model-ffc1de-kserve-5ccc48cc8c-lw9cw"] Jul 22 13:26:12.067190 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:26:12.067156 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89c9900e-e0f4-4e9c-90f5-4c83c518fd6d" path="/var/lib/kubelet/pods/89c9900e-e0f4-4e9c-90f5-4c83c518fd6d/volumes" Jul 22 13:27:06.103140 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:06.103109 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64bbff868f-rt2tl_e9c73605-9677-46a0-a88e-25e22d8428b6/copy-manifests/0.log" Jul 22 13:27:06.110366 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:06.110344 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64bbff868f-rt2tl_e9c73605-9677-46a0-a88e-25e22d8428b6/manager/0.log" Jul 22 13:27:06.711721 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:06.711690 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29745420-skfnw_b69b0193-ea31-438c-a92f-c1b866c1ed68/cleanup/0.log" Jul 22 13:27:06.941446 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:06.941412 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-gvxdk_88665a68-3afd-450d-8321-834747980c41/postgres/0.log" Jul 22 13:27:08.434593 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:08.434563 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-t2zg5_b07a2a52-8017-414c-a7bc-2c1dd6c7cf9b/manager/0.log" Jul 22 13:27:08.561547 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:08.561515 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-stb8r_5c4a4f48-2186-459f-aa7c-81054b32cc31/manager/0.log" Jul 22 13:27:08.801892 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:08.801861 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-tpbmj_50f01f4e-5b13-4a21-bf0e-6c2631057dfd/registry-server/0.log" Jul 22 13:27:08.935717 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:08.935686 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-76ks9_2cb8077a-631e-4ada-b4eb-2be7ce4afc27/manager/0.log" Jul 22 13:27:09.219627 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:09.219548 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-x8bq9_69a15b20-8ebf-4716-8c1b-9d577fa96224/manager/0.log" Jul 22 13:27:10.981913 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:10.981880 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-47494_8598cee8-c42f-4502-8a1f-5b7f20e39400/main/0.log" Jul 22 13:27:11.227159 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:11.227126 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f5std8_c4b45501-1153-49f4-a9a7-753f617969ef/main/0.log" Jul 22 13:27:11.472547 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:11.472515 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-4xkr5_bbd6c631-6153-4e93-94f4-b3f1e756ce5e/main/0.log" Jul 22 13:27:19.735896 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:19.735817 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7gsw5_28d7c937-a848-4d26-861f-29f9d2b0baf7/global-pull-secret-syncer/0.log" Jul 22 13:27:19.900093 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:19.900064 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ns7kz_3dc0c773-526a-4eee-8155-77cf64929958/konnectivity-agent/0.log" Jul 22 13:27:19.920511 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:19.920487 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-34.ec2.internal_ea81ad4ab5ce353c7d7f37c0cd1009b2/haproxy/0.log" Jul 22 13:27:25.539496 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:25.539469 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-t2zg5_b07a2a52-8017-414c-a7bc-2c1dd6c7cf9b/manager/0.log" Jul 22 13:27:25.564890 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:25.564863 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-stb8r_5c4a4f48-2186-459f-aa7c-81054b32cc31/manager/0.log" Jul 22 13:27:25.634542 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:25.634514 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-tpbmj_50f01f4e-5b13-4a21-bf0e-6c2631057dfd/registry-server/0.log" Jul 22 13:27:25.807062 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:25.806989 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-76ks9_2cb8077a-631e-4ada-b4eb-2be7ce4afc27/manager/0.log" Jul 22 13:27:25.920685 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:25.920658 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-x8bq9_69a15b20-8ebf-4716-8c1b-9d577fa96224/manager/0.log" Jul 22 13:27:27.488342 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:27.488307 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_21aa6be9-3e79-42eb-a1df-71a1e072b534/alertmanager/0.log" Jul 22 13:27:27.512654 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:27.512628 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_21aa6be9-3e79-42eb-a1df-71a1e072b534/config-reloader/0.log" Jul 22 13:27:27.538149 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:27.538123 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_21aa6be9-3e79-42eb-a1df-71a1e072b534/kube-rbac-proxy-web/0.log" Jul 22 13:27:27.560811 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:27.560789 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_21aa6be9-3e79-42eb-a1df-71a1e072b534/kube-rbac-proxy/0.log" Jul 22 13:27:27.583825 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:27.583802 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_21aa6be9-3e79-42eb-a1df-71a1e072b534/kube-rbac-proxy-metric/0.log" Jul 22 13:27:27.607812 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:27.607792 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_21aa6be9-3e79-42eb-a1df-71a1e072b534/prom-label-proxy/0.log" Jul 22 13:27:27.632189 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:27.632164 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_21aa6be9-3e79-42eb-a1df-71a1e072b534/init-config-reloader/0.log" Jul 22 13:27:27.695470 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:27.695440 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-75nsf_42f58d4d-89d6-46e4-8598-0a428e9ed439/kube-state-metrics/0.log" Jul 22 13:27:27.716427 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:27.716403 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-75nsf_42f58d4d-89d6-46e4-8598-0a428e9ed439/kube-rbac-proxy-main/0.log" Jul 22 13:27:27.740755 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:27.740682 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-75nsf_42f58d4d-89d6-46e4-8598-0a428e9ed439/kube-rbac-proxy-self/0.log" Jul 22 13:27:27.766610 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:27.766586 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-64766cc68b-h7chl_50a8d9a2-5d26-4f2b-8996-d1bb986e41b7/metrics-server/0.log" Jul 22 13:27:27.793172 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:27.793151 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-p94kz_dc5706d0-452b-4568-8460-30c35440cf14/monitoring-plugin/0.log" Jul 22 13:27:27.831438 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:27.831390 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-48jh9_c98bde4d-82db-4b9d-969a-892fc9ef54f2/node-exporter/0.log" Jul 22 13:27:27.854362 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:27.854343 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-48jh9_c98bde4d-82db-4b9d-969a-892fc9ef54f2/kube-rbac-proxy/0.log" Jul 22 13:27:27.875350 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:27.875331 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-48jh9_c98bde4d-82db-4b9d-969a-892fc9ef54f2/init-textfile/0.log" Jul 22 13:27:28.067493 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:28.067465 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-jnzn4_3cf037ef-07c5-4294-933e-6c61ef336250/kube-rbac-proxy-main/0.log" Jul 22 13:27:28.093374 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:28.093353 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-jnzn4_3cf037ef-07c5-4294-933e-6c61ef336250/kube-rbac-proxy-self/0.log" Jul 22 13:27:28.114688 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:28.114664 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-jnzn4_3cf037ef-07c5-4294-933e-6c61ef336250/openshift-state-metrics/0.log" Jul 22 13:27:28.334711 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:28.334630 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-llw8h_379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d/prometheus-operator/0.log" Jul 22 13:27:28.354754 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:28.354729 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-llw8h_379c5fcb-5fc6-44d5-88b1-d0ab4f6e1f6d/kube-rbac-proxy/0.log" Jul 22 13:27:28.389525 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:28.389501 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-dptcc_1713bba6-d614-4d6b-89be-4880d38a9f29/prometheus-operator-admission-webhook/0.log" Jul 22 13:27:28.500868 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:28.500816 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6979d99b4-hwtbm_b0204876-d2f4-486c-95cb-7a6f0726e566/thanos-query/0.log" Jul 22 13:27:28.524202 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:28.524174 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6979d99b4-hwtbm_b0204876-d2f4-486c-95cb-7a6f0726e566/kube-rbac-proxy-web/0.log" Jul 22 13:27:28.548440 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:28.548412 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6979d99b4-hwtbm_b0204876-d2f4-486c-95cb-7a6f0726e566/kube-rbac-proxy/0.log" Jul 22 13:27:28.570955 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:28.570915 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6979d99b4-hwtbm_b0204876-d2f4-486c-95cb-7a6f0726e566/prom-label-proxy/0.log" Jul 22 13:27:28.594217 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:28.594144 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6979d99b4-hwtbm_b0204876-d2f4-486c-95cb-7a6f0726e566/kube-rbac-proxy-rules/0.log" Jul 22 13:27:28.621414 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:28.621395 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6979d99b4-hwtbm_b0204876-d2f4-486c-95cb-7a6f0726e566/kube-rbac-proxy-metrics/0.log" Jul 22 13:27:29.510964 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:29.510919 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g2jzv/perf-node-gather-daemonset-q2k5n"] Jul 22 13:27:29.511330 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:29.511318 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89c9900e-e0f4-4e9c-90f5-4c83c518fd6d" containerName="main" Jul 22 13:27:29.511330 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:29.511331 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="89c9900e-e0f4-4e9c-90f5-4c83c518fd6d" containerName="main" Jul 22 13:27:29.511403 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:29.511396 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="89c9900e-e0f4-4e9c-90f5-4c83c518fd6d" containerName="main" Jul 22 13:27:29.514823 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:29.514801 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-q2k5n" Jul 22 13:27:29.517614 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:29.517595 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g2jzv\"/\"default-dockercfg-5xtwz\"" Jul 22 13:27:29.517716 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:29.517665 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g2jzv\"/\"kube-root-ca.crt\"" Jul 22 13:27:29.518787 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:29.518771 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g2jzv\"/\"openshift-service-ca.crt\"" Jul 22 13:27:29.525747 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:29.525726 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g2jzv/perf-node-gather-daemonset-q2k5n"] Jul 22 13:27:29.664374 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:29.664341 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7297a636-91ad-43c8-9f98-583eae0b616e-sys\") pod \"perf-node-gather-daemonset-q2k5n\" (UID: \"7297a636-91ad-43c8-9f98-583eae0b616e\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-q2k5n" Jul 22 13:27:29.664553 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:29.664418 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klspc\" (UniqueName: \"kubernetes.io/projected/7297a636-91ad-43c8-9f98-583eae0b616e-kube-api-access-klspc\") pod \"perf-node-gather-daemonset-q2k5n\" (UID: \"7297a636-91ad-43c8-9f98-583eae0b616e\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-q2k5n" Jul 22 13:27:29.664553 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:29.664495 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7297a636-91ad-43c8-9f98-583eae0b616e-proc\") pod \"perf-node-gather-daemonset-q2k5n\" (UID: \"7297a636-91ad-43c8-9f98-583eae0b616e\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-q2k5n" Jul 22 13:27:29.664553 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:29.664518 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7297a636-91ad-43c8-9f98-583eae0b616e-podres\") pod \"perf-node-gather-daemonset-q2k5n\" (UID: \"7297a636-91ad-43c8-9f98-583eae0b616e\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-q2k5n" Jul 22 13:27:29.664553 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:29.664542 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7297a636-91ad-43c8-9f98-583eae0b616e-lib-modules\") pod \"perf-node-gather-daemonset-q2k5n\" (UID: \"7297a636-91ad-43c8-9f98-583eae0b616e\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-q2k5n" Jul 22 13:27:29.765250 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:29.765165 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7297a636-91ad-43c8-9f98-583eae0b616e-proc\") pod \"perf-node-gather-daemonset-q2k5n\" (UID: \"7297a636-91ad-43c8-9f98-583eae0b616e\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-q2k5n" Jul 22 13:27:29.765250 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:29.765201 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7297a636-91ad-43c8-9f98-583eae0b616e-podres\") pod \"perf-node-gather-daemonset-q2k5n\" (UID: \"7297a636-91ad-43c8-9f98-583eae0b616e\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-q2k5n" Jul 22 13:27:29.765250 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:29.765219 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7297a636-91ad-43c8-9f98-583eae0b616e-lib-modules\") pod \"perf-node-gather-daemonset-q2k5n\" (UID: \"7297a636-91ad-43c8-9f98-583eae0b616e\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-q2k5n" Jul 22 13:27:29.765250 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:29.765250 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7297a636-91ad-43c8-9f98-583eae0b616e-sys\") pod \"perf-node-gather-daemonset-q2k5n\" (UID: \"7297a636-91ad-43c8-9f98-583eae0b616e\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-q2k5n" Jul 22 13:27:29.765513 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:29.765300 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7297a636-91ad-43c8-9f98-583eae0b616e-proc\") pod \"perf-node-gather-daemonset-q2k5n\" (UID: \"7297a636-91ad-43c8-9f98-583eae0b616e\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-q2k5n" Jul 22 13:27:29.765513 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:29.765334 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klspc\" (UniqueName: \"kubernetes.io/projected/7297a636-91ad-43c8-9f98-583eae0b616e-kube-api-access-klspc\") pod \"perf-node-gather-daemonset-q2k5n\" (UID: \"7297a636-91ad-43c8-9f98-583eae0b616e\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-q2k5n" Jul 22 13:27:29.765513 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:29.765347 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7297a636-91ad-43c8-9f98-583eae0b616e-podres\") pod \"perf-node-gather-daemonset-q2k5n\" (UID: \"7297a636-91ad-43c8-9f98-583eae0b616e\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-q2k5n" Jul 22 13:27:29.765513 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:29.765364 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7297a636-91ad-43c8-9f98-583eae0b616e-sys\") pod \"perf-node-gather-daemonset-q2k5n\" (UID: \"7297a636-91ad-43c8-9f98-583eae0b616e\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-q2k5n" Jul 22 13:27:29.765513 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:29.765423 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7297a636-91ad-43c8-9f98-583eae0b616e-lib-modules\") pod \"perf-node-gather-daemonset-q2k5n\" (UID: \"7297a636-91ad-43c8-9f98-583eae0b616e\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-q2k5n" Jul 22 13:27:29.774306 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:29.774275 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klspc\" (UniqueName: \"kubernetes.io/projected/7297a636-91ad-43c8-9f98-583eae0b616e-kube-api-access-klspc\") pod \"perf-node-gather-daemonset-q2k5n\" (UID: \"7297a636-91ad-43c8-9f98-583eae0b616e\") " pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-q2k5n" Jul 22 13:27:29.824939 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:29.824893 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-q2k5n" Jul 22 13:27:29.957646 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:29.957620 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g2jzv/perf-node-gather-daemonset-q2k5n"] Jul 22 13:27:29.960093 ip-10-0-129-34 kubenswrapper[2579]: W0722 13:27:29.960067 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7297a636_91ad_43c8_9f98_583eae0b616e.slice/crio-bab1f24a4b3cf3ed0f3f2bfb29fded8ffd99c8e3ef7df00ae548be2ac617c9fe WatchSource:0}: Error finding container bab1f24a4b3cf3ed0f3f2bfb29fded8ffd99c8e3ef7df00ae548be2ac617c9fe: Status 404 returned error can't find the container with id bab1f24a4b3cf3ed0f3f2bfb29fded8ffd99c8e3ef7df00ae548be2ac617c9fe Jul 22 13:27:30.309670 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:30.309582 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-q2k5n" event={"ID":"7297a636-91ad-43c8-9f98-583eae0b616e","Type":"ContainerStarted","Data":"bac8eb7a12e5b5d363c556fdfe1b8ba45bd8783b010c6df6a80e8858a550307d"} Jul 22 13:27:30.309670 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:30.309638 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-q2k5n" event={"ID":"7297a636-91ad-43c8-9f98-583eae0b616e","Type":"ContainerStarted","Data":"bab1f24a4b3cf3ed0f3f2bfb29fded8ffd99c8e3ef7df00ae548be2ac617c9fe"} Jul 22 13:27:30.309893 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:30.309703 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-q2k5n" Jul 22 13:27:30.327487 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:30.327440 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-q2k5n" podStartSLOduration=1.327424658 podStartE2EDuration="1.327424658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-22 13:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-22 13:27:30.326708424 +0000 UTC m=+2798.727939343" watchObservedRunningTime="2026-07-22 13:27:30.327424658 +0000 UTC m=+2798.728655610" Jul 22 13:27:30.992390 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:30.992362 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-565f69cdff-6z46k_e15933ab-6768-4e85-b079-d1860634462a/console/0.log" Jul 22 13:27:32.750519 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:32.750489 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w8md4_394708f4-ca0a-4fb2-a3d2-a22224322e3d/dns/0.log" Jul 22 13:27:32.778354 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:32.778326 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-w8md4_394708f4-ca0a-4fb2-a3d2-a22224322e3d/kube-rbac-proxy/0.log" Jul 22 13:27:32.802594 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:32.802569 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fgc67_8b80aabc-b7ef-4649-bb58-4e0a39906c4f/dns-node-resolver/0.log" Jul 22 13:27:33.502066 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:33.502039 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w6nhq_c504a9e7-e48d-496d-8a1d-3b307361714c/node-ca/0.log" Jul 22 13:27:35.491679 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:35.491655 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7q7fk_e9effaf4-801b-4f94-856f-9a934fd06a77/serve-healthcheck-canary/0.log" Jul 22 13:27:36.091813 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:36.091785 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dkt6h_cfcc578f-7c22-4248-a8d0-7695852ec582/kube-rbac-proxy/0.log" Jul 22 13:27:36.112358 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:36.112330 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dkt6h_cfcc578f-7c22-4248-a8d0-7695852ec582/exporter/0.log" Jul 22 13:27:36.132382 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:36.132359 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dkt6h_cfcc578f-7c22-4248-a8d0-7695852ec582/extractor/0.log" Jul 22 13:27:36.324772 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:36.324742 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-g2jzv/perf-node-gather-daemonset-q2k5n" Jul 22 13:27:39.229605 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:39.229574 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64bbff868f-rt2tl_e9c73605-9677-46a0-a88e-25e22d8428b6/manager/0.log" Jul 22 13:27:39.249939 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:39.249906 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64bbff868f-rt2tl_e9c73605-9677-46a0-a88e-25e22d8428b6/copy-manifests/0.log" Jul 22 13:27:40.668637 ip-10-0-129-34 kubenswrapper[2579]: I0722 13:27:40.668607 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8b6495d78-grmzq_e04e2273-7cc5-46b0-9bb5-0506c6c5af46/manager/0.log"