Jul 13 21:59:38.457296 ip-10-0-132-128 systemd[1]: Starting Kubernetes Kubelet... Jul 13 21:59:38.761840 ip-10-0-132-128 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 21:59:38.761840 ip-10-0-132-128 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 13 21:59:38.761840 ip-10-0-132-128 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 21:59:38.761840 ip-10-0-132-128 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 13 21:59:38.761840 ip-10-0-132-128 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 13 21:59:38.763166 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.763081 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 13 21:59:38.767640 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767623 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 21:59:38.767640 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767639 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 21:59:38.767640 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767643 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 21:59:38.767739 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767646 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 21:59:38.767739 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767649 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 21:59:38.767739 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767652 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 21:59:38.767739 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767654 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 21:59:38.767739 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767657 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 21:59:38.767739 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767661 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 21:59:38.767739 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767664 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 21:59:38.767739 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767666 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 21:59:38.767739 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767669 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 21:59:38.767739 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767672 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 21:59:38.767739 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767674 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 21:59:38.767739 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767677 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 21:59:38.767739 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767680 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 21:59:38.767739 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767683 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 21:59:38.767739 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767685 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 21:59:38.767739 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767688 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 21:59:38.767739 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767691 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 21:59:38.767739 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767693 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 21:59:38.767739 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767696 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 21:59:38.768187 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767698 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 21:59:38.768187 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767702 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 21:59:38.768187 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767704 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 21:59:38.768187 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767706 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 21:59:38.768187 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767709 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 21:59:38.768187 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767714 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 21:59:38.768187 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767719 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 21:59:38.768187 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767722 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 21:59:38.768187 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767725 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 21:59:38.768187 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767728 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 21:59:38.768187 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767732 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 21:59:38.768187 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767737 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 21:59:38.768187 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767740 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 21:59:38.768187 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767743 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 21:59:38.768187 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767746 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 21:59:38.768187 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767749 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 21:59:38.768187 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767753 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 21:59:38.768187 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767757 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 21:59:38.768187 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767760 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 21:59:38.768645 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767764 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 21:59:38.768645 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767767 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 21:59:38.768645 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767770 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 21:59:38.768645 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767772 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 21:59:38.768645 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767774 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 21:59:38.768645 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767777 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 21:59:38.768645 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767779 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 21:59:38.768645 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767782 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 13 21:59:38.768645 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767784 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 21:59:38.768645 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767787 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 21:59:38.768645 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767789 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 21:59:38.768645 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767792 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 21:59:38.768645 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767794 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 21:59:38.768645 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767797 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 21:59:38.768645 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767800 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 21:59:38.768645 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767802 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 21:59:38.768645 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767805 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 21:59:38.768645 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767808 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 21:59:38.768645 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767811 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 21:59:38.768645 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767814 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 21:59:38.769160 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767817 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 21:59:38.769160 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767820 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 21:59:38.769160 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767822 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 21:59:38.769160 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767825 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 21:59:38.769160 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767828 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 21:59:38.769160 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767831 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 21:59:38.769160 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767834 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 21:59:38.769160 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767836 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 21:59:38.769160 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767839 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 21:59:38.769160 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767841 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 21:59:38.769160 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767844 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 21:59:38.769160 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767846 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 21:59:38.769160 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767849 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 21:59:38.769160 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767852 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 21:59:38.769160 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767856 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 21:59:38.769160 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767858 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 21:59:38.769160 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767861 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 21:59:38.769160 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767864 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 21:59:38.769160 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767867 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 21:59:38.769160 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767870 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 21:59:38.769616 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767873 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 21:59:38.769616 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767875 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 21:59:38.769616 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767878 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 21:59:38.769616 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767881 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 21:59:38.769616 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767884 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 21:59:38.769616 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767886 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 21:59:38.769616 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767889 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 21:59:38.769616 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767891 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 21:59:38.769616 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767894 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 21:59:38.769616 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767896 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 21:59:38.769616 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767899 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 21:59:38.769616 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767901 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 21:59:38.769616 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767904 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 21:59:38.769616 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767906 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 21:59:38.769616 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767909 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 21:59:38.769616 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767911 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 21:59:38.769616 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767914 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 21:59:38.769616 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.767916 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 21:59:38.769616 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768620 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 21:59:38.769616 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768626 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 21:59:38.770096 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768630 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 21:59:38.770096 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768633 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 21:59:38.770096 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768635 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 21:59:38.770096 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768638 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 21:59:38.770096 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768640 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 21:59:38.770096 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768643 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 21:59:38.770096 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768646 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 21:59:38.770096 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768649 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 21:59:38.770096 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768652 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 21:59:38.770096 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768656 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 21:59:38.770096 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768661 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 21:59:38.770096 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768664 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 21:59:38.770096 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768667 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 21:59:38.770096 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768672 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 21:59:38.770096 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768675 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 21:59:38.770096 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768679 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 21:59:38.770096 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768682 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 21:59:38.770096 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768685 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 21:59:38.770096 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768688 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 21:59:38.770096 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768691 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 21:59:38.770612 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768694 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 21:59:38.770612 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768697 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 21:59:38.770612 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768700 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 21:59:38.770612 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768703 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 21:59:38.770612 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768705 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 21:59:38.770612 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768708 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 21:59:38.770612 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768710 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 21:59:38.770612 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768713 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 21:59:38.770612 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768715 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 21:59:38.770612 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768718 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 21:59:38.770612 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768721 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 21:59:38.770612 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768724 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 21:59:38.770612 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768727 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 21:59:38.770612 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768729 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 21:59:38.770612 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768732 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 21:59:38.770612 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768734 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 21:59:38.770612 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768737 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 21:59:38.770612 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768739 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 21:59:38.770612 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768742 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 21:59:38.771121 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768744 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 21:59:38.771121 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768748 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 21:59:38.771121 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768750 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 21:59:38.771121 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768753 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 21:59:38.771121 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768755 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 21:59:38.771121 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768757 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 21:59:38.771121 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768761 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 21:59:38.771121 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768764 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 21:59:38.771121 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768768 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 21:59:38.771121 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768771 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 21:59:38.771121 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768774 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 21:59:38.771121 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768777 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 21:59:38.771121 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768780 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 21:59:38.771121 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768782 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 21:59:38.771121 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768785 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 21:59:38.771121 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768800 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 21:59:38.771121 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768804 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 21:59:38.771121 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768806 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 21:59:38.771121 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768810 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 21:59:38.771121 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768812 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 21:59:38.771584 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768815 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 21:59:38.771584 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768817 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 21:59:38.771584 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768820 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 13 21:59:38.771584 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768823 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 21:59:38.771584 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768825 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 21:59:38.771584 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768828 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 21:59:38.771584 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768831 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 21:59:38.771584 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768833 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 21:59:38.771584 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768836 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 21:59:38.771584 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768838 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 21:59:38.771584 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768841 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 21:59:38.771584 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768843 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 21:59:38.771584 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768846 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 21:59:38.771584 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768849 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 21:59:38.771584 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768854 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 21:59:38.771584 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768856 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 21:59:38.771584 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768859 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 21:59:38.771584 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768861 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 21:59:38.771584 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768864 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 21:59:38.771584 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768867 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 21:59:38.772067 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768869 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 21:59:38.772067 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768872 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 21:59:38.772067 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768875 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 21:59:38.772067 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768877 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 21:59:38.772067 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768880 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 21:59:38.772067 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768882 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 21:59:38.772067 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768884 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 21:59:38.772067 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768887 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 21:59:38.772067 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768889 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 21:59:38.772067 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768892 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 21:59:38.772067 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768894 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 21:59:38.772067 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768896 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 21:59:38.772067 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768899 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 21:59:38.772067 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768902 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 21:59:38.772067 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768904 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 21:59:38.772067 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768909 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 21:59:38.772067 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768912 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 21:59:38.772067 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.768914 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 21:59:38.772067 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769306 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 21:59:38.772520 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769311 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 21:59:38.772520 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769314 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 21:59:38.772520 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769317 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 21:59:38.772520 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769320 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 21:59:38.772520 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769323 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 21:59:38.772520 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769326 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 21:59:38.772520 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769328 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 21:59:38.772520 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769332 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 21:59:38.772520 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769335 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 21:59:38.772520 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769337 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 21:59:38.772520 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769340 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 21:59:38.772520 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769342 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 21:59:38.772520 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769345 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 21:59:38.772520 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769347 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 21:59:38.772520 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769350 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 21:59:38.772520 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769353 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 13 21:59:38.772520 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769355 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 21:59:38.772520 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769358 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 21:59:38.772520 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769360 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 21:59:38.772520 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769365 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 21:59:38.773011 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769368 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 21:59:38.773011 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769371 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 21:59:38.773011 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769373 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 21:59:38.773011 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769376 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 21:59:38.773011 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769378 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 21:59:38.773011 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769381 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 21:59:38.773011 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769383 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 21:59:38.773011 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769386 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 21:59:38.773011 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769390 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 21:59:38.773011 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769393 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 21:59:38.773011 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769396 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 21:59:38.773011 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769398 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 21:59:38.773011 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769401 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 21:59:38.773011 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769403 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 21:59:38.773011 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769406 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 21:59:38.773011 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769408 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 21:59:38.773011 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769411 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 21:59:38.773011 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769414 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 21:59:38.773011 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769417 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 21:59:38.773011 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769419 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 21:59:38.773483 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769422 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 21:59:38.773483 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769425 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 21:59:38.773483 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769427 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 21:59:38.773483 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769430 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 21:59:38.773483 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769432 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 21:59:38.773483 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769435 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 21:59:38.773483 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769437 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 21:59:38.773483 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769440 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 21:59:38.773483 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769442 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 21:59:38.773483 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769445 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 21:59:38.773483 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769447 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 21:59:38.773483 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769450 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 21:59:38.773483 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769452 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 21:59:38.773483 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769456 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 21:59:38.773483 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769459 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 21:59:38.773483 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769462 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 21:59:38.773483 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769465 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 21:59:38.773483 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769468 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 21:59:38.773483 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769471 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 21:59:38.773935 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769474 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 21:59:38.773935 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769478 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 21:59:38.773935 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769481 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 21:59:38.773935 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769484 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 21:59:38.773935 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769487 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 21:59:38.773935 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769490 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 21:59:38.773935 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769493 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 21:59:38.773935 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769495 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 21:59:38.773935 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769498 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 21:59:38.773935 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769501 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 21:59:38.773935 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769503 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 21:59:38.773935 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769506 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 21:59:38.773935 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769508 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 21:59:38.773935 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769511 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 21:59:38.773935 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769514 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 21:59:38.773935 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769517 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 21:59:38.773935 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769519 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 21:59:38.773935 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769522 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 21:59:38.773935 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769524 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 21:59:38.774398 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769526 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 21:59:38.774398 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769529 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 21:59:38.774398 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769532 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 21:59:38.774398 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769534 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 21:59:38.774398 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769536 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 21:59:38.774398 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769540 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 21:59:38.774398 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769542 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 21:59:38.774398 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769546 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 21:59:38.774398 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769549 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 21:59:38.774398 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769551 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 21:59:38.774398 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769553 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 21:59:38.774398 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769556 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 21:59:38.774398 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769559 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 21:59:38.774398 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769561 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 21:59:38.774398 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769566 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 21:59:38.774398 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769568 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 21:59:38.774398 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769571 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 21:59:38.774398 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769574 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 21:59:38.774398 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769576 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 21:59:38.774398 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.769578 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 21:59:38.774871 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769654 2562 flags.go:64] FLAG: --address="0.0.0.0" Jul 13 21:59:38.774871 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769660 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 13 21:59:38.774871 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769667 2562 flags.go:64] FLAG: --anonymous-auth="true" Jul 13 21:59:38.774871 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769671 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 13 21:59:38.774871 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769675 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 13 21:59:38.774871 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769678 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 13 21:59:38.774871 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769682 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 13 21:59:38.774871 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769686 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 13 21:59:38.774871 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769691 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 13 21:59:38.774871 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769694 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 13 21:59:38.774871 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769698 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 13 21:59:38.774871 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769701 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 13 21:59:38.774871 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769704 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 13 21:59:38.774871 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769708 2562 flags.go:64] FLAG: --cgroup-root="" Jul 13 21:59:38.774871 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769711 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 13 21:59:38.774871 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769714 2562 flags.go:64] FLAG: --client-ca-file="" Jul 13 21:59:38.774871 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769717 2562 flags.go:64] FLAG: --cloud-provider="external" Jul 13 21:59:38.774871 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769720 2562 flags.go:64] FLAG: --cluster-dns="[]" Jul 13 21:59:38.774871 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769725 2562 flags.go:64] FLAG: --cluster-domain="" Jul 13 21:59:38.774871 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769728 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 13 21:59:38.774871 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769731 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 13 21:59:38.774871 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769735 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 13 21:59:38.774871 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769738 2562 flags.go:64] FLAG: --container-log-max-files="5" Jul 13 21:59:38.774871 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769742 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 13 21:59:38.775453 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769745 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 13 21:59:38.775453 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769748 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 13 21:59:38.775453 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769752 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 13 21:59:38.775453 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769755 2562 flags.go:64] FLAG: --contention-profiling="false" Jul 13 21:59:38.775453 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769758 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 13 21:59:38.775453 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769761 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 13 21:59:38.775453 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769764 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 13 21:59:38.775453 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769767 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 13 21:59:38.775453 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769771 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 13 21:59:38.775453 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769775 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 13 21:59:38.775453 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769777 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 13 21:59:38.775453 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769780 2562 flags.go:64] FLAG: --enable-load-reader="false" Jul 13 21:59:38.775453 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769783 2562 flags.go:64] FLAG: --enable-server="true" Jul 13 21:59:38.775453 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769786 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 13 21:59:38.775453 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769790 2562 flags.go:64] FLAG: --event-burst="100" Jul 13 21:59:38.775453 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769794 2562 flags.go:64] FLAG: --event-qps="50" Jul 13 21:59:38.775453 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769797 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 13 21:59:38.775453 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769800 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 13 21:59:38.775453 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769803 2562 flags.go:64] FLAG: --eviction-hard="" Jul 13 21:59:38.775453 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769807 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 13 21:59:38.775453 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769810 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 13 21:59:38.775453 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769813 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 13 21:59:38.775453 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769816 2562 flags.go:64] FLAG: --eviction-soft="" Jul 13 21:59:38.775453 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769820 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 13 21:59:38.775453 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769822 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 13 21:59:38.776095 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769826 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 13 21:59:38.776095 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769828 2562 flags.go:64] FLAG: --experimental-mounter-path="" Jul 13 21:59:38.776095 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769831 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 13 21:59:38.776095 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769834 2562 flags.go:64] FLAG: --fail-swap-on="true" Jul 13 21:59:38.776095 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769837 2562 flags.go:64] FLAG: --feature-gates="" Jul 13 21:59:38.776095 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769841 2562 flags.go:64] FLAG: --file-check-frequency="20s" Jul 13 21:59:38.776095 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769844 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 13 21:59:38.776095 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769847 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 13 21:59:38.776095 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769850 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 13 21:59:38.776095 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769854 2562 flags.go:64] FLAG: --healthz-port="10248" Jul 13 21:59:38.776095 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769857 2562 flags.go:64] FLAG: --help="false" Jul 13 21:59:38.776095 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769860 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-132-128.ec2.internal" Jul 13 21:59:38.776095 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769863 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 13 21:59:38.776095 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769866 2562 flags.go:64] FLAG: --http-check-frequency="20s" Jul 13 21:59:38.776095 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769869 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 13 21:59:38.776095 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769872 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 13 21:59:38.776095 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769876 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 13 21:59:38.776095 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769879 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 13 21:59:38.776095 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769882 2562 flags.go:64] FLAG: --image-service-endpoint="" Jul 13 21:59:38.776095 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769884 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 13 21:59:38.776095 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769887 2562 flags.go:64] FLAG: --kube-api-burst="100" Jul 13 21:59:38.776095 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769890 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 13 21:59:38.776095 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769893 2562 flags.go:64] FLAG: --kube-api-qps="50" Jul 13 21:59:38.776095 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769896 2562 flags.go:64] FLAG: --kube-reserved="" Jul 13 21:59:38.776670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769899 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 13 21:59:38.776670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769902 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 13 21:59:38.776670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769906 2562 flags.go:64] FLAG: --kubelet-cgroups="" Jul 13 21:59:38.776670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769909 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 13 21:59:38.776670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769912 2562 flags.go:64] FLAG: --lock-file="" Jul 13 21:59:38.776670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769915 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 13 21:59:38.776670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769918 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 13 21:59:38.776670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769921 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 13 21:59:38.776670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769927 2562 flags.go:64] FLAG: --log-json-split-stream="false" Jul 13 21:59:38.776670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769929 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 13 21:59:38.776670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769932 2562 flags.go:64] FLAG: --log-text-split-stream="false" Jul 13 21:59:38.776670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769935 2562 flags.go:64] FLAG: --logging-format="text" Jul 13 21:59:38.776670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769938 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 13 21:59:38.776670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769942 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 13 21:59:38.776670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769944 2562 flags.go:64] FLAG: --manifest-url="" Jul 13 21:59:38.776670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769948 2562 flags.go:64] FLAG: --manifest-url-header="" Jul 13 21:59:38.776670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769952 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 13 21:59:38.776670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769955 2562 flags.go:64] FLAG: --max-open-files="1000000" Jul 13 21:59:38.776670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769959 2562 flags.go:64] FLAG: --max-pods="110" Jul 13 21:59:38.776670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769962 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 13 21:59:38.776670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769965 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 13 21:59:38.776670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769968 2562 flags.go:64] FLAG: --memory-manager-policy="None" Jul 13 21:59:38.776670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769971 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 13 21:59:38.776670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769987 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 13 21:59:38.776670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769990 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 13 21:59:38.777289 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.769993 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 13 21:59:38.777289 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770000 2562 flags.go:64] FLAG: --node-status-max-images="50" Jul 13 21:59:38.777289 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770003 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 13 21:59:38.777289 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770007 2562 flags.go:64] FLAG: --oom-score-adj="-999" Jul 13 21:59:38.777289 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770010 2562 flags.go:64] FLAG: --pod-cidr="" Jul 13 21:59:38.777289 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770013 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 13 21:59:38.777289 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770018 2562 flags.go:64] FLAG: --pod-manifest-path="" Jul 13 21:59:38.777289 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770021 2562 flags.go:64] FLAG: --pod-max-pids="-1" Jul 13 21:59:38.777289 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770025 2562 flags.go:64] FLAG: --pods-per-core="0" Jul 13 21:59:38.777289 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770028 2562 flags.go:64] FLAG: --port="10250" Jul 13 21:59:38.777289 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770031 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 13 21:59:38.777289 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770034 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f34b4ad40eeadbae" Jul 13 21:59:38.777289 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770037 2562 flags.go:64] FLAG: --qos-reserved="" Jul 13 21:59:38.777289 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770040 2562 flags.go:64] FLAG: --read-only-port="10255" Jul 13 21:59:38.777289 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770043 2562 flags.go:64] FLAG: --register-node="true" Jul 13 21:59:38.777289 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770046 2562 flags.go:64] FLAG: --register-with-taints="" Jul 13 21:59:38.777289 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770049 2562 flags.go:64] FLAG: --registry-burst="10" Jul 13 21:59:38.777289 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770052 2562 flags.go:64] FLAG: --registry-qps="5" Jul 13 21:59:38.777289 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770055 2562 flags.go:64] FLAG: --reserved-cpus="" Jul 13 21:59:38.777289 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770058 2562 flags.go:64] FLAG: --reserved-memory="" Jul 13 21:59:38.777289 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770062 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 13 21:59:38.777289 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770065 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 13 21:59:38.777289 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770068 2562 flags.go:64] FLAG: --rotate-certificates="false" Jul 13 21:59:38.777289 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770071 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 13 21:59:38.777289 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770074 2562 flags.go:64] FLAG: --runonce="false" Jul 13 21:59:38.777875 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770076 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 13 21:59:38.777875 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770079 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 13 21:59:38.777875 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770082 2562 flags.go:64] FLAG: --seccomp-default="false" Jul 13 21:59:38.777875 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770085 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 13 21:59:38.777875 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770088 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 13 21:59:38.777875 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770091 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 13 21:59:38.777875 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770094 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 13 21:59:38.777875 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770097 2562 flags.go:64] FLAG: --storage-driver-password="root" Jul 13 21:59:38.777875 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770102 2562 flags.go:64] FLAG: --storage-driver-secure="false" Jul 13 21:59:38.777875 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770106 2562 flags.go:64] FLAG: --storage-driver-table="stats" Jul 13 21:59:38.777875 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770109 2562 flags.go:64] FLAG: --storage-driver-user="root" Jul 13 21:59:38.777875 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770112 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 13 21:59:38.777875 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770115 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 13 21:59:38.777875 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770118 2562 flags.go:64] FLAG: --system-cgroups="" Jul 13 21:59:38.777875 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770122 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 13 21:59:38.777875 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770128 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 13 21:59:38.777875 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770131 2562 flags.go:64] FLAG: --tls-cert-file="" Jul 13 21:59:38.777875 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770134 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 13 21:59:38.777875 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770139 2562 flags.go:64] FLAG: --tls-min-version="" Jul 13 21:59:38.777875 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770143 2562 flags.go:64] FLAG: --tls-private-key-file="" Jul 13 21:59:38.777875 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770145 2562 flags.go:64] FLAG: --topology-manager-policy="none" Jul 13 21:59:38.777875 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770148 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 13 21:59:38.777875 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770151 2562 flags.go:64] FLAG: --topology-manager-scope="container" Jul 13 21:59:38.777875 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770154 2562 flags.go:64] FLAG: --v="2" Jul 13 21:59:38.777875 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770158 2562 flags.go:64] FLAG: --version="false" Jul 13 21:59:38.778487 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770162 2562 flags.go:64] FLAG: --vmodule="" Jul 13 21:59:38.778487 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770166 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 13 21:59:38.778487 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770169 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 13 21:59:38.778487 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770286 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 21:59:38.778487 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770290 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 21:59:38.778487 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770293 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 21:59:38.778487 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770296 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 21:59:38.778487 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770298 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 21:59:38.778487 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770301 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 21:59:38.778487 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770303 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 21:59:38.778487 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770306 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 21:59:38.778487 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770309 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 21:59:38.778487 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770312 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 21:59:38.778487 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770314 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 21:59:38.778487 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770317 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 21:59:38.778487 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770321 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 21:59:38.778487 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770323 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 21:59:38.778487 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770326 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 21:59:38.778487 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770328 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 21:59:38.778487 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770331 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 21:59:38.778487 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770334 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 21:59:38.778994 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770337 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 21:59:38.778994 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770340 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 21:59:38.778994 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770342 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 21:59:38.778994 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770345 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 21:59:38.778994 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770348 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 21:59:38.778994 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770351 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 21:59:38.778994 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770354 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 21:59:38.778994 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770356 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 13 21:59:38.778994 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770359 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 21:59:38.778994 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770361 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 21:59:38.778994 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770364 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 21:59:38.778994 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770366 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 21:59:38.778994 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770369 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 21:59:38.778994 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770371 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 21:59:38.778994 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770374 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 21:59:38.778994 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770377 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 21:59:38.778994 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770381 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 21:59:38.778994 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770385 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 21:59:38.778994 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770388 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 21:59:38.778994 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770391 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 21:59:38.779470 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770394 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 21:59:38.779470 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770396 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 21:59:38.779470 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770399 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 21:59:38.779470 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770403 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 21:59:38.779470 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770405 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 21:59:38.779470 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770408 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 21:59:38.779470 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770412 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 21:59:38.779470 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770415 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 21:59:38.779470 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770418 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 21:59:38.779470 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770420 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 21:59:38.779470 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770423 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 21:59:38.779470 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770426 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 21:59:38.779470 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770430 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 21:59:38.779470 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770432 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 21:59:38.779470 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770435 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 21:59:38.779470 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770437 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 21:59:38.779470 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770440 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 21:59:38.779470 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770443 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 21:59:38.779470 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770446 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 21:59:38.779951 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770449 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 21:59:38.779951 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770451 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 21:59:38.779951 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770454 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 21:59:38.779951 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770456 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 21:59:38.779951 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770458 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 21:59:38.779951 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770461 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 21:59:38.779951 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770463 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 21:59:38.779951 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770466 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 21:59:38.779951 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770469 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 21:59:38.779951 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770471 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 21:59:38.779951 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770474 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 21:59:38.779951 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770476 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 21:59:38.779951 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770479 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 21:59:38.779951 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770483 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 21:59:38.779951 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770486 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 21:59:38.779951 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770489 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 21:59:38.779951 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770491 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 21:59:38.779951 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770496 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 21:59:38.779951 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770499 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 21:59:38.780419 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770503 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 21:59:38.780419 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770506 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 21:59:38.780419 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770509 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 21:59:38.780419 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770511 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 21:59:38.780419 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770514 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 21:59:38.780419 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770516 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 21:59:38.780419 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770521 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 21:59:38.780419 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770524 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 21:59:38.780419 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770526 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 21:59:38.780419 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770529 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 21:59:38.780419 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770531 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 21:59:38.780419 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770534 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 21:59:38.780419 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770537 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 21:59:38.780419 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770540 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 21:59:38.780419 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770542 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 21:59:38.780419 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770545 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 21:59:38.780419 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770547 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 21:59:38.780419 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770549 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 21:59:38.780419 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770552 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 21:59:38.780419 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770555 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 21:59:38.780906 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770558 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 21:59:38.780906 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770560 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 21:59:38.780906 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.770563 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 21:59:38.780906 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.770571 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 21:59:38.780906 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.777156 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 13 21:59:38.780906 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.777170 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 13 21:59:38.780906 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.777809 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 13 21:59:38.780906 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.777816 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 13 21:59:38.780906 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777890 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 21:59:38.780906 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777896 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 21:59:38.780906 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777899 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 21:59:38.780906 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777902 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 21:59:38.780906 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777906 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 21:59:38.780906 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777909 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 21:59:38.780906 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777912 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 21:59:38.781286 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777915 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 21:59:38.781286 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777917 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 21:59:38.781286 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777920 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 21:59:38.781286 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777923 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 21:59:38.781286 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777926 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 21:59:38.781286 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777929 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 21:59:38.781286 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777932 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 21:59:38.781286 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777934 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 21:59:38.781286 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777937 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 21:59:38.781286 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777939 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 21:59:38.781286 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777942 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 21:59:38.781286 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777944 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 21:59:38.781286 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777947 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 21:59:38.781286 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777949 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 21:59:38.781286 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777951 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 21:59:38.781286 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777954 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 21:59:38.781286 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777956 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 21:59:38.781286 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777959 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 21:59:38.781286 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777961 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 21:59:38.781286 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777964 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 21:59:38.781752 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777966 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 21:59:38.781752 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777969 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 21:59:38.781752 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777983 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 21:59:38.781752 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777986 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 21:59:38.781752 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777989 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 21:59:38.781752 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777991 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 21:59:38.781752 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777994 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 21:59:38.781752 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777996 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 21:59:38.781752 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.777999 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 21:59:38.781752 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778002 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 21:59:38.781752 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778004 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 21:59:38.781752 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778007 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 21:59:38.781752 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778009 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 21:59:38.781752 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778012 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 13 21:59:38.781752 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778015 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 21:59:38.781752 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778017 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 21:59:38.781752 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778020 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 21:59:38.781752 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778023 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 21:59:38.781752 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778025 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 21:59:38.781752 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778028 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 21:59:38.782241 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778031 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 21:59:38.782241 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778033 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 21:59:38.782241 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778036 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 21:59:38.782241 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778039 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 21:59:38.782241 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778041 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 21:59:38.782241 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778044 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 21:59:38.782241 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778046 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 21:59:38.782241 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778049 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 21:59:38.782241 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778051 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 21:59:38.782241 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778054 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 21:59:38.782241 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778056 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 21:59:38.782241 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778059 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 21:59:38.782241 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778062 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 21:59:38.782241 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778064 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 21:59:38.782241 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778067 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 21:59:38.782241 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778070 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 21:59:38.782241 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778072 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 21:59:38.782241 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778075 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 21:59:38.782241 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778077 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 21:59:38.782241 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778079 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 21:59:38.782716 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778082 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 21:59:38.782716 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778086 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 21:59:38.782716 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778091 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 21:59:38.782716 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778094 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 21:59:38.782716 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778097 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 21:59:38.782716 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778099 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 21:59:38.782716 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778102 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 21:59:38.782716 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778104 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 21:59:38.782716 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778107 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 21:59:38.782716 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778110 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 21:59:38.782716 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778112 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 21:59:38.782716 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778115 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 21:59:38.782716 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778117 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 21:59:38.782716 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778120 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 21:59:38.782716 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778122 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 21:59:38.782716 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778124 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 21:59:38.782716 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778127 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 21:59:38.782716 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778129 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 21:59:38.782716 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778132 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 21:59:38.783181 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778134 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 21:59:38.783181 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778138 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 21:59:38.783181 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778141 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 21:59:38.783181 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778144 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 21:59:38.783181 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778147 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 21:59:38.783181 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778150 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 21:59:38.783181 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778152 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 21:59:38.783181 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778155 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 21:59:38.783181 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778158 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 21:59:38.783181 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778161 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 21:59:38.783181 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778164 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 21:59:38.783181 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778167 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 21:59:38.783181 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778169 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 21:59:38.783181 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.778174 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 21:59:38.783514 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778288 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 13 21:59:38.783514 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778293 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 13 21:59:38.783514 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778296 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 13 21:59:38.783514 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778299 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 13 21:59:38.783514 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778302 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 13 21:59:38.783514 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778305 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 13 21:59:38.783514 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778307 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 13 21:59:38.783514 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778310 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 13 21:59:38.783514 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778312 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 13 21:59:38.783514 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778315 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 13 21:59:38.783514 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778318 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 13 21:59:38.783514 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778320 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 13 21:59:38.783514 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778323 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 13 21:59:38.783514 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778325 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 13 21:59:38.783514 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778328 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 13 21:59:38.783514 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778330 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 13 21:59:38.783514 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778333 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 13 21:59:38.783514 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778335 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 13 21:59:38.783514 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778338 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 13 21:59:38.783514 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778340 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 13 21:59:38.784007 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778343 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 13 21:59:38.784007 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778345 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 13 21:59:38.784007 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778348 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 13 21:59:38.784007 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778350 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 13 21:59:38.784007 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778353 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 13 21:59:38.784007 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778356 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 13 21:59:38.784007 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778358 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 13 21:59:38.784007 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778361 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 13 21:59:38.784007 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778364 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 13 21:59:38.784007 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778367 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 13 21:59:38.784007 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778370 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 13 21:59:38.784007 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778373 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 13 21:59:38.784007 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778376 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 13 21:59:38.784007 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778380 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 13 21:59:38.784007 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778383 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 13 21:59:38.784007 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778385 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 13 21:59:38.784007 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778388 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 13 21:59:38.784007 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778390 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 13 21:59:38.784007 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778393 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 13 21:59:38.784007 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778395 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 13 21:59:38.784471 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778398 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 13 21:59:38.784471 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778400 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 13 21:59:38.784471 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778403 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 13 21:59:38.784471 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778405 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 13 21:59:38.784471 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778408 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 13 21:59:38.784471 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778410 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 13 21:59:38.784471 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778414 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 13 21:59:38.784471 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778417 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 13 21:59:38.784471 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778420 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 13 21:59:38.784471 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778423 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 13 21:59:38.784471 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778425 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 13 21:59:38.784471 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778428 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 13 21:59:38.784471 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778431 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 13 21:59:38.784471 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778434 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 13 21:59:38.784471 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778437 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 13 21:59:38.784471 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778439 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 13 21:59:38.784471 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778442 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 13 21:59:38.784471 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778445 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 13 21:59:38.784471 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778447 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 13 21:59:38.784471 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778449 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 13 21:59:38.785345 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778452 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 13 21:59:38.785345 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778455 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 13 21:59:38.785345 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778457 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 13 21:59:38.785345 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778460 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 13 21:59:38.785345 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778462 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 13 21:59:38.785345 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778465 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 13 21:59:38.785345 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778469 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 13 21:59:38.785345 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778472 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 13 21:59:38.785345 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778474 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 13 21:59:38.785345 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778477 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 13 21:59:38.785345 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778479 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 13 21:59:38.785345 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778482 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 13 21:59:38.785345 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778484 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 13 21:59:38.785345 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778487 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 13 21:59:38.785345 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778489 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 13 21:59:38.785345 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778492 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 13 21:59:38.785345 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778494 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 13 21:59:38.785345 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778497 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 13 21:59:38.785345 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778500 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 13 21:59:38.786098 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778504 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 13 21:59:38.786098 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778507 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 13 21:59:38.786098 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778511 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 13 21:59:38.786098 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778514 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 13 21:59:38.786098 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778517 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 13 21:59:38.786098 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778519 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 13 21:59:38.786098 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778522 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 13 21:59:38.786098 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778524 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 13 21:59:38.786098 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778527 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 13 21:59:38.786098 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778529 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 13 21:59:38.786098 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778532 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 13 21:59:38.786098 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778534 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 13 21:59:38.786098 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778536 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 13 21:59:38.786098 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778539 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 13 21:59:38.786098 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778541 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 13 21:59:38.786098 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778544 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 13 21:59:38.786098 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778547 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 13 21:59:38.786098 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778550 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 13 21:59:38.786098 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778552 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 13 21:59:38.786556 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:38.778555 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 13 21:59:38.786556 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.778560 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 13 21:59:38.786556 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.778659 2562 server.go:962] "Client rotation is on, will bootstrap in background" Jul 13 21:59:38.786556 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.780596 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 13 21:59:38.786556 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.781771 2562 server.go:1019] "Starting client certificate rotation" Jul 13 21:59:38.786556 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.781873 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 13 21:59:38.786556 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.781904 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 13 21:59:38.786556 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.783910 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 13 21:59:38.787730 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.787705 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 13 21:59:38.796534 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.796519 2562 log.go:25] "Validated CRI v1 runtime API" Jul 13 21:59:38.798398 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.798383 2562 log.go:25] "Validated CRI v1 image API" Jul 13 21:59:38.799085 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.799069 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 13 21:59:38.801165 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.801147 2562 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7ed90ff7-4c4e-44f5-9b0a-fc77c30d804a:/dev/nvme0n1p3 e560ab08-2f81-4d93-951f-1275bf20b408:/dev/nvme0n1p4] Jul 13 21:59:38.801224 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.801165 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 13 21:59:38.806199 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.806045 2562 manager.go:216] Machine: {Timestamp:2026-07-13 21:59:38.805229283 +0000 UTC m=+0.255784815 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099757 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b629c6db9c4bfa8fdd8b779d86e8c SystemUUID:ec2b629c-6db9-c4bf-a8fd-d8b779d86e8c BootID:639a9da8-41b8-4dad-ab3d-a16b785626e9 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:64:96:6a:ec:9d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:64:96:6a:ec:9d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:a7:79:bc:84:76 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 13 21:59:38.806199 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.806196 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 13 21:59:38.806299 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.806288 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 13 21:59:38.808920 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.808897 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 13 21:59:38.809066 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.808923 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-128.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 13 21:59:38.809548 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.809538 2562 topology_manager.go:138] "Creating topology manager with none policy" Jul 13 21:59:38.809578 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.809550 2562 container_manager_linux.go:309] "Creating device plugin manager" Jul 13 21:59:38.809578 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.809557 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 13 21:59:38.810155 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.810145 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 13 21:59:38.811347 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.811338 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 13 21:59:38.815348 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.815336 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 13 21:59:38.815474 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.815465 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 13 21:59:38.816220 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.816211 2562 kubelet.go:486] "Attempting to sync node with API server" Jul 13 21:59:38.816252 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.816224 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 13 21:59:38.816252 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.816236 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 13 21:59:38.816252 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.816245 2562 kubelet.go:398] "Adding apiserver pod source" Jul 13 21:59:38.816350 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.816255 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 13 21:59:38.816656 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.816642 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 13 21:59:38.816703 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.816670 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 13 21:59:38.818640 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.818623 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 13 21:59:38.820359 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.820346 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 13 21:59:38.820410 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.820375 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 13 21:59:38.820581 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.820569 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 13 21:59:38.820612 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.820587 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 13 21:59:38.820612 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.820593 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 13 21:59:38.820612 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.820599 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 13 21:59:38.820612 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.820607 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 13 21:59:38.820612 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.820613 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 13 21:59:38.821036 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.821027 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 13 21:59:38.821069 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.821036 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 13 21:59:38.821069 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.821048 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 13 21:59:38.821069 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.821053 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 13 21:59:38.821069 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.821066 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 13 21:59:38.821168 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.821074 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 13 21:59:38.821655 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.821645 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 13 21:59:38.821655 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.821654 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 13 21:59:38.824902 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.824879 2562 server.go:1268] "Started kubelet" Jul 13 21:59:38.825294 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.825038 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 13 21:59:38.825588 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.825041 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 13 21:59:38.825829 ip-10-0-132-128 systemd[1]: Started Kubernetes Kubelet. Jul 13 21:59:38.827681 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.827655 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 13 21:59:38.828785 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.828760 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 13 21:59:38.831814 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.831799 2562 server.go:310] "Adding debug handlers to kubelet server" Jul 13 21:59:38.831958 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:38.831936 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 13 21:59:38.834604 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.834592 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 13 21:59:38.834664 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.834602 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 13 21:59:38.834724 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:38.834685 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-128.ec2.internal\" not found" Jul 13 21:59:38.835484 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.835455 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 13 21:59:38.835484 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.835458 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 13 21:59:38.835484 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.835488 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 13 21:59:38.835670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.835498 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 13 21:59:38.835670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.835510 2562 factory.go:55] Registering systemd factory Jul 13 21:59:38.835670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.835522 2562 factory.go:223] Registration of the systemd container factory successfully Jul 13 21:59:38.835670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.835556 2562 reconstruct.go:97] "Volume reconstruction finished" Jul 13 21:59:38.835670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.835568 2562 reconciler.go:29] "Reconciler: start to sync state" Jul 13 21:59:38.836002 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.835969 2562 factory.go:153] Registering CRI-O factory Jul 13 21:59:38.836002 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.836002 2562 factory.go:223] Registration of the crio container factory successfully Jul 13 21:59:38.836136 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.836025 2562 factory.go:103] Registering Raw factory Jul 13 21:59:38.836136 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.836041 2562 manager.go:1196] Started watching for new ooms in manager Jul 13 21:59:38.836534 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.836513 2562 manager.go:349] Starting recovery of all containers Jul 13 21:59:38.839670 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:38.839643 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-128.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 13 21:59:38.839769 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.839677 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-128.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 13 21:59:38.839769 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:38.839695 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 13 21:59:38.840199 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:38.840173 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 13 21:59:38.840269 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:38.840249 2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-128.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 13 21:59:38.840656 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:38.839835 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-128.ec2.internal.18c1f856bc423d73 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-128.ec2.internal,UID:ip-10-0-132-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-128.ec2.internal,},FirstTimestamp:2026-07-13 21:59:38.824760691 +0000 UTC m=+0.275316218,LastTimestamp:2026-07-13 21:59:38.824760691 +0000 UTC m=+0.275316218,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-128.ec2.internal,}" Jul 13 21:59:38.844416 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.844400 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 13 21:59:38.846385 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.846368 2562 manager.go:354] Recovery completed Jul 13 21:59:38.851054 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.851041 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 21:59:38.854441 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.854425 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-128.ec2.internal" event="NodeHasSufficientMemory" Jul 13 21:59:38.854530 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.854456 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-128.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 21:59:38.854530 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.854469 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-128.ec2.internal" event="NodeHasSufficientPID" Jul 13 21:59:38.854964 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.854941 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 13 21:59:38.854964 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.854965 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 13 21:59:38.855076 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.854955 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 21:59:38.855076 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.854996 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 13 21:59:38.855835 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.855819 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-128.ec2.internal" event="NodeHasSufficientMemory" Jul 13 21:59:38.855886 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.855849 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-128.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 21:59:38.855886 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.855864 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-128.ec2.internal" event="NodeHasSufficientPID" Jul 13 21:59:38.857146 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.857134 2562 policy_none.go:49] "None policy: Start" Jul 13 21:59:38.857193 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.857150 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 13 21:59:38.857193 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.857171 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 13 21:59:38.859173 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.859162 2562 policy_none.go:47] "Start" Jul 13 21:59:38.859235 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:38.859176 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-128.ec2.internal.18c1f856be0721e0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-128.ec2.internal,UID:ip-10-0-132-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-128.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-128.ec2.internal,},FirstTimestamp:2026-07-13 21:59:38.85444144 +0000 UTC m=+0.304996968,LastTimestamp:2026-07-13 21:59:38.85444144 +0000 UTC m=+0.304996968,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-128.ec2.internal,}" Jul 13 21:59:38.874145 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:38.873959 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-128.ec2.internal.18c1f856be077739 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-128.ec2.internal,UID:ip-10-0-132-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-128.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-128.ec2.internal,},FirstTimestamp:2026-07-13 21:59:38.854463289 +0000 UTC m=+0.305018818,LastTimestamp:2026-07-13 21:59:38.854463289 +0000 UTC m=+0.305018818,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-128.ec2.internal,}" Jul 13 21:59:38.885770 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:38.885698 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-128.ec2.internal.18c1f856be079ed5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-128.ec2.internal,UID:ip-10-0-132-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-128.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-128.ec2.internal,},FirstTimestamp:2026-07-13 21:59:38.854473429 +0000 UTC m=+0.305028957,LastTimestamp:2026-07-13 21:59:38.854473429 +0000 UTC m=+0.305028957,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-128.ec2.internal,}" Jul 13 21:59:38.898499 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.898479 2562 manager.go:337] "Starting Device Plugin manager" Jul 13 21:59:38.898562 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:38.898517 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 13 21:59:38.898562 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.898526 2562 server.go:87] "Starting device plugin registration server" Jul 13 21:59:38.898770 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.898757 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 13 21:59:38.898843 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.898773 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 13 21:59:38.898906 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.898855 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 13 21:59:38.898952 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.898923 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 13 21:59:38.898952 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.898931 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 13 21:59:38.899691 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:38.899662 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 13 21:59:38.899772 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:38.899705 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-128.ec2.internal\" not found" Jul 13 21:59:38.904831 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:38.904762 2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-128.ec2.internal.18c1f856be0721e0\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-128.ec2.internal.18c1f856be0721e0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-128.ec2.internal,UID:ip-10-0-132-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-128.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-128.ec2.internal,},FirstTimestamp:2026-07-13 21:59:38.85444144 +0000 UTC m=+0.304996968,LastTimestamp:2026-07-13 21:59:38.855834991 +0000 UTC m=+0.306390520,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-128.ec2.internal,}" Jul 13 21:59:38.916568 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:38.916491 2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-128.ec2.internal.18c1f856be077739\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-128.ec2.internal.18c1f856be077739 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-128.ec2.internal,UID:ip-10-0-132-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-128.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-128.ec2.internal,},FirstTimestamp:2026-07-13 21:59:38.854463289 +0000 UTC m=+0.305018818,LastTimestamp:2026-07-13 21:59:38.855856415 +0000 UTC m=+0.306412156,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-128.ec2.internal,}" Jul 13 21:59:38.931449 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:38.927574 2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-128.ec2.internal.18c1f856be079ed5\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-128.ec2.internal.18c1f856be079ed5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-128.ec2.internal,UID:ip-10-0-132-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-128.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-128.ec2.internal,},FirstTimestamp:2026-07-13 21:59:38.854473429 +0000 UTC m=+0.305028957,LastTimestamp:2026-07-13 21:59:38.855870469 +0000 UTC m=+0.306425998,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-128.ec2.internal,}" Jul 13 21:59:38.943532 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:38.943475 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-128.ec2.internal.18c1f856c0c6b057 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-128.ec2.internal,UID:ip-10-0-132-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-132-128.ec2.internal,},FirstTimestamp:2026-07-13 21:59:38.900549719 +0000 UTC m=+0.351105237,LastTimestamp:2026-07-13 21:59:38.900549719 +0000 UTC m=+0.351105237,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-128.ec2.internal,}" Jul 13 21:59:38.958501 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.958470 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 13 21:59:38.959670 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.959648 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 13 21:59:38.959750 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.959687 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 13 21:59:38.959750 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.959710 2562 kubelet.go:2448] "Starting kubelet main sync loop" Jul 13 21:59:38.959750 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:38.959742 2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 13 21:59:38.977144 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:38.977115 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Jul 13 21:59:38.999222 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:38.999198 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 21:59:39.000108 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.000091 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-128.ec2.internal" event="NodeHasSufficientMemory" Jul 13 21:59:39.000182 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.000124 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-128.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 21:59:39.000182 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.000137 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-128.ec2.internal" event="NodeHasSufficientPID" Jul 13 21:59:39.000182 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.000160 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-128.ec2.internal" Jul 13 21:59:39.011377 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:39.011296 2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-128.ec2.internal.18c1f856be0721e0\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-128.ec2.internal.18c1f856be0721e0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-128.ec2.internal,UID:ip-10-0-132-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-128.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-128.ec2.internal,},FirstTimestamp:2026-07-13 21:59:38.85444144 +0000 UTC m=+0.304996968,LastTimestamp:2026-07-13 21:59:39.000107536 +0000 UTC m=+0.450663085,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-128.ec2.internal,}" Jul 13 21:59:39.024625 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:39.024522 2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-128.ec2.internal.18c1f856be077739\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-128.ec2.internal.18c1f856be077739 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-128.ec2.internal,UID:ip-10-0-132-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-128.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-128.ec2.internal,},FirstTimestamp:2026-07-13 21:59:38.854463289 +0000 UTC m=+0.305018818,LastTimestamp:2026-07-13 21:59:39.00012937 +0000 UTC m=+0.450684898,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-128.ec2.internal,}" Jul 13 21:59:39.024823 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:39.024804 2562 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-132-128.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-132-128.ec2.internal" Jul 13 21:59:39.035679 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:39.035618 2562 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-128.ec2.internal.18c1f856be079ed5\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-128.ec2.internal.18c1f856be079ed5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-128.ec2.internal,UID:ip-10-0-132-128.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-128.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-128.ec2.internal,},FirstTimestamp:2026-07-13 21:59:38.854473429 +0000 UTC m=+0.305028957,LastTimestamp:2026-07-13 21:59:39.000141639 +0000 UTC m=+0.450697167,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-128.ec2.internal,}" Jul 13 21:59:39.053168 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:39.053138 2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-128.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms" Jul 13 21:59:39.054576 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.054560 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hqxsp" Jul 13 21:59:39.060616 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.060596 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-128.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-128.ec2.internal"] Jul 13 21:59:39.060667 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.060657 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 21:59:39.062024 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.062009 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-128.ec2.internal" event="NodeHasSufficientMemory" Jul 13 21:59:39.062096 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.062034 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-128.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 21:59:39.062096 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.062045 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-128.ec2.internal" event="NodeHasSufficientPID" Jul 13 21:59:39.062818 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.062807 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hqxsp" Jul 13 21:59:39.064331 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.064319 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 21:59:39.064436 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.064423 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-128.ec2.internal" Jul 13 21:59:39.064482 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.064450 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 21:59:39.064986 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.064953 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-128.ec2.internal" event="NodeHasSufficientMemory" Jul 13 21:59:39.065052 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.064997 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-128.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 21:59:39.065052 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.065010 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-128.ec2.internal" event="NodeHasSufficientPID" Jul 13 21:59:39.065439 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.065426 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-128.ec2.internal" event="NodeHasSufficientMemory" Jul 13 21:59:39.065490 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.065449 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-128.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 21:59:39.065490 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.065457 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-128.ec2.internal" event="NodeHasSufficientPID" Jul 13 21:59:39.067145 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.067129 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-128.ec2.internal" Jul 13 21:59:39.067207 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.067162 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 21:59:39.067710 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.067696 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-128.ec2.internal" event="NodeHasSufficientMemory" Jul 13 21:59:39.067773 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.067716 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-128.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 21:59:39.067773 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.067728 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-128.ec2.internal" event="NodeHasSufficientPID" Jul 13 21:59:39.089522 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:39.089504 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-128.ec2.internal\" not found" node="ip-10-0-132-128.ec2.internal" Jul 13 21:59:39.093582 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:39.093567 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-128.ec2.internal\" not found" node="ip-10-0-132-128.ec2.internal" Jul 13 21:59:39.137458 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.137429 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a282de8bb107da660700ce7cc100840d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-128.ec2.internal\" (UID: \"a282de8bb107da660700ce7cc100840d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-128.ec2.internal" Jul 13 21:59:39.137458 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.137461 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a282de8bb107da660700ce7cc100840d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-128.ec2.internal\" (UID: \"a282de8bb107da660700ce7cc100840d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-128.ec2.internal" Jul 13 21:59:39.137600 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.137478 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fad131b74fcb56c348327d4dd10f9830-config\") pod \"kube-apiserver-proxy-ip-10-0-132-128.ec2.internal\" (UID: \"fad131b74fcb56c348327d4dd10f9830\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-128.ec2.internal" Jul 13 21:59:39.225699 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.225667 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 13 21:59:39.226593 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.226569 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-128.ec2.internal" event="NodeHasSufficientMemory" Jul 13 21:59:39.226679 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.226600 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-128.ec2.internal" event="NodeHasNoDiskPressure" Jul 13 21:59:39.226679 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.226609 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-128.ec2.internal" event="NodeHasSufficientPID" Jul 13 21:59:39.226679 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.226638 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-128.ec2.internal" Jul 13 21:59:39.232943 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.232930 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-128.ec2.internal" Jul 13 21:59:39.233000 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:39.232951 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-128.ec2.internal\": node \"ip-10-0-132-128.ec2.internal\" not found" Jul 13 21:59:39.237838 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.237821 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a282de8bb107da660700ce7cc100840d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-128.ec2.internal\" (UID: \"a282de8bb107da660700ce7cc100840d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-128.ec2.internal" Jul 13 21:59:39.237909 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.237844 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a282de8bb107da660700ce7cc100840d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-128.ec2.internal\" (UID: \"a282de8bb107da660700ce7cc100840d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-128.ec2.internal" Jul 13 21:59:39.237909 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.237859 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fad131b74fcb56c348327d4dd10f9830-config\") pod \"kube-apiserver-proxy-ip-10-0-132-128.ec2.internal\" (UID: \"fad131b74fcb56c348327d4dd10f9830\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-128.ec2.internal" Jul 13 21:59:39.237909 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.237896 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fad131b74fcb56c348327d4dd10f9830-config\") pod \"kube-apiserver-proxy-ip-10-0-132-128.ec2.internal\" (UID: \"fad131b74fcb56c348327d4dd10f9830\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-128.ec2.internal" Jul 13 21:59:39.238034 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.237922 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a282de8bb107da660700ce7cc100840d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-128.ec2.internal\" (UID: \"a282de8bb107da660700ce7cc100840d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-128.ec2.internal" Jul 13 21:59:39.238034 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.237930 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a282de8bb107da660700ce7cc100840d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-128.ec2.internal\" (UID: \"a282de8bb107da660700ce7cc100840d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-128.ec2.internal" Jul 13 21:59:39.261551 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:39.261526 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-128.ec2.internal\" not found" Jul 13 21:59:39.362085 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:39.362029 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-128.ec2.internal\" not found" Jul 13 21:59:39.391184 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.391159 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-128.ec2.internal" Jul 13 21:59:39.396830 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.396812 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-128.ec2.internal" Jul 13 21:59:39.462580 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:39.462548 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-128.ec2.internal\" not found" Jul 13 21:59:39.562983 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:39.562951 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-128.ec2.internal\" not found" Jul 13 21:59:39.660072 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.660008 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 21:59:39.663504 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:39.663484 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-128.ec2.internal\" not found" Jul 13 21:59:39.764060 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:39.764029 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-128.ec2.internal\" not found" Jul 13 21:59:39.781207 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.781178 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 13 21:59:39.781346 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.781330 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 13 21:59:39.864281 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:39.864252 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-128.ec2.internal\" not found" Jul 13 21:59:39.890343 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.890315 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 21:59:39.935178 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.935112 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-128.ec2.internal" Jul 13 21:59:39.947457 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.947439 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 13 21:59:39.948894 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.948877 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-128.ec2.internal" Jul 13 21:59:39.953990 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:39.953933 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad131b74fcb56c348327d4dd10f9830.slice/crio-42f5dae75080ad07b141671a0d09db8c673114262321fa5a46dfa57cb01cbac5 WatchSource:0}: Error finding container 42f5dae75080ad07b141671a0d09db8c673114262321fa5a46dfa57cb01cbac5: Status 404 returned error can't find the container with id 42f5dae75080ad07b141671a0d09db8c673114262321fa5a46dfa57cb01cbac5 Jul 13 21:59:39.954234 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:39.954220 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda282de8bb107da660700ce7cc100840d.slice/crio-bc8c712d3a79494e6a638a893bc1639b8e7c9defb7627e7fa459c57bc9b9c76a WatchSource:0}: Error finding container bc8c712d3a79494e6a638a893bc1639b8e7c9defb7627e7fa459c57bc9b9c76a: Status 404 returned error can't find the container with id bc8c712d3a79494e6a638a893bc1639b8e7c9defb7627e7fa459c57bc9b9c76a Jul 13 21:59:39.956138 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.956122 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 13 21:59:39.958350 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.958337 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 21:59:39.962761 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.962722 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-128.ec2.internal" event={"ID":"a282de8bb107da660700ce7cc100840d","Type":"ContainerStarted","Data":"bc8c712d3a79494e6a638a893bc1639b8e7c9defb7627e7fa459c57bc9b9c76a"} Jul 13 21:59:39.963629 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:39.963592 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-128.ec2.internal" event={"ID":"fad131b74fcb56c348327d4dd10f9830","Type":"ContainerStarted","Data":"42f5dae75080ad07b141671a0d09db8c673114262321fa5a46dfa57cb01cbac5"} Jul 13 21:59:40.066338 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.066294 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-12 21:54:38 +0000 UTC" deadline="2028-03-06 08:04:58.308843209 +0000 UTC" Jul 13 21:59:40.066338 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.066330 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14434h5m18.242517258s" Jul 13 21:59:40.072827 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.072809 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 21:59:40.156465 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.156435 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 21:59:40.817650 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.817614 2562 apiserver.go:52] "Watching apiserver" Jul 13 21:59:40.824606 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.824587 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 13 21:59:40.825010 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.824989 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-lvr8p","openshift-multus/network-metrics-daemon-w8fht","openshift-cluster-node-tuning-operator/tuned-nshp2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-128.ec2.internal","openshift-multus/multus-svxd6","openshift-network-diagnostics/network-check-target-nbdz9","openshift-network-operator/iptables-alerter-29z8f","openshift-ovn-kubernetes/ovnkube-node-zw4h7","kube-system/global-pull-secret-syncer-zkrhd","kube-system/konnectivity-agent-pz6k2","kube-system/kube-apiserver-proxy-ip-10-0-132-128.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx","openshift-dns/node-resolver-ccs2h","openshift-image-registry/node-ca-7dpl6"] Jul 13 21:59:40.827613 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.827593 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.829610 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.829587 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" Jul 13 21:59:40.830197 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.830177 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 13 21:59:40.830596 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.830524 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nmw8g\"" Jul 13 21:59:40.830681 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.830530 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 13 21:59:40.831170 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.831151 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 13 21:59:40.831255 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.831167 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 13 21:59:40.831255 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.831177 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 13 21:59:40.831478 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.831464 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 13 21:59:40.831591 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.831575 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 13 21:59:40.831763 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.831749 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 13 21:59:40.832052 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.831920 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w85hd\"" Jul 13 21:59:40.832052 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.832002 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-29z8f" Jul 13 21:59:40.832182 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.832147 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 13 21:59:40.834044 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.834029 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.835671 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.835648 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 13 21:59:40.836331 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.836315 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 21:59:40.836412 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:40.836375 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nbdz9" podUID="71a3685e-3328-41cb-8b35-c8e8e1efe021" Jul 13 21:59:40.839226 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.838513 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lvr8p" Jul 13 21:59:40.842404 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.841025 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 13 21:59:40.842404 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.841212 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 13 21:59:40.842404 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.841310 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hx8pb\"" Jul 13 21:59:40.842404 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.841546 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 13 21:59:40.842404 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.841679 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 13 21:59:40.842404 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.841773 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 13 21:59:40.842404 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.841804 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 13 21:59:40.842404 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.841882 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 13 21:59:40.842404 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.841965 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jl6dz\"" Jul 13 21:59:40.842404 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.842061 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 13 21:59:40.842404 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.842281 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 13 21:59:40.842893 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.842531 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:40.843010 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.842967 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2xbj7\"" Jul 13 21:59:40.844948 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.844929 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 13 21:59:40.845282 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.845265 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-b976d\"" Jul 13 21:59:40.845624 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.845605 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-run-systemd\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.845688 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.845632 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-run-openvswitch\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.845688 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.845655 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-host-cni-netd\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.845688 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.845675 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/582b8d35-ee74-490e-aad6-e89ac36cb61c-env-overrides\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.845819 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.845697 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4bwt\" (UniqueName: \"kubernetes.io/projected/582b8d35-ee74-490e-aad6-e89ac36cb61c-kube-api-access-w4bwt\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.845819 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.845732 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1700b208-5391-4bed-8255-1594e9b3eac5-socket-dir\") pod \"aws-ebs-csi-driver-node-n2llx\" (UID: \"1700b208-5391-4bed-8255-1594e9b3eac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" Jul 13 21:59:40.845819 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.845754 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-system-cni-dir\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.845819 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.845780 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-multus-cni-dir\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.845819 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.845804 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-cnibin\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.846071 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.845827 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-host-run-netns\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.846071 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.845850 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-node-log\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.846071 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.845871 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-host-cni-bin\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.846071 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.845900 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-hostroot\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.846071 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.845922 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-multus-conf-dir\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.846071 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.845946 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-host-run-multus-certs\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.846071 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.846012 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-host-kubelet\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.846071 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.846037 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-run-ovn\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.846332 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.846071 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/582b8d35-ee74-490e-aad6-e89ac36cb61c-ovn-node-metrics-cert\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.846332 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.846097 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1700b208-5391-4bed-8255-1594e9b3eac5-device-dir\") pod \"aws-ebs-csi-driver-node-n2llx\" (UID: \"1700b208-5391-4bed-8255-1594e9b3eac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" Jul 13 21:59:40.846332 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.846118 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-systemd-units\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.846332 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.846128 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 13 21:59:40.846332 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.846138 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-host-run-ovn-kubernetes\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.846332 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.846160 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.846332 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.846183 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/582b8d35-ee74-490e-aad6-e89ac36cb61c-ovnkube-config\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.846332 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.846204 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/582b8d35-ee74-490e-aad6-e89ac36cb61c-ovnkube-script-lib\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.846332 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.846226 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1700b208-5391-4bed-8255-1594e9b3eac5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n2llx\" (UID: \"1700b208-5391-4bed-8255-1594e9b3eac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" Jul 13 21:59:40.846636 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.846360 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b79a451-a145-4116-8834-29040f299b3f-host-slash\") pod \"iptables-alerter-29z8f\" (UID: \"8b79a451-a145-4116-8834-29040f299b3f\") " pod="openshift-network-operator/iptables-alerter-29z8f" Jul 13 21:59:40.846636 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.846412 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-host-slash\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.846636 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.846450 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-var-lib-openvswitch\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.846636 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.846473 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1700b208-5391-4bed-8255-1594e9b3eac5-registration-dir\") pod \"aws-ebs-csi-driver-node-n2llx\" (UID: \"1700b208-5391-4bed-8255-1594e9b3eac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" Jul 13 21:59:40.846636 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.846498 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1700b208-5391-4bed-8255-1594e9b3eac5-sys-fs\") pod \"aws-ebs-csi-driver-node-n2llx\" (UID: \"1700b208-5391-4bed-8255-1594e9b3eac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" Jul 13 21:59:40.846636 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.846527 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs799\" (UniqueName: \"kubernetes.io/projected/8b79a451-a145-4116-8834-29040f299b3f-kube-api-access-xs799\") pod \"iptables-alerter-29z8f\" (UID: \"8b79a451-a145-4116-8834-29040f299b3f\") " pod="openshift-network-operator/iptables-alerter-29z8f" Jul 13 21:59:40.846636 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.846550 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6be816ca-053d-4a86-b5e8-755c240303cd-cni-binary-copy\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.846636 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.846581 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-multus-socket-dir-parent\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.846636 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.846619 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-host-var-lib-cni-multus\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.847043 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.846644 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-etc-openvswitch\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.847043 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.846676 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-host-run-k8s-cni-cncf-io\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.847043 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.846720 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-host-var-lib-cni-bin\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.847043 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.846791 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6be816ca-053d-4a86-b5e8-755c240303cd-multus-daemon-config\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.847043 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.846929 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 21:59:40.847043 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.846954 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-etc-kubernetes\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.847043 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:40.847007 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zkrhd" podUID="09fdf756-6037-48d0-a284-719f7770ade3" Jul 13 21:59:40.847043 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.847012 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmsbn\" (UniqueName: \"kubernetes.io/projected/6be816ca-053d-4a86-b5e8-755c240303cd-kube-api-access-kmsbn\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.847390 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.847057 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5xql\" (UniqueName: \"kubernetes.io/projected/1700b208-5391-4bed-8255-1594e9b3eac5-kube-api-access-r5xql\") pod \"aws-ebs-csi-driver-node-n2llx\" (UID: \"1700b208-5391-4bed-8255-1594e9b3eac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" Jul 13 21:59:40.847390 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.847085 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b79a451-a145-4116-8834-29040f299b3f-iptables-alerter-script\") pod \"iptables-alerter-29z8f\" (UID: \"8b79a451-a145-4116-8834-29040f299b3f\") " pod="openshift-network-operator/iptables-alerter-29z8f" Jul 13 21:59:40.847390 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.847106 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-os-release\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.847390 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.847127 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-host-run-netns\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.847390 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.847147 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-host-var-lib-kubelet\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.847390 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.847166 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg6gq\" (UniqueName: \"kubernetes.io/projected/71a3685e-3328-41cb-8b35-c8e8e1efe021-kube-api-access-pg6gq\") pod \"network-check-target-nbdz9\" (UID: \"71a3685e-3328-41cb-8b35-c8e8e1efe021\") " pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 21:59:40.847390 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.847179 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1700b208-5391-4bed-8255-1594e9b3eac5-etc-selinux\") pod \"aws-ebs-csi-driver-node-n2llx\" (UID: \"1700b208-5391-4bed-8255-1594e9b3eac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" Jul 13 21:59:40.847390 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.847192 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-log-socket\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.849509 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.849239 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pz6k2" Jul 13 21:59:40.849646 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.849628 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 13 21:59:40.851134 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.851116 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-v44dd\"" Jul 13 21:59:40.851214 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.851200 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 13 21:59:40.851214 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.851208 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 13 21:59:40.851639 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.851625 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 21:59:40.851708 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:40.851676 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8fht" podUID="a48e3e9b-b015-44af-89b9-42c2977ae9a8" Jul 13 21:59:40.854113 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.853779 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ccs2h" Jul 13 21:59:40.856407 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.856392 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 13 21:59:40.856764 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.856749 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 13 21:59:40.857057 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.856749 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s4gsv\"" Jul 13 21:59:40.857589 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.857571 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7dpl6" Jul 13 21:59:40.859317 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.859300 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 13 21:59:40.859736 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.859719 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pxknk\"" Jul 13 21:59:40.859820 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.859723 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 13 21:59:40.859820 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.859727 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 13 21:59:40.883498 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.883476 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lldhb" Jul 13 21:59:40.895672 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.895652 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lldhb" Jul 13 21:59:40.936659 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.936631 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 13 21:59:40.947623 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.947583 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-hostroot\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.947789 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.947628 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-host-run-multus-certs\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.947789 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.947657 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/09fdf756-6037-48d0-a284-719f7770ade3-dbus\") pod \"global-pull-secret-syncer-zkrhd\" (UID: \"09fdf756-6037-48d0-a284-719f7770ade3\") " pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 21:59:40.947789 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.947700 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/75ac4b1c-ecb1-4550-84ab-7220e7445349-os-release\") pod \"multus-additional-cni-plugins-lvr8p\" (UID: \"75ac4b1c-ecb1-4550-84ab-7220e7445349\") " pod="openshift-multus/multus-additional-cni-plugins-lvr8p" Jul 13 21:59:40.947932 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.947773 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-hostroot\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.947932 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.947784 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-host-run-multus-certs\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.947932 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.947820 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75ac4b1c-ecb1-4550-84ab-7220e7445349-cni-binary-copy\") pod \"multus-additional-cni-plugins-lvr8p\" (UID: \"75ac4b1c-ecb1-4550-84ab-7220e7445349\") " pod="openshift-multus/multus-additional-cni-plugins-lvr8p" Jul 13 21:59:40.947932 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.947854 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-sys\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:40.947932 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.947890 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-host-kubelet\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.947932 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.947920 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-etc-sysconfig\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:40.948179 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.947959 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-host-kubelet\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.948179 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948046 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-etc-sysctl-d\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:40.948483 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948462 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-etc-sysctl-conf\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:40.948541 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948499 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-lib-modules\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:40.948541 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948518 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-host-var-lib-kubelet\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.948541 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948537 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1700b208-5391-4bed-8255-1594e9b3eac5-device-dir\") pod \"aws-ebs-csi-driver-node-n2llx\" (UID: \"1700b208-5391-4bed-8255-1594e9b3eac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" Jul 13 21:59:40.948665 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948564 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-systemd-units\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.948665 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948581 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-host-var-lib-kubelet\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.948665 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948588 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.948665 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948601 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1700b208-5391-4bed-8255-1594e9b3eac5-device-dir\") pod \"aws-ebs-csi-driver-node-n2llx\" (UID: \"1700b208-5391-4bed-8255-1594e9b3eac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" Jul 13 21:59:40.948665 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948618 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/582b8d35-ee74-490e-aad6-e89ac36cb61c-ovnkube-config\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.948665 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948629 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.948665 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948642 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/582b8d35-ee74-490e-aad6-e89ac36cb61c-ovnkube-script-lib\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.949014 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948666 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-systemd-units\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.949014 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948662 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b79a451-a145-4116-8834-29040f299b3f-host-slash\") pod \"iptables-alerter-29z8f\" (UID: \"8b79a451-a145-4116-8834-29040f299b3f\") " pod="openshift-network-operator/iptables-alerter-29z8f" Jul 13 21:59:40.949014 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948702 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b79a451-a145-4116-8834-29040f299b3f-host-slash\") pod \"iptables-alerter-29z8f\" (UID: \"8b79a451-a145-4116-8834-29040f299b3f\") " pod="openshift-network-operator/iptables-alerter-29z8f" Jul 13 21:59:40.949014 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948723 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-multus-socket-dir-parent\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.949014 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948746 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-host-slash\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.949014 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948767 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1700b208-5391-4bed-8255-1594e9b3eac5-registration-dir\") pod \"aws-ebs-csi-driver-node-n2llx\" (UID: \"1700b208-5391-4bed-8255-1594e9b3eac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" Jul 13 21:59:40.949014 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948800 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1700b208-5391-4bed-8255-1594e9b3eac5-sys-fs\") pod \"aws-ebs-csi-driver-node-n2llx\" (UID: \"1700b208-5391-4bed-8255-1594e9b3eac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" Jul 13 21:59:40.949014 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948809 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-host-slash\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.949014 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948825 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6be816ca-053d-4a86-b5e8-755c240303cd-cni-binary-copy\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.949014 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948834 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1700b208-5391-4bed-8255-1594e9b3eac5-registration-dir\") pod \"aws-ebs-csi-driver-node-n2llx\" (UID: \"1700b208-5391-4bed-8255-1594e9b3eac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" Jul 13 21:59:40.949014 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948855 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1700b208-5391-4bed-8255-1594e9b3eac5-sys-fs\") pod \"aws-ebs-csi-driver-node-n2llx\" (UID: \"1700b208-5391-4bed-8255-1594e9b3eac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" Jul 13 21:59:40.949014 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948852 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-host-var-lib-cni-multus\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.949014 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948888 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-etc-kubernetes\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.949014 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948913 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6phf\" (UniqueName: \"kubernetes.io/projected/348d04ea-a0a4-45da-ba4b-480288fd701b-kube-api-access-p6phf\") pod \"node-ca-7dpl6\" (UID: \"348d04ea-a0a4-45da-ba4b-480288fd701b\") " pod="openshift-image-registry/node-ca-7dpl6" Jul 13 21:59:40.949014 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948938 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/75ac4b1c-ecb1-4550-84ab-7220e7445349-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lvr8p\" (UID: \"75ac4b1c-ecb1-4550-84ab-7220e7445349\") " pod="openshift-multus/multus-additional-cni-plugins-lvr8p" Jul 13 21:59:40.949014 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948969 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-etc-openvswitch\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.949014 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948988 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-multus-socket-dir-parent\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.949707 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.948995 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-host-var-lib-cni-multus\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.949707 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949007 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqwsx\" (UniqueName: \"kubernetes.io/projected/75ac4b1c-ecb1-4550-84ab-7220e7445349-kube-api-access-rqwsx\") pod \"multus-additional-cni-plugins-lvr8p\" (UID: \"75ac4b1c-ecb1-4550-84ab-7220e7445349\") " pod="openshift-multus/multus-additional-cni-plugins-lvr8p" Jul 13 21:59:40.949707 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949035 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-etc-kubernetes\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.949707 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949048 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-os-release\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.949707 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949056 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-etc-openvswitch\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.949707 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949094 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75ac4b1c-ecb1-4550-84ab-7220e7445349-system-cni-dir\") pod \"multus-additional-cni-plugins-lvr8p\" (UID: \"75ac4b1c-ecb1-4550-84ab-7220e7445349\") " pod="openshift-multus/multus-additional-cni-plugins-lvr8p" Jul 13 21:59:40.949707 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949134 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-var-lib-kubelet\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:40.949707 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949146 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-os-release\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.949707 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949160 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f77cde0-c7bd-4803-8c7b-b5e4071ace68-tmp-dir\") pod \"node-resolver-ccs2h\" (UID: \"3f77cde0-c7bd-4803-8c7b-b5e4071ace68\") " pod="openshift-dns/node-resolver-ccs2h" Jul 13 21:59:40.949707 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949185 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1700b208-5391-4bed-8255-1594e9b3eac5-etc-selinux\") pod \"aws-ebs-csi-driver-node-n2llx\" (UID: \"1700b208-5391-4bed-8255-1594e9b3eac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" Jul 13 21:59:40.949707 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949205 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-log-socket\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.949707 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949228 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-run-systemd\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.949707 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949218 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/582b8d35-ee74-490e-aad6-e89ac36cb61c-ovnkube-config\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.949707 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949246 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-log-socket\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.949707 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949253 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-run-openvswitch\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.949707 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949275 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/582b8d35-ee74-490e-aad6-e89ac36cb61c-env-overrides\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.949707 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949276 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-run-systemd\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.950345 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949280 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/582b8d35-ee74-490e-aad6-e89ac36cb61c-ovnkube-script-lib\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.950345 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949287 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1700b208-5391-4bed-8255-1594e9b3eac5-etc-selinux\") pod \"aws-ebs-csi-driver-node-n2llx\" (UID: \"1700b208-5391-4bed-8255-1594e9b3eac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" Jul 13 21:59:40.950345 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949297 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4bwt\" (UniqueName: \"kubernetes.io/projected/582b8d35-ee74-490e-aad6-e89ac36cb61c-kube-api-access-w4bwt\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.950345 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949297 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-run-openvswitch\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.950345 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949333 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-multus-cni-dir\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.950345 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949355 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6be816ca-053d-4a86-b5e8-755c240303cd-cni-binary-copy\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.950345 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949366 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-cnibin\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.950345 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949397 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a48e3e9b-b015-44af-89b9-42c2977ae9a8-metrics-certs\") pod \"network-metrics-daemon-w8fht\" (UID: \"a48e3e9b-b015-44af-89b9-42c2977ae9a8\") " pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 21:59:40.950345 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949418 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/348d04ea-a0a4-45da-ba4b-480288fd701b-host\") pod \"node-ca-7dpl6\" (UID: \"348d04ea-a0a4-45da-ba4b-480288fd701b\") " pod="openshift-image-registry/node-ca-7dpl6" Jul 13 21:59:40.950345 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949438 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-run\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:40.950345 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949455 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-cnibin\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.950345 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949459 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/64c6ea85-17e8-4f97-b043-5da490a75040-konnectivity-ca\") pod \"konnectivity-agent-pz6k2\" (UID: \"64c6ea85-17e8-4f97-b043-5da490a75040\") " pod="kube-system/konnectivity-agent-pz6k2" Jul 13 21:59:40.950345 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949490 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-node-log\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.950345 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949502 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-multus-cni-dir\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.950345 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949515 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-host-cni-bin\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.950345 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949537 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-node-log\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.950345 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949578 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-multus-conf-dir\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.950345 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949605 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75ac4b1c-ecb1-4550-84ab-7220e7445349-cnibin\") pod \"multus-additional-cni-plugins-lvr8p\" (UID: \"75ac4b1c-ecb1-4550-84ab-7220e7445349\") " pod="openshift-multus/multus-additional-cni-plugins-lvr8p" Jul 13 21:59:40.951153 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949626 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a0952063-5741-4b6c-9a72-28e3efcf9eaf-etc-tuned\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:40.951153 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949582 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-host-cni-bin\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.951153 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949645 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-multus-conf-dir\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.951153 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949647 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-run-ovn\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.951153 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949684 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-run-ovn\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.951153 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949684 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/582b8d35-ee74-490e-aad6-e89ac36cb61c-ovn-node-metrics-cert\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.951153 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949714 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/582b8d35-ee74-490e-aad6-e89ac36cb61c-env-overrides\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.951153 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949719 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg8gz\" (UniqueName: \"kubernetes.io/projected/a48e3e9b-b015-44af-89b9-42c2977ae9a8-kube-api-access-wg8gz\") pod \"network-metrics-daemon-w8fht\" (UID: \"a48e3e9b-b015-44af-89b9-42c2977ae9a8\") " pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 21:59:40.951153 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949765 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/348d04ea-a0a4-45da-ba4b-480288fd701b-serviceca\") pod \"node-ca-7dpl6\" (UID: \"348d04ea-a0a4-45da-ba4b-480288fd701b\") " pod="openshift-image-registry/node-ca-7dpl6" Jul 13 21:59:40.951153 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949791 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/75ac4b1c-ecb1-4550-84ab-7220e7445349-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lvr8p\" (UID: \"75ac4b1c-ecb1-4550-84ab-7220e7445349\") " pod="openshift-multus/multus-additional-cni-plugins-lvr8p" Jul 13 21:59:40.951153 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949809 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x5wz\" (UniqueName: \"kubernetes.io/projected/3f77cde0-c7bd-4803-8c7b-b5e4071ace68-kube-api-access-8x5wz\") pod \"node-resolver-ccs2h\" (UID: \"3f77cde0-c7bd-4803-8c7b-b5e4071ace68\") " pod="openshift-dns/node-resolver-ccs2h" Jul 13 21:59:40.951153 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949828 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-host-run-ovn-kubernetes\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.951153 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949845 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1700b208-5391-4bed-8255-1594e9b3eac5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n2llx\" (UID: \"1700b208-5391-4bed-8255-1594e9b3eac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" Jul 13 21:59:40.951153 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949871 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-etc-modprobe-d\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:40.951153 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949891 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-host\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:40.951153 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949903 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-host-run-ovn-kubernetes\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.951153 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949924 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxv4t\" (UniqueName: \"kubernetes.io/projected/a0952063-5741-4b6c-9a72-28e3efcf9eaf-kube-api-access-nxv4t\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:40.951729 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949947 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3f77cde0-c7bd-4803-8c7b-b5e4071ace68-hosts-file\") pod \"node-resolver-ccs2h\" (UID: \"3f77cde0-c7bd-4803-8c7b-b5e4071ace68\") " pod="openshift-dns/node-resolver-ccs2h" Jul 13 21:59:40.951729 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949951 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1700b208-5391-4bed-8255-1594e9b3eac5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n2llx\" (UID: \"1700b208-5391-4bed-8255-1594e9b3eac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" Jul 13 21:59:40.951729 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949966 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-var-lib-openvswitch\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.951729 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.949989 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 13 21:59:40.951729 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950014 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs799\" (UniqueName: \"kubernetes.io/projected/8b79a451-a145-4116-8834-29040f299b3f-kube-api-access-xs799\") pod \"iptables-alerter-29z8f\" (UID: \"8b79a451-a145-4116-8834-29040f299b3f\") " pod="openshift-network-operator/iptables-alerter-29z8f" Jul 13 21:59:40.951729 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950028 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmsbn\" (UniqueName: \"kubernetes.io/projected/6be816ca-053d-4a86-b5e8-755c240303cd-kube-api-access-kmsbn\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.951729 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950037 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-var-lib-openvswitch\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.951729 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950047 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75ac4b1c-ecb1-4550-84ab-7220e7445349-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lvr8p\" (UID: \"75ac4b1c-ecb1-4550-84ab-7220e7445349\") " pod="openshift-multus/multus-additional-cni-plugins-lvr8p" Jul 13 21:59:40.951729 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950068 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0952063-5741-4b6c-9a72-28e3efcf9eaf-tmp\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:40.951729 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950086 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/64c6ea85-17e8-4f97-b043-5da490a75040-agent-certs\") pod \"konnectivity-agent-pz6k2\" (UID: \"64c6ea85-17e8-4f97-b043-5da490a75040\") " pod="kube-system/konnectivity-agent-pz6k2" Jul 13 21:59:40.951729 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950109 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-host-run-k8s-cni-cncf-io\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.951729 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950129 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-host-var-lib-cni-bin\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.951729 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950152 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6be816ca-053d-4a86-b5e8-755c240303cd-multus-daemon-config\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.951729 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950166 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5xql\" (UniqueName: \"kubernetes.io/projected/1700b208-5391-4bed-8255-1594e9b3eac5-kube-api-access-r5xql\") pod \"aws-ebs-csi-driver-node-n2llx\" (UID: \"1700b208-5391-4bed-8255-1594e9b3eac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" Jul 13 21:59:40.951729 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950187 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b79a451-a145-4116-8834-29040f299b3f-iptables-alerter-script\") pod \"iptables-alerter-29z8f\" (UID: \"8b79a451-a145-4116-8834-29040f299b3f\") " pod="openshift-network-operator/iptables-alerter-29z8f" Jul 13 21:59:40.951729 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950209 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-host-run-netns\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.951729 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950231 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg6gq\" (UniqueName: \"kubernetes.io/projected/71a3685e-3328-41cb-8b35-c8e8e1efe021-kube-api-access-pg6gq\") pod \"network-check-target-nbdz9\" (UID: \"71a3685e-3328-41cb-8b35-c8e8e1efe021\") " pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 21:59:40.952364 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950272 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-host-run-k8s-cni-cncf-io\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.952364 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950322 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-host-var-lib-cni-bin\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.952364 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950501 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-host-run-netns\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.952364 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950554 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-etc-kubernetes\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:40.952364 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950596 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-host-cni-netd\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.952364 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950618 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-etc-systemd\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:40.952364 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950638 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1700b208-5391-4bed-8255-1594e9b3eac5-socket-dir\") pod \"aws-ebs-csi-driver-node-n2llx\" (UID: \"1700b208-5391-4bed-8255-1594e9b3eac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" Jul 13 21:59:40.952364 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950658 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-system-cni-dir\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.952364 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950676 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/09fdf756-6037-48d0-a284-719f7770ade3-kubelet-config\") pod \"global-pull-secret-syncer-zkrhd\" (UID: \"09fdf756-6037-48d0-a284-719f7770ade3\") " pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 21:59:40.952364 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950691 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09fdf756-6037-48d0-a284-719f7770ade3-original-pull-secret\") pod \"global-pull-secret-syncer-zkrhd\" (UID: \"09fdf756-6037-48d0-a284-719f7770ade3\") " pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 21:59:40.952364 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950705 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-host-cni-netd\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.952364 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950710 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-host-run-netns\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.952364 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950737 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/582b8d35-ee74-490e-aad6-e89ac36cb61c-host-run-netns\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.952364 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950737 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6be816ca-053d-4a86-b5e8-755c240303cd-multus-daemon-config\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.952364 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950759 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b79a451-a145-4116-8834-29040f299b3f-iptables-alerter-script\") pod \"iptables-alerter-29z8f\" (UID: \"8b79a451-a145-4116-8834-29040f299b3f\") " pod="openshift-network-operator/iptables-alerter-29z8f" Jul 13 21:59:40.952364 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950811 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6be816ca-053d-4a86-b5e8-755c240303cd-system-cni-dir\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.952364 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.950849 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1700b208-5391-4bed-8255-1594e9b3eac5-socket-dir\") pod \"aws-ebs-csi-driver-node-n2llx\" (UID: \"1700b208-5391-4bed-8255-1594e9b3eac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" Jul 13 21:59:40.953267 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.953251 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/582b8d35-ee74-490e-aad6-e89ac36cb61c-ovn-node-metrics-cert\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.955840 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:40.955820 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 21:59:40.955840 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:40.955850 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 21:59:40.956005 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:40.955862 2562 projected.go:196] Error preparing data for projected volume kube-api-access-pg6gq for pod openshift-network-diagnostics/network-check-target-nbdz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:59:40.956005 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:40.955925 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71a3685e-3328-41cb-8b35-c8e8e1efe021-kube-api-access-pg6gq podName:71a3685e-3328-41cb-8b35-c8e8e1efe021 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:41.455898938 +0000 UTC m=+2.906454470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pg6gq" (UniqueName: "kubernetes.io/projected/71a3685e-3328-41cb-8b35-c8e8e1efe021-kube-api-access-pg6gq") pod "network-check-target-nbdz9" (UID: "71a3685e-3328-41cb-8b35-c8e8e1efe021") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:59:40.957261 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.957238 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4bwt\" (UniqueName: \"kubernetes.io/projected/582b8d35-ee74-490e-aad6-e89ac36cb61c-kube-api-access-w4bwt\") pod \"ovnkube-node-zw4h7\" (UID: \"582b8d35-ee74-490e-aad6-e89ac36cb61c\") " pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:40.957722 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.957698 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5xql\" (UniqueName: \"kubernetes.io/projected/1700b208-5391-4bed-8255-1594e9b3eac5-kube-api-access-r5xql\") pod \"aws-ebs-csi-driver-node-n2llx\" (UID: \"1700b208-5391-4bed-8255-1594e9b3eac5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" Jul 13 21:59:40.958118 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.958099 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmsbn\" (UniqueName: \"kubernetes.io/projected/6be816ca-053d-4a86-b5e8-755c240303cd-kube-api-access-kmsbn\") pod \"multus-svxd6\" (UID: \"6be816ca-053d-4a86-b5e8-755c240303cd\") " pod="openshift-multus/multus-svxd6" Jul 13 21:59:40.958713 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:40.958693 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs799\" (UniqueName: \"kubernetes.io/projected/8b79a451-a145-4116-8834-29040f299b3f-kube-api-access-xs799\") pod \"iptables-alerter-29z8f\" (UID: \"8b79a451-a145-4116-8834-29040f299b3f\") " pod="openshift-network-operator/iptables-alerter-29z8f" Jul 13 21:59:41.051704 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.051672 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75ac4b1c-ecb1-4550-84ab-7220e7445349-system-cni-dir\") pod \"multus-additional-cni-plugins-lvr8p\" (UID: \"75ac4b1c-ecb1-4550-84ab-7220e7445349\") " pod="openshift-multus/multus-additional-cni-plugins-lvr8p" Jul 13 21:59:41.051876 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.051712 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-var-lib-kubelet\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.051876 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.051734 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f77cde0-c7bd-4803-8c7b-b5e4071ace68-tmp-dir\") pod \"node-resolver-ccs2h\" (UID: \"3f77cde0-c7bd-4803-8c7b-b5e4071ace68\") " pod="openshift-dns/node-resolver-ccs2h" Jul 13 21:59:41.051876 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.051773 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a48e3e9b-b015-44af-89b9-42c2977ae9a8-metrics-certs\") pod \"network-metrics-daemon-w8fht\" (UID: \"a48e3e9b-b015-44af-89b9-42c2977ae9a8\") " pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 21:59:41.051876 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.051792 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/348d04ea-a0a4-45da-ba4b-480288fd701b-host\") pod \"node-ca-7dpl6\" (UID: \"348d04ea-a0a4-45da-ba4b-480288fd701b\") " pod="openshift-image-registry/node-ca-7dpl6" Jul 13 21:59:41.051876 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.051815 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-run\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.051876 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.051830 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-var-lib-kubelet\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.051876 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.051835 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/64c6ea85-17e8-4f97-b043-5da490a75040-konnectivity-ca\") pod \"konnectivity-agent-pz6k2\" (UID: \"64c6ea85-17e8-4f97-b043-5da490a75040\") " pod="kube-system/konnectivity-agent-pz6k2" Jul 13 21:59:41.052212 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.051896 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75ac4b1c-ecb1-4550-84ab-7220e7445349-cnibin\") pod \"multus-additional-cni-plugins-lvr8p\" (UID: \"75ac4b1c-ecb1-4550-84ab-7220e7445349\") " pod="openshift-multus/multus-additional-cni-plugins-lvr8p" Jul 13 21:59:41.052212 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:41.051912 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:59:41.052212 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.051921 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a0952063-5741-4b6c-9a72-28e3efcf9eaf-etc-tuned\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.052212 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.051930 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/348d04ea-a0a4-45da-ba4b-480288fd701b-host\") pod \"node-ca-7dpl6\" (UID: \"348d04ea-a0a4-45da-ba4b-480288fd701b\") " pod="openshift-image-registry/node-ca-7dpl6" Jul 13 21:59:41.052212 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.051948 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg8gz\" (UniqueName: \"kubernetes.io/projected/a48e3e9b-b015-44af-89b9-42c2977ae9a8-kube-api-access-wg8gz\") pod \"network-metrics-daemon-w8fht\" (UID: \"a48e3e9b-b015-44af-89b9-42c2977ae9a8\") " pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 21:59:41.052212 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:41.052009 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a48e3e9b-b015-44af-89b9-42c2977ae9a8-metrics-certs podName:a48e3e9b-b015-44af-89b9-42c2977ae9a8 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:41.551987659 +0000 UTC m=+3.002543197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a48e3e9b-b015-44af-89b9-42c2977ae9a8-metrics-certs") pod "network-metrics-daemon-w8fht" (UID: "a48e3e9b-b015-44af-89b9-42c2977ae9a8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:59:41.052212 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052040 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/348d04ea-a0a4-45da-ba4b-480288fd701b-serviceca\") pod \"node-ca-7dpl6\" (UID: \"348d04ea-a0a4-45da-ba4b-480288fd701b\") " pod="openshift-image-registry/node-ca-7dpl6" Jul 13 21:59:41.052212 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052045 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-run\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.052212 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052068 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/75ac4b1c-ecb1-4550-84ab-7220e7445349-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lvr8p\" (UID: \"75ac4b1c-ecb1-4550-84ab-7220e7445349\") " pod="openshift-multus/multus-additional-cni-plugins-lvr8p" Jul 13 21:59:41.052212 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052097 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75ac4b1c-ecb1-4550-84ab-7220e7445349-cnibin\") pod \"multus-additional-cni-plugins-lvr8p\" (UID: \"75ac4b1c-ecb1-4550-84ab-7220e7445349\") " pod="openshift-multus/multus-additional-cni-plugins-lvr8p" Jul 13 21:59:41.052212 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052096 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x5wz\" (UniqueName: \"kubernetes.io/projected/3f77cde0-c7bd-4803-8c7b-b5e4071ace68-kube-api-access-8x5wz\") pod \"node-resolver-ccs2h\" (UID: \"3f77cde0-c7bd-4803-8c7b-b5e4071ace68\") " pod="openshift-dns/node-resolver-ccs2h" Jul 13 21:59:41.052212 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052127 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-etc-modprobe-d\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.052212 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052150 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-host\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.052212 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052171 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxv4t\" (UniqueName: \"kubernetes.io/projected/a0952063-5741-4b6c-9a72-28e3efcf9eaf-kube-api-access-nxv4t\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.052212 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052187 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3f77cde0-c7bd-4803-8c7b-b5e4071ace68-hosts-file\") pod \"node-resolver-ccs2h\" (UID: \"3f77cde0-c7bd-4803-8c7b-b5e4071ace68\") " pod="openshift-dns/node-resolver-ccs2h" Jul 13 21:59:41.052212 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052205 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75ac4b1c-ecb1-4550-84ab-7220e7445349-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lvr8p\" (UID: \"75ac4b1c-ecb1-4550-84ab-7220e7445349\") " pod="openshift-multus/multus-additional-cni-plugins-lvr8p" Jul 13 21:59:41.052212 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052226 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0952063-5741-4b6c-9a72-28e3efcf9eaf-tmp\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.053029 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052241 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/64c6ea85-17e8-4f97-b043-5da490a75040-agent-certs\") pod \"konnectivity-agent-pz6k2\" (UID: \"64c6ea85-17e8-4f97-b043-5da490a75040\") " pod="kube-system/konnectivity-agent-pz6k2" Jul 13 21:59:41.053029 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052268 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-etc-kubernetes\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.053029 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052280 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f77cde0-c7bd-4803-8c7b-b5e4071ace68-tmp-dir\") pod \"node-resolver-ccs2h\" (UID: \"3f77cde0-c7bd-4803-8c7b-b5e4071ace68\") " pod="openshift-dns/node-resolver-ccs2h" Jul 13 21:59:41.053029 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052291 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-etc-systemd\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.053029 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052333 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/09fdf756-6037-48d0-a284-719f7770ade3-kubelet-config\") pod \"global-pull-secret-syncer-zkrhd\" (UID: \"09fdf756-6037-48d0-a284-719f7770ade3\") " pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 21:59:41.053029 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052374 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3f77cde0-c7bd-4803-8c7b-b5e4071ace68-hosts-file\") pod \"node-resolver-ccs2h\" (UID: \"3f77cde0-c7bd-4803-8c7b-b5e4071ace68\") " pod="openshift-dns/node-resolver-ccs2h" Jul 13 21:59:41.053029 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052393 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-etc-modprobe-d\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.053029 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052404 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09fdf756-6037-48d0-a284-719f7770ade3-original-pull-secret\") pod \"global-pull-secret-syncer-zkrhd\" (UID: \"09fdf756-6037-48d0-a284-719f7770ade3\") " pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 21:59:41.053029 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052432 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/09fdf756-6037-48d0-a284-719f7770ade3-dbus\") pod \"global-pull-secret-syncer-zkrhd\" (UID: \"09fdf756-6037-48d0-a284-719f7770ade3\") " pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 21:59:41.053029 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052437 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-host\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.053029 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052459 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/75ac4b1c-ecb1-4550-84ab-7220e7445349-os-release\") pod \"multus-additional-cni-plugins-lvr8p\" (UID: \"75ac4b1c-ecb1-4550-84ab-7220e7445349\") " pod="openshift-multus/multus-additional-cni-plugins-lvr8p" Jul 13 21:59:41.053029 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052483 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75ac4b1c-ecb1-4550-84ab-7220e7445349-cni-binary-copy\") pod \"multus-additional-cni-plugins-lvr8p\" (UID: \"75ac4b1c-ecb1-4550-84ab-7220e7445349\") " pod="openshift-multus/multus-additional-cni-plugins-lvr8p" Jul 13 21:59:41.053029 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052506 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-sys\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.053029 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052529 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-etc-sysconfig\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.053029 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052555 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-etc-sysctl-d\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.053029 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052571 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75ac4b1c-ecb1-4550-84ab-7220e7445349-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lvr8p\" (UID: \"75ac4b1c-ecb1-4550-84ab-7220e7445349\") " pod="openshift-multus/multus-additional-cni-plugins-lvr8p" Jul 13 21:59:41.053029 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052590 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-etc-sysctl-conf\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.053786 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052642 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-etc-kubernetes\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.053786 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052634 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-lib-modules\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.053786 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052707 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6phf\" (UniqueName: \"kubernetes.io/projected/348d04ea-a0a4-45da-ba4b-480288fd701b-kube-api-access-p6phf\") pod \"node-ca-7dpl6\" (UID: \"348d04ea-a0a4-45da-ba4b-480288fd701b\") " pod="openshift-image-registry/node-ca-7dpl6" Jul 13 21:59:41.053786 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052716 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/09fdf756-6037-48d0-a284-719f7770ade3-dbus\") pod \"global-pull-secret-syncer-zkrhd\" (UID: \"09fdf756-6037-48d0-a284-719f7770ade3\") " pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 21:59:41.053786 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052431 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/64c6ea85-17e8-4f97-b043-5da490a75040-konnectivity-ca\") pod \"konnectivity-agent-pz6k2\" (UID: \"64c6ea85-17e8-4f97-b043-5da490a75040\") " pod="kube-system/konnectivity-agent-pz6k2" Jul 13 21:59:41.053786 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052757 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-lib-modules\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.053786 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052773 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/75ac4b1c-ecb1-4550-84ab-7220e7445349-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lvr8p\" (UID: \"75ac4b1c-ecb1-4550-84ab-7220e7445349\") " pod="openshift-multus/multus-additional-cni-plugins-lvr8p" Jul 13 21:59:41.053786 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052797 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-sys\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.053786 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052808 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-etc-sysconfig\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.053786 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052801 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqwsx\" (UniqueName: \"kubernetes.io/projected/75ac4b1c-ecb1-4550-84ab-7220e7445349-kube-api-access-rqwsx\") pod \"multus-additional-cni-plugins-lvr8p\" (UID: \"75ac4b1c-ecb1-4550-84ab-7220e7445349\") " pod="openshift-multus/multus-additional-cni-plugins-lvr8p" Jul 13 21:59:41.053786 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052010 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75ac4b1c-ecb1-4550-84ab-7220e7445349-system-cni-dir\") pod \"multus-additional-cni-plugins-lvr8p\" (UID: \"75ac4b1c-ecb1-4550-84ab-7220e7445349\") " pod="openshift-multus/multus-additional-cni-plugins-lvr8p" Jul 13 21:59:41.053786 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052858 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-etc-systemd\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.053786 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052861 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/75ac4b1c-ecb1-4550-84ab-7220e7445349-os-release\") pod \"multus-additional-cni-plugins-lvr8p\" (UID: \"75ac4b1c-ecb1-4550-84ab-7220e7445349\") " pod="openshift-multus/multus-additional-cni-plugins-lvr8p" Jul 13 21:59:41.053786 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052853 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/75ac4b1c-ecb1-4550-84ab-7220e7445349-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lvr8p\" (UID: \"75ac4b1c-ecb1-4550-84ab-7220e7445349\") " pod="openshift-multus/multus-additional-cni-plugins-lvr8p" Jul 13 21:59:41.053786 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052391 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/09fdf756-6037-48d0-a284-719f7770ade3-kubelet-config\") pod \"global-pull-secret-syncer-zkrhd\" (UID: \"09fdf756-6037-48d0-a284-719f7770ade3\") " pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 21:59:41.053786 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:41.052956 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 21:59:41.053786 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:41.053035 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09fdf756-6037-48d0-a284-719f7770ade3-original-pull-secret podName:09fdf756-6037-48d0-a284-719f7770ade3 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:41.553018542 +0000 UTC m=+3.003574058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09fdf756-6037-48d0-a284-719f7770ade3-original-pull-secret") pod "global-pull-secret-syncer-zkrhd" (UID: "09fdf756-6037-48d0-a284-719f7770ade3") : object "kube-system"/"original-pull-secret" not registered Jul 13 21:59:41.054577 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.052957 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-etc-sysctl-d\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.054577 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.053083 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a0952063-5741-4b6c-9a72-28e3efcf9eaf-etc-sysctl-conf\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.054577 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.053335 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/348d04ea-a0a4-45da-ba4b-480288fd701b-serviceca\") pod \"node-ca-7dpl6\" (UID: \"348d04ea-a0a4-45da-ba4b-480288fd701b\") " pod="openshift-image-registry/node-ca-7dpl6" Jul 13 21:59:41.054577 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.053534 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/75ac4b1c-ecb1-4550-84ab-7220e7445349-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lvr8p\" (UID: \"75ac4b1c-ecb1-4550-84ab-7220e7445349\") " pod="openshift-multus/multus-additional-cni-plugins-lvr8p" Jul 13 21:59:41.054577 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.053607 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75ac4b1c-ecb1-4550-84ab-7220e7445349-cni-binary-copy\") pod \"multus-additional-cni-plugins-lvr8p\" (UID: \"75ac4b1c-ecb1-4550-84ab-7220e7445349\") " pod="openshift-multus/multus-additional-cni-plugins-lvr8p" Jul 13 21:59:41.054577 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.054575 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a0952063-5741-4b6c-9a72-28e3efcf9eaf-etc-tuned\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.054865 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.054843 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0952063-5741-4b6c-9a72-28e3efcf9eaf-tmp\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.055353 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.055335 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/64c6ea85-17e8-4f97-b043-5da490a75040-agent-certs\") pod \"konnectivity-agent-pz6k2\" (UID: \"64c6ea85-17e8-4f97-b043-5da490a75040\") " pod="kube-system/konnectivity-agent-pz6k2" Jul 13 21:59:41.061566 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.061545 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x5wz\" (UniqueName: \"kubernetes.io/projected/3f77cde0-c7bd-4803-8c7b-b5e4071ace68-kube-api-access-8x5wz\") pod \"node-resolver-ccs2h\" (UID: \"3f77cde0-c7bd-4803-8c7b-b5e4071ace68\") " pod="openshift-dns/node-resolver-ccs2h" Jul 13 21:59:41.062386 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.062345 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6phf\" (UniqueName: \"kubernetes.io/projected/348d04ea-a0a4-45da-ba4b-480288fd701b-kube-api-access-p6phf\") pod \"node-ca-7dpl6\" (UID: \"348d04ea-a0a4-45da-ba4b-480288fd701b\") " pod="openshift-image-registry/node-ca-7dpl6" Jul 13 21:59:41.062706 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.062684 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxv4t\" (UniqueName: \"kubernetes.io/projected/a0952063-5741-4b6c-9a72-28e3efcf9eaf-kube-api-access-nxv4t\") pod \"tuned-nshp2\" (UID: \"a0952063-5741-4b6c-9a72-28e3efcf9eaf\") " pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.062906 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.062891 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqwsx\" (UniqueName: \"kubernetes.io/projected/75ac4b1c-ecb1-4550-84ab-7220e7445349-kube-api-access-rqwsx\") pod \"multus-additional-cni-plugins-lvr8p\" (UID: \"75ac4b1c-ecb1-4550-84ab-7220e7445349\") " pod="openshift-multus/multus-additional-cni-plugins-lvr8p" Jul 13 21:59:41.063157 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.063135 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg8gz\" (UniqueName: \"kubernetes.io/projected/a48e3e9b-b015-44af-89b9-42c2977ae9a8-kube-api-access-wg8gz\") pod \"network-metrics-daemon-w8fht\" (UID: \"a48e3e9b-b015-44af-89b9-42c2977ae9a8\") " pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 21:59:41.139261 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.139189 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 21:59:41.149020 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.148999 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" Jul 13 21:59:41.158577 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.158561 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-29z8f" Jul 13 21:59:41.166031 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.166006 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-svxd6" Jul 13 21:59:41.171452 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.171435 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lvr8p" Jul 13 21:59:41.177986 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.177955 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nshp2" Jul 13 21:59:41.186083 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.186067 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pz6k2" Jul 13 21:59:41.192397 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.192381 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ccs2h" Jul 13 21:59:41.199665 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.199650 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7dpl6" Jul 13 21:59:41.456763 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.456688 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg6gq\" (UniqueName: \"kubernetes.io/projected/71a3685e-3328-41cb-8b35-c8e8e1efe021-kube-api-access-pg6gq\") pod \"network-check-target-nbdz9\" (UID: \"71a3685e-3328-41cb-8b35-c8e8e1efe021\") " pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 21:59:41.456905 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:41.456872 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 21:59:41.456905 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:41.456889 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 21:59:41.456905 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:41.456899 2562 projected.go:196] Error preparing data for projected volume kube-api-access-pg6gq for pod openshift-network-diagnostics/network-check-target-nbdz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:59:41.457069 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:41.456987 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71a3685e-3328-41cb-8b35-c8e8e1efe021-kube-api-access-pg6gq podName:71a3685e-3328-41cb-8b35-c8e8e1efe021 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:42.456953774 +0000 UTC m=+3.907509307 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pg6gq" (UniqueName: "kubernetes.io/projected/71a3685e-3328-41cb-8b35-c8e8e1efe021-kube-api-access-pg6gq") pod "network-check-target-nbdz9" (UID: "71a3685e-3328-41cb-8b35-c8e8e1efe021") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:59:41.557751 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.557716 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09fdf756-6037-48d0-a284-719f7770ade3-original-pull-secret\") pod \"global-pull-secret-syncer-zkrhd\" (UID: \"09fdf756-6037-48d0-a284-719f7770ade3\") " pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 21:59:41.557901 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.557787 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a48e3e9b-b015-44af-89b9-42c2977ae9a8-metrics-certs\") pod \"network-metrics-daemon-w8fht\" (UID: \"a48e3e9b-b015-44af-89b9-42c2977ae9a8\") " pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 21:59:41.557901 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:41.557890 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:59:41.558004 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:41.557893 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 21:59:41.558051 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:41.558007 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09fdf756-6037-48d0-a284-719f7770ade3-original-pull-secret podName:09fdf756-6037-48d0-a284-719f7770ade3 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:42.557953567 +0000 UTC m=+4.008509118 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09fdf756-6037-48d0-a284-719f7770ade3-original-pull-secret") pod "global-pull-secret-syncer-zkrhd" (UID: "09fdf756-6037-48d0-a284-719f7770ade3") : object "kube-system"/"original-pull-secret" not registered Jul 13 21:59:41.558051 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:41.558033 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a48e3e9b-b015-44af-89b9-42c2977ae9a8-metrics-certs podName:a48e3e9b-b015-44af-89b9-42c2977ae9a8 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:42.558016333 +0000 UTC m=+4.008571847 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a48e3e9b-b015-44af-89b9-42c2977ae9a8-metrics-certs") pod "network-metrics-daemon-w8fht" (UID: "a48e3e9b-b015-44af-89b9-42c2977ae9a8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:59:41.711612 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:41.711356 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f77cde0_c7bd_4803_8c7b_b5e4071ace68.slice/crio-b8f840a93dc34e1992c267fce4d5f0425ffdd79844fadb95dacc68b18a89a51d WatchSource:0}: Error finding container b8f840a93dc34e1992c267fce4d5f0425ffdd79844fadb95dacc68b18a89a51d: Status 404 returned error can't find the container with id b8f840a93dc34e1992c267fce4d5f0425ffdd79844fadb95dacc68b18a89a51d Jul 13 21:59:41.712313 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:41.712104 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod348d04ea_a0a4_45da_ba4b_480288fd701b.slice/crio-830dd729cc3402aa0eaf834bc7d6b49397a4d717dd0df92bfe6fc20da4f43d70 WatchSource:0}: Error finding container 830dd729cc3402aa0eaf834bc7d6b49397a4d717dd0df92bfe6fc20da4f43d70: Status 404 returned error can't find the container with id 830dd729cc3402aa0eaf834bc7d6b49397a4d717dd0df92bfe6fc20da4f43d70 Jul 13 21:59:41.715170 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:41.715146 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod582b8d35_ee74_490e_aad6_e89ac36cb61c.slice/crio-4a7ff644ec87030e5dccf8ce7e662c3c77d238dcdf21af00bec1666c3a026c69 WatchSource:0}: Error finding container 4a7ff644ec87030e5dccf8ce7e662c3c77d238dcdf21af00bec1666c3a026c69: Status 404 returned error can't find the container with id 4a7ff644ec87030e5dccf8ce7e662c3c77d238dcdf21af00bec1666c3a026c69 Jul 13 21:59:41.716433 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:41.716413 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0952063_5741_4b6c_9a72_28e3efcf9eaf.slice/crio-9912f21d3b122c76c5de8364a417b9b35cb79189369f37b7689d4d65a31d4d42 WatchSource:0}: Error finding container 9912f21d3b122c76c5de8364a417b9b35cb79189369f37b7689d4d65a31d4d42: Status 404 returned error can't find the container with id 9912f21d3b122c76c5de8364a417b9b35cb79189369f37b7689d4d65a31d4d42 Jul 13 21:59:41.736236 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:41.736215 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1700b208_5391_4bed_8255_1594e9b3eac5.slice/crio-baac98a1f45269eaf38cce064633841b4283aae507fbd28cabfd0e6e206ecbb5 WatchSource:0}: Error finding container baac98a1f45269eaf38cce064633841b4283aae507fbd28cabfd0e6e206ecbb5: Status 404 returned error can't find the container with id baac98a1f45269eaf38cce064633841b4283aae507fbd28cabfd0e6e206ecbb5 Jul 13 21:59:41.736829 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:41.736811 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b79a451_a145_4116_8834_29040f299b3f.slice/crio-4de8a0b1cf3cf372b75ba1a44bf095f1f0d7e8cdfe58b1e830fad6bf5db1052c WatchSource:0}: Error finding container 4de8a0b1cf3cf372b75ba1a44bf095f1f0d7e8cdfe58b1e830fad6bf5db1052c: Status 404 returned error can't find the container with id 4de8a0b1cf3cf372b75ba1a44bf095f1f0d7e8cdfe58b1e830fad6bf5db1052c Jul 13 21:59:41.737758 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:41.737737 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c6ea85_17e8_4f97_b043_5da490a75040.slice/crio-3afc7bc574d45fffe4887764c6b519b912d1d5a0aaa5388c56fd62116a5e671c WatchSource:0}: Error finding container 3afc7bc574d45fffe4887764c6b519b912d1d5a0aaa5388c56fd62116a5e671c: Status 404 returned error can't find the container with id 3afc7bc574d45fffe4887764c6b519b912d1d5a0aaa5388c56fd62116a5e671c Jul 13 21:59:41.739235 ip-10-0-132-128 kubenswrapper[2562]: W0713 21:59:41.739214 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ac4b1c_ecb1_4550_84ab_7220e7445349.slice/crio-adc8bd815d0b94d11d549ec77a810507a366a5722c69ddd2310e1ca5ead95d45 WatchSource:0}: Error finding container adc8bd815d0b94d11d549ec77a810507a366a5722c69ddd2310e1ca5ead95d45: Status 404 returned error can't find the container with id adc8bd815d0b94d11d549ec77a810507a366a5722c69ddd2310e1ca5ead95d45 Jul 13 21:59:41.899573 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.899404 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 21:54:40 +0000 UTC" deadline="2028-02-13 06:54:10.79079396 +0000 UTC" Jul 13 21:59:41.899573 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.899572 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13904h54m28.891225402s" Jul 13 21:59:41.966962 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.966909 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pz6k2" event={"ID":"64c6ea85-17e8-4f97-b043-5da490a75040","Type":"ContainerStarted","Data":"3afc7bc574d45fffe4887764c6b519b912d1d5a0aaa5388c56fd62116a5e671c"} Jul 13 21:59:41.967798 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.967781 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" event={"ID":"1700b208-5391-4bed-8255-1594e9b3eac5","Type":"ContainerStarted","Data":"baac98a1f45269eaf38cce064633841b4283aae507fbd28cabfd0e6e206ecbb5"} Jul 13 21:59:41.968629 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.968613 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ccs2h" event={"ID":"3f77cde0-c7bd-4803-8c7b-b5e4071ace68","Type":"ContainerStarted","Data":"b8f840a93dc34e1992c267fce4d5f0425ffdd79844fadb95dacc68b18a89a51d"} Jul 13 21:59:41.969465 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.969449 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-svxd6" event={"ID":"6be816ca-053d-4a86-b5e8-755c240303cd","Type":"ContainerStarted","Data":"f921aef37a4ff3950fbc4cbf8c0117ebc2ed681caa5d20732a45ab84e424b1cb"} Jul 13 21:59:41.970848 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.970826 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-128.ec2.internal" event={"ID":"fad131b74fcb56c348327d4dd10f9830","Type":"ContainerStarted","Data":"fe91b0baff7f92c5e36b088df2f33c5972e9ade143badb51aedd153ea7f95786"} Jul 13 21:59:41.971861 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.971844 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-29z8f" event={"ID":"8b79a451-a145-4116-8834-29040f299b3f","Type":"ContainerStarted","Data":"4de8a0b1cf3cf372b75ba1a44bf095f1f0d7e8cdfe58b1e830fad6bf5db1052c"} Jul 13 21:59:41.972708 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.972689 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nshp2" event={"ID":"a0952063-5741-4b6c-9a72-28e3efcf9eaf","Type":"ContainerStarted","Data":"9912f21d3b122c76c5de8364a417b9b35cb79189369f37b7689d4d65a31d4d42"} Jul 13 21:59:41.973570 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.973546 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" event={"ID":"582b8d35-ee74-490e-aad6-e89ac36cb61c","Type":"ContainerStarted","Data":"4a7ff644ec87030e5dccf8ce7e662c3c77d238dcdf21af00bec1666c3a026c69"} Jul 13 21:59:41.974343 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.974328 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7dpl6" event={"ID":"348d04ea-a0a4-45da-ba4b-480288fd701b","Type":"ContainerStarted","Data":"830dd729cc3402aa0eaf834bc7d6b49397a4d717dd0df92bfe6fc20da4f43d70"} Jul 13 21:59:41.975502 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.975483 2562 generic.go:356] "Generic (PLEG): container finished" podID="a282de8bb107da660700ce7cc100840d" containerID="0a85ea39c3fbb48b2f0f065d79e3a9a22f07884e49cd1a215e44d93d55388226" exitCode=0 Jul 13 21:59:41.975577 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.975528 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-128.ec2.internal" event={"ID":"a282de8bb107da660700ce7cc100840d","Type":"ContainerDied","Data":"0a85ea39c3fbb48b2f0f065d79e3a9a22f07884e49cd1a215e44d93d55388226"} Jul 13 21:59:41.976645 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.976628 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lvr8p" event={"ID":"75ac4b1c-ecb1-4550-84ab-7220e7445349","Type":"ContainerStarted","Data":"adc8bd815d0b94d11d549ec77a810507a366a5722c69ddd2310e1ca5ead95d45"} Jul 13 21:59:41.986960 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.986911 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-128.ec2.internal" podStartSLOduration=2.986901309 podStartE2EDuration="2.986901309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:59:41.986898065 +0000 UTC m=+3.437453590" watchObservedRunningTime="2026-07-13 21:59:41.986901309 +0000 UTC m=+3.437456842" Jul 13 21:59:41.998309 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:41.997807 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 13 21:59:42.465189 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:42.465099 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg6gq\" (UniqueName: \"kubernetes.io/projected/71a3685e-3328-41cb-8b35-c8e8e1efe021-kube-api-access-pg6gq\") pod \"network-check-target-nbdz9\" (UID: \"71a3685e-3328-41cb-8b35-c8e8e1efe021\") " pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 21:59:42.465352 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:42.465329 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 21:59:42.465352 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:42.465352 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 21:59:42.465460 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:42.465365 2562 projected.go:196] Error preparing data for projected volume kube-api-access-pg6gq for pod openshift-network-diagnostics/network-check-target-nbdz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:59:42.465460 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:42.465423 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71a3685e-3328-41cb-8b35-c8e8e1efe021-kube-api-access-pg6gq podName:71a3685e-3328-41cb-8b35-c8e8e1efe021 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:44.46540477 +0000 UTC m=+5.915960288 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pg6gq" (UniqueName: "kubernetes.io/projected/71a3685e-3328-41cb-8b35-c8e8e1efe021-kube-api-access-pg6gq") pod "network-check-target-nbdz9" (UID: "71a3685e-3328-41cb-8b35-c8e8e1efe021") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:59:42.566167 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:42.566123 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a48e3e9b-b015-44af-89b9-42c2977ae9a8-metrics-certs\") pod \"network-metrics-daemon-w8fht\" (UID: \"a48e3e9b-b015-44af-89b9-42c2977ae9a8\") " pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 21:59:42.566343 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:42.566206 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09fdf756-6037-48d0-a284-719f7770ade3-original-pull-secret\") pod \"global-pull-secret-syncer-zkrhd\" (UID: \"09fdf756-6037-48d0-a284-719f7770ade3\") " pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 21:59:42.566343 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:42.566335 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 21:59:42.566447 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:42.566397 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09fdf756-6037-48d0-a284-719f7770ade3-original-pull-secret podName:09fdf756-6037-48d0-a284-719f7770ade3 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:44.566378036 +0000 UTC m=+6.016933567 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09fdf756-6037-48d0-a284-719f7770ade3-original-pull-secret") pod "global-pull-secret-syncer-zkrhd" (UID: "09fdf756-6037-48d0-a284-719f7770ade3") : object "kube-system"/"original-pull-secret" not registered Jul 13 21:59:42.566649 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:42.566621 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:59:42.566719 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:42.566701 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a48e3e9b-b015-44af-89b9-42c2977ae9a8-metrics-certs podName:a48e3e9b-b015-44af-89b9-42c2977ae9a8 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:44.566682202 +0000 UTC m=+6.017237731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a48e3e9b-b015-44af-89b9-42c2977ae9a8-metrics-certs") pod "network-metrics-daemon-w8fht" (UID: "a48e3e9b-b015-44af-89b9-42c2977ae9a8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:59:42.900605 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:42.900467 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-12 21:54:40 +0000 UTC" deadline="2028-04-25 01:32:16.172182736 +0000 UTC" Jul 13 21:59:42.900605 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:42.900528 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15627h32m33.271658272s" Jul 13 21:59:42.963042 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:42.963011 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 21:59:42.963219 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:42.963142 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zkrhd" podUID="09fdf756-6037-48d0-a284-719f7770ade3" Jul 13 21:59:42.963596 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:42.963575 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 21:59:42.963705 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:42.963685 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8fht" podUID="a48e3e9b-b015-44af-89b9-42c2977ae9a8" Jul 13 21:59:42.963813 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:42.963785 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 21:59:42.963909 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:42.963887 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nbdz9" podUID="71a3685e-3328-41cb-8b35-c8e8e1efe021" Jul 13 21:59:43.018639 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:43.018572 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-128.ec2.internal" event={"ID":"a282de8bb107da660700ce7cc100840d","Type":"ContainerStarted","Data":"64d54df104254be64685f3b5aae4a5a8f525d09c614cfb384813185fadd5c40b"} Jul 13 21:59:43.033281 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:43.033230 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-128.ec2.internal" podStartSLOduration=4.033208878 podStartE2EDuration="4.033208878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 21:59:43.032156733 +0000 UTC m=+4.482712271" watchObservedRunningTime="2026-07-13 21:59:43.033208878 +0000 UTC m=+4.483764417" Jul 13 21:59:44.484626 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:44.484017 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg6gq\" (UniqueName: \"kubernetes.io/projected/71a3685e-3328-41cb-8b35-c8e8e1efe021-kube-api-access-pg6gq\") pod \"network-check-target-nbdz9\" (UID: \"71a3685e-3328-41cb-8b35-c8e8e1efe021\") " pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 21:59:44.484626 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:44.484179 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 21:59:44.484626 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:44.484199 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 21:59:44.484626 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:44.484212 2562 projected.go:196] Error preparing data for projected volume kube-api-access-pg6gq for pod openshift-network-diagnostics/network-check-target-nbdz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:59:44.484626 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:44.484279 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71a3685e-3328-41cb-8b35-c8e8e1efe021-kube-api-access-pg6gq podName:71a3685e-3328-41cb-8b35-c8e8e1efe021 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:48.48425961 +0000 UTC m=+9.934815147 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pg6gq" (UniqueName: "kubernetes.io/projected/71a3685e-3328-41cb-8b35-c8e8e1efe021-kube-api-access-pg6gq") pod "network-check-target-nbdz9" (UID: "71a3685e-3328-41cb-8b35-c8e8e1efe021") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:59:44.585170 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:44.584448 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09fdf756-6037-48d0-a284-719f7770ade3-original-pull-secret\") pod \"global-pull-secret-syncer-zkrhd\" (UID: \"09fdf756-6037-48d0-a284-719f7770ade3\") " pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 21:59:44.585170 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:44.584524 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a48e3e9b-b015-44af-89b9-42c2977ae9a8-metrics-certs\") pod \"network-metrics-daemon-w8fht\" (UID: \"a48e3e9b-b015-44af-89b9-42c2977ae9a8\") " pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 21:59:44.585170 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:44.584643 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:59:44.585170 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:44.584706 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a48e3e9b-b015-44af-89b9-42c2977ae9a8-metrics-certs podName:a48e3e9b-b015-44af-89b9-42c2977ae9a8 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:48.584685887 +0000 UTC m=+10.035241403 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a48e3e9b-b015-44af-89b9-42c2977ae9a8-metrics-certs") pod "network-metrics-daemon-w8fht" (UID: "a48e3e9b-b015-44af-89b9-42c2977ae9a8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:59:44.585170 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:44.585109 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 21:59:44.585170 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:44.585161 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09fdf756-6037-48d0-a284-719f7770ade3-original-pull-secret podName:09fdf756-6037-48d0-a284-719f7770ade3 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:48.585146033 +0000 UTC m=+10.035701551 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09fdf756-6037-48d0-a284-719f7770ade3-original-pull-secret") pod "global-pull-secret-syncer-zkrhd" (UID: "09fdf756-6037-48d0-a284-719f7770ade3") : object "kube-system"/"original-pull-secret" not registered Jul 13 21:59:44.960609 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:44.960467 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 21:59:44.960609 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:44.960484 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 21:59:44.960609 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:44.960607 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 21:59:44.960877 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:44.960633 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zkrhd" podUID="09fdf756-6037-48d0-a284-719f7770ade3" Jul 13 21:59:44.960877 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:44.960748 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8fht" podUID="a48e3e9b-b015-44af-89b9-42c2977ae9a8" Jul 13 21:59:44.960877 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:44.960828 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nbdz9" podUID="71a3685e-3328-41cb-8b35-c8e8e1efe021" Jul 13 21:59:46.961009 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:46.960956 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 21:59:46.961431 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:46.961107 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nbdz9" podUID="71a3685e-3328-41cb-8b35-c8e8e1efe021" Jul 13 21:59:46.961431 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:46.961394 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 21:59:46.961431 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:46.961417 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 21:59:46.961618 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:46.961573 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8fht" podUID="a48e3e9b-b015-44af-89b9-42c2977ae9a8" Jul 13 21:59:46.961728 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:46.961680 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zkrhd" podUID="09fdf756-6037-48d0-a284-719f7770ade3" Jul 13 21:59:48.519218 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:48.519084 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg6gq\" (UniqueName: \"kubernetes.io/projected/71a3685e-3328-41cb-8b35-c8e8e1efe021-kube-api-access-pg6gq\") pod \"network-check-target-nbdz9\" (UID: \"71a3685e-3328-41cb-8b35-c8e8e1efe021\") " pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 21:59:48.519646 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:48.519235 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 21:59:48.519646 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:48.519257 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 21:59:48.519646 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:48.519268 2562 projected.go:196] Error preparing data for projected volume kube-api-access-pg6gq for pod openshift-network-diagnostics/network-check-target-nbdz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:59:48.519646 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:48.519326 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71a3685e-3328-41cb-8b35-c8e8e1efe021-kube-api-access-pg6gq podName:71a3685e-3328-41cb-8b35-c8e8e1efe021 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:56.519306445 +0000 UTC m=+17.969861960 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pg6gq" (UniqueName: "kubernetes.io/projected/71a3685e-3328-41cb-8b35-c8e8e1efe021-kube-api-access-pg6gq") pod "network-check-target-nbdz9" (UID: "71a3685e-3328-41cb-8b35-c8e8e1efe021") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:59:48.619633 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:48.619599 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a48e3e9b-b015-44af-89b9-42c2977ae9a8-metrics-certs\") pod \"network-metrics-daemon-w8fht\" (UID: \"a48e3e9b-b015-44af-89b9-42c2977ae9a8\") " pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 21:59:48.619780 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:48.619679 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09fdf756-6037-48d0-a284-719f7770ade3-original-pull-secret\") pod \"global-pull-secret-syncer-zkrhd\" (UID: \"09fdf756-6037-48d0-a284-719f7770ade3\") " pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 21:59:48.619840 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:48.619803 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 21:59:48.619885 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:48.619872 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09fdf756-6037-48d0-a284-719f7770ade3-original-pull-secret podName:09fdf756-6037-48d0-a284-719f7770ade3 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:56.619850397 +0000 UTC m=+18.070405915 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09fdf756-6037-48d0-a284-719f7770ade3-original-pull-secret") pod "global-pull-secret-syncer-zkrhd" (UID: "09fdf756-6037-48d0-a284-719f7770ade3") : object "kube-system"/"original-pull-secret" not registered Jul 13 21:59:48.619942 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:48.619882 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:59:48.619993 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:48.619956 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a48e3e9b-b015-44af-89b9-42c2977ae9a8-metrics-certs podName:a48e3e9b-b015-44af-89b9-42c2977ae9a8 nodeName:}" failed. No retries permitted until 2026-07-13 21:59:56.619937023 +0000 UTC m=+18.070492544 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a48e3e9b-b015-44af-89b9-42c2977ae9a8-metrics-certs") pod "network-metrics-daemon-w8fht" (UID: "a48e3e9b-b015-44af-89b9-42c2977ae9a8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:59:48.961872 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:48.961103 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 21:59:48.961872 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:48.961220 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nbdz9" podUID="71a3685e-3328-41cb-8b35-c8e8e1efe021" Jul 13 21:59:48.961872 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:48.961599 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 21:59:48.961872 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:48.961699 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zkrhd" podUID="09fdf756-6037-48d0-a284-719f7770ade3" Jul 13 21:59:48.961872 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:48.961758 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 21:59:48.961872 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:48.961838 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8fht" podUID="a48e3e9b-b015-44af-89b9-42c2977ae9a8" Jul 13 21:59:50.961021 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:50.960964 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 21:59:50.961473 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:50.961132 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 21:59:50.961473 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:50.961225 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8fht" podUID="a48e3e9b-b015-44af-89b9-42c2977ae9a8" Jul 13 21:59:50.961473 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:50.961123 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zkrhd" podUID="09fdf756-6037-48d0-a284-719f7770ade3" Jul 13 21:59:50.961473 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:50.961288 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 21:59:50.961473 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:50.961357 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nbdz9" podUID="71a3685e-3328-41cb-8b35-c8e8e1efe021" Jul 13 21:59:52.960186 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:52.960153 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 21:59:52.960186 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:52.960188 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 21:59:52.960693 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:52.960153 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 21:59:52.960693 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:52.960295 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nbdz9" podUID="71a3685e-3328-41cb-8b35-c8e8e1efe021" Jul 13 21:59:52.960693 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:52.960370 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zkrhd" podUID="09fdf756-6037-48d0-a284-719f7770ade3" Jul 13 21:59:52.960693 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:52.960503 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8fht" podUID="a48e3e9b-b015-44af-89b9-42c2977ae9a8" Jul 13 21:59:54.961071 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:54.960959 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 21:59:54.961071 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:54.960959 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 21:59:54.961579 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:54.961102 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 21:59:54.961579 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:54.961114 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zkrhd" podUID="09fdf756-6037-48d0-a284-719f7770ade3" Jul 13 21:59:54.961579 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:54.961242 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8fht" podUID="a48e3e9b-b015-44af-89b9-42c2977ae9a8" Jul 13 21:59:54.961579 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:54.961337 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nbdz9" podUID="71a3685e-3328-41cb-8b35-c8e8e1efe021" Jul 13 21:59:56.580062 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:56.580028 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg6gq\" (UniqueName: \"kubernetes.io/projected/71a3685e-3328-41cb-8b35-c8e8e1efe021-kube-api-access-pg6gq\") pod \"network-check-target-nbdz9\" (UID: \"71a3685e-3328-41cb-8b35-c8e8e1efe021\") " pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 21:59:56.580520 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:56.580171 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 21:59:56.580520 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:56.580195 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 21:59:56.580520 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:56.580209 2562 projected.go:196] Error preparing data for projected volume kube-api-access-pg6gq for pod openshift-network-diagnostics/network-check-target-nbdz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:59:56.580520 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:56.580271 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71a3685e-3328-41cb-8b35-c8e8e1efe021-kube-api-access-pg6gq podName:71a3685e-3328-41cb-8b35-c8e8e1efe021 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:12.580257194 +0000 UTC m=+34.030812709 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pg6gq" (UniqueName: "kubernetes.io/projected/71a3685e-3328-41cb-8b35-c8e8e1efe021-kube-api-access-pg6gq") pod "network-check-target-nbdz9" (UID: "71a3685e-3328-41cb-8b35-c8e8e1efe021") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 21:59:56.680846 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:56.680805 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09fdf756-6037-48d0-a284-719f7770ade3-original-pull-secret\") pod \"global-pull-secret-syncer-zkrhd\" (UID: \"09fdf756-6037-48d0-a284-719f7770ade3\") " pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 21:59:56.681036 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:56.680878 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a48e3e9b-b015-44af-89b9-42c2977ae9a8-metrics-certs\") pod \"network-metrics-daemon-w8fht\" (UID: \"a48e3e9b-b015-44af-89b9-42c2977ae9a8\") " pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 21:59:56.681036 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:56.680963 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 21:59:56.681036 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:56.681014 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:59:56.681175 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:56.681043 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09fdf756-6037-48d0-a284-719f7770ade3-original-pull-secret podName:09fdf756-6037-48d0-a284-719f7770ade3 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:12.681027708 +0000 UTC m=+34.131583230 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09fdf756-6037-48d0-a284-719f7770ade3-original-pull-secret") pod "global-pull-secret-syncer-zkrhd" (UID: "09fdf756-6037-48d0-a284-719f7770ade3") : object "kube-system"/"original-pull-secret" not registered Jul 13 21:59:56.681175 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:56.681069 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a48e3e9b-b015-44af-89b9-42c2977ae9a8-metrics-certs podName:a48e3e9b-b015-44af-89b9-42c2977ae9a8 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:12.681053565 +0000 UTC m=+34.131609081 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a48e3e9b-b015-44af-89b9-42c2977ae9a8-metrics-certs") pod "network-metrics-daemon-w8fht" (UID: "a48e3e9b-b015-44af-89b9-42c2977ae9a8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 21:59:56.960868 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:56.960825 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 21:59:56.961043 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:56.960830 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 21:59:56.961043 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:56.960956 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zkrhd" podUID="09fdf756-6037-48d0-a284-719f7770ade3" Jul 13 21:59:56.961150 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:56.961052 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8fht" podUID="a48e3e9b-b015-44af-89b9-42c2977ae9a8" Jul 13 21:59:56.961150 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:56.960836 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 21:59:56.961249 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:56.961147 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nbdz9" podUID="71a3685e-3328-41cb-8b35-c8e8e1efe021" Jul 13 21:59:58.961016 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:58.960959 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 21:59:58.961501 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:58.961065 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 21:59:58.961501 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:58.961080 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nbdz9" podUID="71a3685e-3328-41cb-8b35-c8e8e1efe021" Jul 13 21:59:58.961501 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:58.961194 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8fht" podUID="a48e3e9b-b015-44af-89b9-42c2977ae9a8" Jul 13 21:59:58.961501 ip-10-0-132-128 kubenswrapper[2562]: I0713 21:59:58.961228 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 21:59:58.961501 ip-10-0-132-128 kubenswrapper[2562]: E0713 21:59:58.961349 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zkrhd" podUID="09fdf756-6037-48d0-a284-719f7770ade3" Jul 13 22:00:00.961036 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:00.960772 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 22:00:00.961571 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:00.960772 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 22:00:00.961571 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:00.961149 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zkrhd" podUID="09fdf756-6037-48d0-a284-719f7770ade3" Jul 13 22:00:00.961571 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:00.960794 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 22:00:00.961571 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:00.961226 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8fht" podUID="a48e3e9b-b015-44af-89b9-42c2977ae9a8" Jul 13 22:00:00.961571 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:00.961283 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nbdz9" podUID="71a3685e-3328-41cb-8b35-c8e8e1efe021" Jul 13 22:00:01.068758 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:01.068722 2562 generic.go:356] "Generic (PLEG): container finished" podID="75ac4b1c-ecb1-4550-84ab-7220e7445349" containerID="597b2af71f4e82ec9b5f067b5f8db2b782dc7194936279ce81aca666a458dae8" exitCode=0 Jul 13 22:00:01.068880 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:01.068815 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lvr8p" event={"ID":"75ac4b1c-ecb1-4550-84ab-7220e7445349","Type":"ContainerDied","Data":"597b2af71f4e82ec9b5f067b5f8db2b782dc7194936279ce81aca666a458dae8"} Jul 13 22:00:01.070350 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:01.070303 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pz6k2" event={"ID":"64c6ea85-17e8-4f97-b043-5da490a75040","Type":"ContainerStarted","Data":"ca2a2593787f37b2ff9f19526058249e35f2ea4df31e8ba74942959cc8eab2f4"} Jul 13 22:00:01.072070 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:01.072043 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" event={"ID":"1700b208-5391-4bed-8255-1594e9b3eac5","Type":"ContainerStarted","Data":"a37fd9dd7fe2bf9b865084a4de565493e4dfeb5fc7d3421c31201155c1af9020"} Jul 13 22:00:01.073300 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:01.073270 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ccs2h" event={"ID":"3f77cde0-c7bd-4803-8c7b-b5e4071ace68","Type":"ContainerStarted","Data":"c73e22cc56d0aae61d429565e059740fa3017c793b2796b02f41bf6867051ef2"} Jul 13 22:00:01.074717 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:01.074689 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-svxd6" event={"ID":"6be816ca-053d-4a86-b5e8-755c240303cd","Type":"ContainerStarted","Data":"460e6b6e848dc97e65c136e339530f5f522efb430824558d25515081f041fb90"} Jul 13 22:00:01.076207 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:01.076177 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nshp2" event={"ID":"a0952063-5741-4b6c-9a72-28e3efcf9eaf","Type":"ContainerStarted","Data":"19c48c6b8a4ea7da94a1dbc6006c73d605fd20dabf755dac16caae3ec2a88c28"} Jul 13 22:00:01.077801 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:01.077781 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" event={"ID":"582b8d35-ee74-490e-aad6-e89ac36cb61c","Type":"ContainerStarted","Data":"54c9f4f0f8f4163cac2a38f76de3f23022e1a9e71306caff7fa99d70cd1c2bc2"} Jul 13 22:00:01.079074 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:01.079051 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7dpl6" event={"ID":"348d04ea-a0a4-45da-ba4b-480288fd701b","Type":"ContainerStarted","Data":"613fbfc1a2f82729c41f973815f93487832f5dcd2cf69b2a5214b7ddbb599f2a"} Jul 13 22:00:01.117444 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:01.117397 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7dpl6" podStartSLOduration=3.24168944 podStartE2EDuration="22.117380177s" totalImagesPullingTime="18.875690737s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:59:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:01.117002833 +0000 UTC m=+22.567558367" watchObservedRunningTime="2026-07-13 22:00:01.117380177 +0000 UTC m=+22.567935713" Jul 13 22:00:01.137302 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:01.137122 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-svxd6" podStartSLOduration=3.240756338 podStartE2EDuration="22.137099068s" totalImagesPullingTime="18.89634273s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:59:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:01.136370975 +0000 UTC m=+22.586926523" watchObservedRunningTime="2026-07-13 22:00:01.137099068 +0000 UTC m=+22.587654608" Jul 13 22:00:01.156287 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:01.156237 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pz6k2" podStartSLOduration=7.841331646 podStartE2EDuration="22.156219986s" totalImagesPullingTime="14.31488834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:59:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:01.155767407 +0000 UTC m=+22.606322943" watchObservedRunningTime="2026-07-13 22:00:01.156219986 +0000 UTC m=+22.606775522" Jul 13 22:00:01.173960 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:01.173919 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nshp2" podStartSLOduration=3.3175292069999998 podStartE2EDuration="22.17390338s" totalImagesPullingTime="18.856374173s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:59:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:01.173795616 +0000 UTC m=+22.624351153" watchObservedRunningTime="2026-07-13 22:00:01.17390338 +0000 UTC m=+22.624458917" Jul 13 22:00:02.085668 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:02.085626 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-29z8f" event={"ID":"8b79a451-a145-4116-8834-29040f299b3f","Type":"ContainerStarted","Data":"ca796f23b51abdbd9535df78ca323ba2e38ccd3aeeb046a74479abecedac59f3"} Jul 13 22:00:02.088118 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:02.088099 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw4h7_582b8d35-ee74-490e-aad6-e89ac36cb61c/ovn-acl-logging/0.log" Jul 13 22:00:02.088397 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:02.088377 2562 generic.go:356] "Generic (PLEG): container finished" podID="582b8d35-ee74-490e-aad6-e89ac36cb61c" containerID="97378f84ed53a8b80acdc5c3a4b8346307e3ba3c8a616ab79de4d20fbf6a735a" exitCode=1 Jul 13 22:00:02.088489 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:02.088455 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" event={"ID":"582b8d35-ee74-490e-aad6-e89ac36cb61c","Type":"ContainerDied","Data":"97378f84ed53a8b80acdc5c3a4b8346307e3ba3c8a616ab79de4d20fbf6a735a"} Jul 13 22:00:02.088618 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:02.088546 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" event={"ID":"582b8d35-ee74-490e-aad6-e89ac36cb61c","Type":"ContainerStarted","Data":"8b8351173527e2a59dd6f84da0b00206c06d32c68c5c5820ede18bb9c181428d"} Jul 13 22:00:02.088618 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:02.088608 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" event={"ID":"582b8d35-ee74-490e-aad6-e89ac36cb61c","Type":"ContainerStarted","Data":"e038b24d671fac7a36b029fe06c37e9b5b619f9e4944828097557de8583422af"} Jul 13 22:00:02.088731 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:02.088663 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" event={"ID":"582b8d35-ee74-490e-aad6-e89ac36cb61c","Type":"ContainerStarted","Data":"d2799a94b59b716d948b12ea60cebd96ec5492b38039f06c8d303ed84a29fb51"} Jul 13 22:00:02.088731 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:02.088712 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" event={"ID":"582b8d35-ee74-490e-aad6-e89ac36cb61c","Type":"ContainerStarted","Data":"ffb848d2216239de14928c77472d136ad7221542f9017dd2b32c60ebe0697712"} Jul 13 22:00:02.099813 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:02.099776 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-ccs2h" podStartSLOduration=4.238114368 podStartE2EDuration="23.099762496s" totalImagesPullingTime="18.861648128s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:59:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:01.200840157 +0000 UTC m=+22.651395694" watchObservedRunningTime="2026-07-13 22:00:02.099762496 +0000 UTC m=+23.550318026" Jul 13 22:00:02.100203 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:02.100181 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-29z8f" podStartSLOduration=4.263546721 podStartE2EDuration="23.100172631s" totalImagesPullingTime="18.83662591s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:59:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:02.099749548 +0000 UTC m=+23.550305086" watchObservedRunningTime="2026-07-13 22:00:02.100172631 +0000 UTC m=+23.550728168" Jul 13 22:00:02.240095 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:02.240057 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 13 22:00:02.910739 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:02.910637 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-13T22:00:02.24007637Z","UUID":"0b18c46f-8732-401a-92c2-61aada39accd","Handler":null,"Name":"","Endpoint":""} Jul 13 22:00:02.912327 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:02.912293 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 13 22:00:02.912452 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:02.912340 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 13 22:00:02.960122 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:02.960091 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 22:00:02.960303 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:02.960091 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 22:00:02.960303 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:02.960225 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8fht" podUID="a48e3e9b-b015-44af-89b9-42c2977ae9a8" Jul 13 22:00:02.960303 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:02.960095 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 22:00:02.960462 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:02.960269 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nbdz9" podUID="71a3685e-3328-41cb-8b35-c8e8e1efe021" Jul 13 22:00:02.960462 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:02.960384 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zkrhd" podUID="09fdf756-6037-48d0-a284-719f7770ade3" Jul 13 22:00:03.092892 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:03.092855 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" event={"ID":"1700b208-5391-4bed-8255-1594e9b3eac5","Type":"ContainerStarted","Data":"e1b95cd32e02521a8387501d42cb03b5bc0f206bf0bf1a7b23cb102bb3c65a1c"} Jul 13 22:00:04.097546 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:04.097465 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" event={"ID":"1700b208-5391-4bed-8255-1594e9b3eac5","Type":"ContainerStarted","Data":"95760cc96f8fd05e97ecb5a035e9bc7650a4c556acb444fbbbc9a5d7a7fb5422"} Jul 13 22:00:04.100845 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:04.100824 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw4h7_582b8d35-ee74-490e-aad6-e89ac36cb61c/ovn-acl-logging/0.log" Jul 13 22:00:04.101232 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:04.101202 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" event={"ID":"582b8d35-ee74-490e-aad6-e89ac36cb61c","Type":"ContainerStarted","Data":"173af863ba1a151d4d73d6a620a5f46f14ad26f8198fb4488a97bfbe6a8f2eb7"} Jul 13 22:00:04.117560 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:04.117519 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n2llx" podStartSLOduration=4.301671726 podStartE2EDuration="26.117499626s" totalImagesPullingTime="21.8158279s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:59:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:04.117495625 +0000 UTC m=+25.568051162" watchObservedRunningTime="2026-07-13 22:00:04.117499626 +0000 UTC m=+25.568055164" Jul 13 22:00:04.488364 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:04.488170 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pz6k2" Jul 13 22:00:04.488771 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:04.488749 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pz6k2" Jul 13 22:00:04.960569 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:04.960540 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 22:00:04.960753 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:04.960577 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 22:00:04.960753 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:04.960545 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 22:00:04.960753 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:04.960660 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nbdz9" podUID="71a3685e-3328-41cb-8b35-c8e8e1efe021" Jul 13 22:00:04.960753 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:04.960714 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zkrhd" podUID="09fdf756-6037-48d0-a284-719f7770ade3" Jul 13 22:00:04.960933 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:04.960794 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8fht" podUID="a48e3e9b-b015-44af-89b9-42c2977ae9a8" Jul 13 22:00:06.110370 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:06.110328 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lvr8p" event={"ID":"75ac4b1c-ecb1-4550-84ab-7220e7445349","Type":"ContainerStarted","Data":"e983dbc367e16ec1444ceb0f657ac591152590cf7197b8cc5dd7f2fffdb43dcc"} Jul 13 22:00:06.114945 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:06.114717 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw4h7_582b8d35-ee74-490e-aad6-e89ac36cb61c/ovn-acl-logging/0.log" Jul 13 22:00:06.115624 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:06.115358 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" event={"ID":"582b8d35-ee74-490e-aad6-e89ac36cb61c","Type":"ContainerStarted","Data":"ae3d6825207655b2a6e1b71413da1f1ba58ac5e76c9ad3d7bfc84cc81ab6797d"} Jul 13 22:00:06.115624 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:06.115566 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 22:00:06.115789 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:06.115629 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 22:00:06.115860 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:06.115845 2562 scope.go:117] "RemoveContainer" containerID="97378f84ed53a8b80acdc5c3a4b8346307e3ba3c8a616ab79de4d20fbf6a735a" Jul 13 22:00:06.131909 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:06.131885 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 22:00:06.960426 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:06.960393 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 22:00:06.960582 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:06.960393 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 22:00:06.960582 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:06.960500 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nbdz9" podUID="71a3685e-3328-41cb-8b35-c8e8e1efe021" Jul 13 22:00:06.960659 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:06.960588 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8fht" podUID="a48e3e9b-b015-44af-89b9-42c2977ae9a8" Jul 13 22:00:06.960659 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:06.960391 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 22:00:06.960721 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:06.960669 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zkrhd" podUID="09fdf756-6037-48d0-a284-719f7770ade3" Jul 13 22:00:07.119185 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:07.119106 2562 generic.go:356] "Generic (PLEG): container finished" podID="75ac4b1c-ecb1-4550-84ab-7220e7445349" containerID="e983dbc367e16ec1444ceb0f657ac591152590cf7197b8cc5dd7f2fffdb43dcc" exitCode=0 Jul 13 22:00:07.119586 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:07.119185 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lvr8p" event={"ID":"75ac4b1c-ecb1-4550-84ab-7220e7445349","Type":"ContainerDied","Data":"e983dbc367e16ec1444ceb0f657ac591152590cf7197b8cc5dd7f2fffdb43dcc"} Jul 13 22:00:07.122858 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:07.122831 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw4h7_582b8d35-ee74-490e-aad6-e89ac36cb61c/ovn-acl-logging/0.log" Jul 13 22:00:07.123210 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:07.123183 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" event={"ID":"582b8d35-ee74-490e-aad6-e89ac36cb61c","Type":"ContainerStarted","Data":"7dc1b6c6743bb3cf56814c7c10953affbf73eecd284aac4ff8d22a8af65e71bd"} Jul 13 22:00:07.123453 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:07.123415 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 22:00:07.145535 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:07.145496 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 22:00:07.171753 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:07.171686 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" podStartSLOduration=10.102008497 podStartE2EDuration="29.171664965s" totalImagesPullingTime="19.069656468s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:07.170547386 +0000 UTC m=+28.621102924" watchObservedRunningTime="2026-07-13 22:00:07.171664965 +0000 UTC m=+28.622220503" Jul 13 22:00:07.890821 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:07.890787 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w8fht"] Jul 13 22:00:07.891114 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:07.890918 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 22:00:07.891114 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:07.891025 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8fht" podUID="a48e3e9b-b015-44af-89b9-42c2977ae9a8" Jul 13 22:00:07.894229 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:07.894203 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zkrhd"] Jul 13 22:00:07.894423 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:07.894314 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 22:00:07.894423 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:07.894410 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zkrhd" podUID="09fdf756-6037-48d0-a284-719f7770ade3" Jul 13 22:00:07.894934 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:07.894912 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nbdz9"] Jul 13 22:00:07.895032 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:07.895015 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 22:00:07.895095 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:07.895080 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nbdz9" podUID="71a3685e-3328-41cb-8b35-c8e8e1efe021" Jul 13 22:00:08.051178 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:08.051142 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pz6k2" Jul 13 22:00:08.051326 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:08.051287 2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 13 22:00:08.051836 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:08.051817 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pz6k2" Jul 13 22:00:09.128915 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:09.128883 2562 generic.go:356] "Generic (PLEG): container finished" podID="75ac4b1c-ecb1-4550-84ab-7220e7445349" containerID="bb1094d38f4c45426b693c4a26bb46e2165872c1a5425ec19b49bf086437bcf3" exitCode=0 Jul 13 22:00:09.129353 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:09.128963 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lvr8p" event={"ID":"75ac4b1c-ecb1-4550-84ab-7220e7445349","Type":"ContainerDied","Data":"bb1094d38f4c45426b693c4a26bb46e2165872c1a5425ec19b49bf086437bcf3"} Jul 13 22:00:09.960790 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:09.960767 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 22:00:09.960903 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:09.960768 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 22:00:09.960963 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:09.960901 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8fht" podUID="a48e3e9b-b015-44af-89b9-42c2977ae9a8" Jul 13 22:00:09.960963 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:09.960767 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 22:00:09.960963 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:09.960943 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zkrhd" podUID="09fdf756-6037-48d0-a284-719f7770ade3" Jul 13 22:00:09.961136 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:09.961043 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nbdz9" podUID="71a3685e-3328-41cb-8b35-c8e8e1efe021" Jul 13 22:00:11.136262 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:11.136086 2562 generic.go:356] "Generic (PLEG): container finished" podID="75ac4b1c-ecb1-4550-84ab-7220e7445349" containerID="e2790932be5b2ef306e4000279753a25ff0253db99be4dbc1e26af40fc2ce42d" exitCode=0 Jul 13 22:00:11.136713 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:11.136168 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lvr8p" event={"ID":"75ac4b1c-ecb1-4550-84ab-7220e7445349","Type":"ContainerDied","Data":"e2790932be5b2ef306e4000279753a25ff0253db99be4dbc1e26af40fc2ce42d"} Jul 13 22:00:11.959890 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:11.959862 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 22:00:11.960056 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:11.959953 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 22:00:11.960056 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:11.959958 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nbdz9" podUID="71a3685e-3328-41cb-8b35-c8e8e1efe021" Jul 13 22:00:11.960056 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:11.960000 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 22:00:11.960221 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:11.960058 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8fht" podUID="a48e3e9b-b015-44af-89b9-42c2977ae9a8" Jul 13 22:00:11.960221 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:11.960140 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zkrhd" podUID="09fdf756-6037-48d0-a284-719f7770ade3" Jul 13 22:00:12.603321 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:12.603231 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg6gq\" (UniqueName: \"kubernetes.io/projected/71a3685e-3328-41cb-8b35-c8e8e1efe021-kube-api-access-pg6gq\") pod \"network-check-target-nbdz9\" (UID: \"71a3685e-3328-41cb-8b35-c8e8e1efe021\") " pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 22:00:12.603733 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:12.603402 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 13 22:00:12.603733 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:12.603425 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 13 22:00:12.603733 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:12.603435 2562 projected.go:196] Error preparing data for projected volume kube-api-access-pg6gq for pod openshift-network-diagnostics/network-check-target-nbdz9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 22:00:12.603733 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:12.603492 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/71a3685e-3328-41cb-8b35-c8e8e1efe021-kube-api-access-pg6gq podName:71a3685e-3328-41cb-8b35-c8e8e1efe021 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:44.60347379 +0000 UTC m=+66.054029314 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pg6gq" (UniqueName: "kubernetes.io/projected/71a3685e-3328-41cb-8b35-c8e8e1efe021-kube-api-access-pg6gq") pod "network-check-target-nbdz9" (UID: "71a3685e-3328-41cb-8b35-c8e8e1efe021") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 13 22:00:12.704572 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:12.704536 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09fdf756-6037-48d0-a284-719f7770ade3-original-pull-secret\") pod \"global-pull-secret-syncer-zkrhd\" (UID: \"09fdf756-6037-48d0-a284-719f7770ade3\") " pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 22:00:12.704737 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:12.704612 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a48e3e9b-b015-44af-89b9-42c2977ae9a8-metrics-certs\") pod \"network-metrics-daemon-w8fht\" (UID: \"a48e3e9b-b015-44af-89b9-42c2977ae9a8\") " pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 22:00:12.704737 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:12.704692 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 13 22:00:12.704845 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:12.704737 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 22:00:12.704845 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:12.704757 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09fdf756-6037-48d0-a284-719f7770ade3-original-pull-secret podName:09fdf756-6037-48d0-a284-719f7770ade3 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:44.70473944 +0000 UTC m=+66.155294954 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/09fdf756-6037-48d0-a284-719f7770ade3-original-pull-secret") pod "global-pull-secret-syncer-zkrhd" (UID: "09fdf756-6037-48d0-a284-719f7770ade3") : object "kube-system"/"original-pull-secret" not registered Jul 13 22:00:12.704845 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:12.704779 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a48e3e9b-b015-44af-89b9-42c2977ae9a8-metrics-certs podName:a48e3e9b-b015-44af-89b9-42c2977ae9a8 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:44.704767454 +0000 UTC m=+66.155322969 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a48e3e9b-b015-44af-89b9-42c2977ae9a8-metrics-certs") pod "network-metrics-daemon-w8fht" (UID: "a48e3e9b-b015-44af-89b9-42c2977ae9a8") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 13 22:00:13.960236 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:13.960202 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 22:00:13.960236 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:13.960202 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 22:00:13.960803 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:13.960205 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 22:00:13.960803 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:13.960328 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-w8fht" podUID="a48e3e9b-b015-44af-89b9-42c2977ae9a8" Jul 13 22:00:13.960803 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:13.960405 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zkrhd" podUID="09fdf756-6037-48d0-a284-719f7770ade3" Jul 13 22:00:13.960803 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:13.960494 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nbdz9" podUID="71a3685e-3328-41cb-8b35-c8e8e1efe021" Jul 13 22:00:14.394862 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.394790 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-128.ec2.internal" event="NodeReady" Jul 13 22:00:14.395031 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.394929 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 13 22:00:14.433272 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.433236 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-86f7bbf7c5-9h5nl"] Jul 13 22:00:14.437291 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.437263 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:14.441015 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.440948 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 13 22:00:14.441960 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.441934 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 13 22:00:14.442080 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.442029 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 13 22:00:14.442080 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.442073 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s74j5\"" Jul 13 22:00:14.449902 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.449873 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hj2nl"] Jul 13 22:00:14.450287 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.450263 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 13 22:00:14.453796 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.453777 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hj2nl" Jul 13 22:00:14.454336 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.454312 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86f7bbf7c5-9h5nl"] Jul 13 22:00:14.456353 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.456335 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-89pgf\"" Jul 13 22:00:14.456564 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.456543 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 13 22:00:14.456660 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.456565 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 13 22:00:14.456660 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.456614 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 13 22:00:14.464766 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.464748 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hj2nl"] Jul 13 22:00:14.545791 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.545762 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qzmt4"] Jul 13 22:00:14.549374 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.549346 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qzmt4" Jul 13 22:00:14.551774 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.551752 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 13 22:00:14.551894 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.551815 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 13 22:00:14.552000 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.551967 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lvbhm\"" Jul 13 22:00:14.562656 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.562637 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qzmt4"] Jul 13 22:00:14.618279 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.618242 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/505f1712-2250-43ff-a8ab-270f7cc2b859-installation-pull-secrets\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:14.618422 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.618290 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b64f1d4a-95b8-4abb-8a05-8b246fe11842-cert\") pod \"ingress-canary-hj2nl\" (UID: \"b64f1d4a-95b8-4abb-8a05-8b246fe11842\") " pod="openshift-ingress-canary/ingress-canary-hj2nl" Jul 13 22:00:14.618422 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.618341 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-certificates\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:14.618422 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.618376 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/505f1712-2250-43ff-a8ab-270f7cc2b859-ca-trust-extracted\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:14.618422 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.618409 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-tls\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:14.618628 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.618450 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsfqb\" (UniqueName: \"kubernetes.io/projected/b64f1d4a-95b8-4abb-8a05-8b246fe11842-kube-api-access-zsfqb\") pod \"ingress-canary-hj2nl\" (UID: \"b64f1d4a-95b8-4abb-8a05-8b246fe11842\") " pod="openshift-ingress-canary/ingress-canary-hj2nl" Jul 13 22:00:14.618628 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.618478 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/505f1712-2250-43ff-a8ab-270f7cc2b859-trusted-ca\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:14.618628 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.618505 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gttvw\" (UniqueName: \"kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-kube-api-access-gttvw\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:14.618628 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.618594 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-bound-sa-token\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:14.618808 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.618647 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/505f1712-2250-43ff-a8ab-270f7cc2b859-image-registry-private-configuration\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:14.719585 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.719550 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/505f1712-2250-43ff-a8ab-270f7cc2b859-ca-trust-extracted\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:14.719766 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.719595 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06450a5e-03fe-4a9e-a6e8-722e0fc36664-metrics-tls\") pod \"dns-default-qzmt4\" (UID: \"06450a5e-03fe-4a9e-a6e8-722e0fc36664\") " pod="openshift-dns/dns-default-qzmt4" Jul 13 22:00:14.719766 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.719626 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hld92\" (UniqueName: \"kubernetes.io/projected/06450a5e-03fe-4a9e-a6e8-722e0fc36664-kube-api-access-hld92\") pod \"dns-default-qzmt4\" (UID: \"06450a5e-03fe-4a9e-a6e8-722e0fc36664\") " pod="openshift-dns/dns-default-qzmt4" Jul 13 22:00:14.719766 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.719684 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-tls\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:14.719766 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.719720 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsfqb\" (UniqueName: \"kubernetes.io/projected/b64f1d4a-95b8-4abb-8a05-8b246fe11842-kube-api-access-zsfqb\") pod \"ingress-canary-hj2nl\" (UID: \"b64f1d4a-95b8-4abb-8a05-8b246fe11842\") " pod="openshift-ingress-canary/ingress-canary-hj2nl" Jul 13 22:00:14.719766 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.719741 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/505f1712-2250-43ff-a8ab-270f7cc2b859-trusted-ca\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:14.719766 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.719756 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gttvw\" (UniqueName: \"kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-kube-api-access-gttvw\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:14.720104 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.719787 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-bound-sa-token\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:14.720104 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:14.719794 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 22:00:14.720104 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.719811 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06450a5e-03fe-4a9e-a6e8-722e0fc36664-tmp-dir\") pod \"dns-default-qzmt4\" (UID: \"06450a5e-03fe-4a9e-a6e8-722e0fc36664\") " pod="openshift-dns/dns-default-qzmt4" Jul 13 22:00:14.720104 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:14.719820 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f7bbf7c5-9h5nl: secret "image-registry-tls" not found Jul 13 22:00:14.720104 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.719838 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/505f1712-2250-43ff-a8ab-270f7cc2b859-image-registry-private-configuration\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:14.720104 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.719863 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/06450a5e-03fe-4a9e-a6e8-722e0fc36664-config-volume\") pod \"dns-default-qzmt4\" (UID: \"06450a5e-03fe-4a9e-a6e8-722e0fc36664\") " pod="openshift-dns/dns-default-qzmt4" Jul 13 22:00:14.720104 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.719907 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/505f1712-2250-43ff-a8ab-270f7cc2b859-installation-pull-secrets\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:14.720104 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.719943 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b64f1d4a-95b8-4abb-8a05-8b246fe11842-cert\") pod \"ingress-canary-hj2nl\" (UID: \"b64f1d4a-95b8-4abb-8a05-8b246fe11842\") " pod="openshift-ingress-canary/ingress-canary-hj2nl" Jul 13 22:00:14.720104 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.720016 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-certificates\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:14.720104 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.720032 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/505f1712-2250-43ff-a8ab-270f7cc2b859-ca-trust-extracted\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:14.720104 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:14.720109 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-tls podName:505f1712-2250-43ff-a8ab-270f7cc2b859 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:15.220077762 +0000 UTC m=+36.670633290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-tls") pod "image-registry-86f7bbf7c5-9h5nl" (UID: "505f1712-2250-43ff-a8ab-270f7cc2b859") : secret "image-registry-tls" not found Jul 13 22:00:14.720707 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.720682 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-certificates\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:14.721464 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.721304 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/505f1712-2250-43ff-a8ab-270f7cc2b859-trusted-ca\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:14.724782 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.724760 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/505f1712-2250-43ff-a8ab-270f7cc2b859-installation-pull-secrets\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:14.724880 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.724819 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/505f1712-2250-43ff-a8ab-270f7cc2b859-image-registry-private-configuration\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:14.724880 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.724867 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b64f1d4a-95b8-4abb-8a05-8b246fe11842-cert\") pod \"ingress-canary-hj2nl\" (UID: \"b64f1d4a-95b8-4abb-8a05-8b246fe11842\") " pod="openshift-ingress-canary/ingress-canary-hj2nl" Jul 13 22:00:14.728912 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.728891 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-bound-sa-token\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:14.729016 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.728934 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsfqb\" (UniqueName: \"kubernetes.io/projected/b64f1d4a-95b8-4abb-8a05-8b246fe11842-kube-api-access-zsfqb\") pod \"ingress-canary-hj2nl\" (UID: \"b64f1d4a-95b8-4abb-8a05-8b246fe11842\") " pod="openshift-ingress-canary/ingress-canary-hj2nl" Jul 13 22:00:14.729441 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.729397 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gttvw\" (UniqueName: \"kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-kube-api-access-gttvw\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:14.764374 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.764340 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hj2nl" Jul 13 22:00:14.821078 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.821047 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/06450a5e-03fe-4a9e-a6e8-722e0fc36664-config-volume\") pod \"dns-default-qzmt4\" (UID: \"06450a5e-03fe-4a9e-a6e8-722e0fc36664\") " pod="openshift-dns/dns-default-qzmt4" Jul 13 22:00:14.821191 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.821151 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06450a5e-03fe-4a9e-a6e8-722e0fc36664-metrics-tls\") pod \"dns-default-qzmt4\" (UID: \"06450a5e-03fe-4a9e-a6e8-722e0fc36664\") " pod="openshift-dns/dns-default-qzmt4" Jul 13 22:00:14.821191 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.821181 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hld92\" (UniqueName: \"kubernetes.io/projected/06450a5e-03fe-4a9e-a6e8-722e0fc36664-kube-api-access-hld92\") pod \"dns-default-qzmt4\" (UID: \"06450a5e-03fe-4a9e-a6e8-722e0fc36664\") " pod="openshift-dns/dns-default-qzmt4" Jul 13 22:00:14.821301 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.821252 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06450a5e-03fe-4a9e-a6e8-722e0fc36664-tmp-dir\") pod \"dns-default-qzmt4\" (UID: \"06450a5e-03fe-4a9e-a6e8-722e0fc36664\") " pod="openshift-dns/dns-default-qzmt4" Jul 13 22:00:14.821351 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:14.821302 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 22:00:14.821391 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:14.821373 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06450a5e-03fe-4a9e-a6e8-722e0fc36664-metrics-tls podName:06450a5e-03fe-4a9e-a6e8-722e0fc36664 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:15.321352732 +0000 UTC m=+36.771908246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06450a5e-03fe-4a9e-a6e8-722e0fc36664-metrics-tls") pod "dns-default-qzmt4" (UID: "06450a5e-03fe-4a9e-a6e8-722e0fc36664") : secret "dns-default-metrics-tls" not found Jul 13 22:00:14.821566 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.821537 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06450a5e-03fe-4a9e-a6e8-722e0fc36664-tmp-dir\") pod \"dns-default-qzmt4\" (UID: \"06450a5e-03fe-4a9e-a6e8-722e0fc36664\") " pod="openshift-dns/dns-default-qzmt4" Jul 13 22:00:14.821677 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.821659 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/06450a5e-03fe-4a9e-a6e8-722e0fc36664-config-volume\") pod \"dns-default-qzmt4\" (UID: \"06450a5e-03fe-4a9e-a6e8-722e0fc36664\") " pod="openshift-dns/dns-default-qzmt4" Jul 13 22:00:14.830786 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:14.830760 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hld92\" (UniqueName: \"kubernetes.io/projected/06450a5e-03fe-4a9e-a6e8-722e0fc36664-kube-api-access-hld92\") pod \"dns-default-qzmt4\" (UID: \"06450a5e-03fe-4a9e-a6e8-722e0fc36664\") " pod="openshift-dns/dns-default-qzmt4" Jul 13 22:00:15.224849 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:15.224809 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-tls\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:15.225423 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:15.225009 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 22:00:15.225423 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:15.225031 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f7bbf7c5-9h5nl: secret "image-registry-tls" not found Jul 13 22:00:15.225423 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:15.225089 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-tls podName:505f1712-2250-43ff-a8ab-270f7cc2b859 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:16.225071324 +0000 UTC m=+37.675626840 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-tls") pod "image-registry-86f7bbf7c5-9h5nl" (UID: "505f1712-2250-43ff-a8ab-270f7cc2b859") : secret "image-registry-tls" not found Jul 13 22:00:15.325441 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:15.325408 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06450a5e-03fe-4a9e-a6e8-722e0fc36664-metrics-tls\") pod \"dns-default-qzmt4\" (UID: \"06450a5e-03fe-4a9e-a6e8-722e0fc36664\") " pod="openshift-dns/dns-default-qzmt4" Jul 13 22:00:15.325626 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:15.325521 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 22:00:15.325626 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:15.325585 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06450a5e-03fe-4a9e-a6e8-722e0fc36664-metrics-tls podName:06450a5e-03fe-4a9e-a6e8-722e0fc36664 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:16.325568803 +0000 UTC m=+37.776124319 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06450a5e-03fe-4a9e-a6e8-722e0fc36664-metrics-tls") pod "dns-default-qzmt4" (UID: "06450a5e-03fe-4a9e-a6e8-722e0fc36664") : secret "dns-default-metrics-tls" not found Jul 13 22:00:15.740217 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:15.740074 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hj2nl"] Jul 13 22:00:15.743935 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:00:15.743910 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64f1d4a_95b8_4abb_8a05_8b246fe11842.slice/crio-aed4b5afdf33e340af747ab70c7db0d52c0c905728e7a7f5db6b4276a55ff0cf WatchSource:0}: Error finding container aed4b5afdf33e340af747ab70c7db0d52c0c905728e7a7f5db6b4276a55ff0cf: Status 404 returned error can't find the container with id aed4b5afdf33e340af747ab70c7db0d52c0c905728e7a7f5db6b4276a55ff0cf Jul 13 22:00:15.960077 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:15.960041 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 22:00:15.960077 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:15.960076 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 22:00:15.960274 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:15.960044 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 22:00:15.962743 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:15.962723 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 13 22:00:15.962866 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:15.962722 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 13 22:00:15.963281 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:15.963267 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 13 22:00:15.963512 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:15.963500 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gmvbg\"" Jul 13 22:00:15.963564 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:15.963505 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 13 22:00:15.963564 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:15.963531 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b6tsw\"" Jul 13 22:00:16.153422 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:16.153337 2562 generic.go:356] "Generic (PLEG): container finished" podID="75ac4b1c-ecb1-4550-84ab-7220e7445349" containerID="02afe2ac9a48f53c3c0967e4b1eb5b964d3ac1a8c0bae7ec09ad9a60e82457ac" exitCode=0 Jul 13 22:00:16.153549 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:16.153413 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lvr8p" event={"ID":"75ac4b1c-ecb1-4550-84ab-7220e7445349","Type":"ContainerDied","Data":"02afe2ac9a48f53c3c0967e4b1eb5b964d3ac1a8c0bae7ec09ad9a60e82457ac"} Jul 13 22:00:16.157068 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:16.157040 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hj2nl" event={"ID":"b64f1d4a-95b8-4abb-8a05-8b246fe11842","Type":"ContainerStarted","Data":"aed4b5afdf33e340af747ab70c7db0d52c0c905728e7a7f5db6b4276a55ff0cf"} Jul 13 22:00:16.235454 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:16.235407 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-tls\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:16.235930 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:16.235566 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 22:00:16.235930 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:16.235583 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f7bbf7c5-9h5nl: secret "image-registry-tls" not found Jul 13 22:00:16.235930 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:16.235634 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-tls podName:505f1712-2250-43ff-a8ab-270f7cc2b859 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:18.235615051 +0000 UTC m=+39.686170566 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-tls") pod "image-registry-86f7bbf7c5-9h5nl" (UID: "505f1712-2250-43ff-a8ab-270f7cc2b859") : secret "image-registry-tls" not found Jul 13 22:00:16.336229 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:16.336199 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06450a5e-03fe-4a9e-a6e8-722e0fc36664-metrics-tls\") pod \"dns-default-qzmt4\" (UID: \"06450a5e-03fe-4a9e-a6e8-722e0fc36664\") " pod="openshift-dns/dns-default-qzmt4" Jul 13 22:00:16.337131 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:16.336528 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 22:00:16.337131 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:16.336597 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06450a5e-03fe-4a9e-a6e8-722e0fc36664-metrics-tls podName:06450a5e-03fe-4a9e-a6e8-722e0fc36664 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:18.336578184 +0000 UTC m=+39.787133700 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06450a5e-03fe-4a9e-a6e8-722e0fc36664-metrics-tls") pod "dns-default-qzmt4" (UID: "06450a5e-03fe-4a9e-a6e8-722e0fc36664") : secret "dns-default-metrics-tls" not found Jul 13 22:00:17.162790 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:17.162760 2562 generic.go:356] "Generic (PLEG): container finished" podID="75ac4b1c-ecb1-4550-84ab-7220e7445349" containerID="f95ac71faf9f3eb7235e08722ae1df8a2538ed323fac35cd6ca3c88a00cdb4ee" exitCode=0 Jul 13 22:00:17.162950 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:17.162796 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lvr8p" event={"ID":"75ac4b1c-ecb1-4550-84ab-7220e7445349","Type":"ContainerDied","Data":"f95ac71faf9f3eb7235e08722ae1df8a2538ed323fac35cd6ca3c88a00cdb4ee"} Jul 13 22:00:18.168047 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:18.168014 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lvr8p" event={"ID":"75ac4b1c-ecb1-4550-84ab-7220e7445349","Type":"ContainerStarted","Data":"d93a6dd12befb86929020193999ec484dffcafca3f694653308fac94cb579b00"} Jul 13 22:00:18.254119 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:18.254089 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-tls\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:18.254265 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:18.254250 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 22:00:18.254322 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:18.254267 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f7bbf7c5-9h5nl: secret "image-registry-tls" not found Jul 13 22:00:18.254322 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:18.254322 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-tls podName:505f1712-2250-43ff-a8ab-270f7cc2b859 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:22.254307341 +0000 UTC m=+43.704862856 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-tls") pod "image-registry-86f7bbf7c5-9h5nl" (UID: "505f1712-2250-43ff-a8ab-270f7cc2b859") : secret "image-registry-tls" not found Jul 13 22:00:18.355189 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:18.355162 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06450a5e-03fe-4a9e-a6e8-722e0fc36664-metrics-tls\") pod \"dns-default-qzmt4\" (UID: \"06450a5e-03fe-4a9e-a6e8-722e0fc36664\") " pod="openshift-dns/dns-default-qzmt4" Jul 13 22:00:18.355339 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:18.355316 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 22:00:18.355476 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:18.355380 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06450a5e-03fe-4a9e-a6e8-722e0fc36664-metrics-tls podName:06450a5e-03fe-4a9e-a6e8-722e0fc36664 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:22.355359468 +0000 UTC m=+43.805914986 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06450a5e-03fe-4a9e-a6e8-722e0fc36664-metrics-tls") pod "dns-default-qzmt4" (UID: "06450a5e-03fe-4a9e-a6e8-722e0fc36664") : secret "dns-default-metrics-tls" not found Jul 13 22:00:19.171476 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.171440 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hj2nl" event={"ID":"b64f1d4a-95b8-4abb-8a05-8b246fe11842","Type":"ContainerStarted","Data":"a4bb08287531ac97907208cc22ca305f37e5c88b6a925b152bee1ca0e24e50a9"} Jul 13 22:00:19.191255 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.191210 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hj2nl" podStartSLOduration=2.582385698 podStartE2EDuration="5.191196867s" totalImagesPullingTime="2.608811169s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:19.190835758 +0000 UTC m=+40.641391288" watchObservedRunningTime="2026-07-13 22:00:19.191196867 +0000 UTC m=+40.641752431" Jul 13 22:00:19.191409 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.191391 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lvr8p" podStartSLOduration=9.727473992 podStartE2EDuration="40.191385549s" totalImagesPullingTime="30.463911557s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:59:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:18.19586151 +0000 UTC m=+39.646417047" watchObservedRunningTime="2026-07-13 22:00:19.191385549 +0000 UTC m=+40.641941085" Jul 13 22:00:19.261050 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.261016 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-dlj2d"] Jul 13 22:00:19.283570 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.283545 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-g497d"] Jul 13 22:00:19.283718 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.283699 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-dlj2d" Jul 13 22:00:19.286804 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.286779 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 13 22:00:19.286937 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.286804 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 13 22:00:19.287132 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.287112 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 13 22:00:19.287222 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.287119 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 13 22:00:19.287222 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.287175 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wwdv5\"" Jul 13 22:00:19.307581 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.307561 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-dlj2d"] Jul 13 22:00:19.307666 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.307598 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-g497d"] Jul 13 22:00:19.307704 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.307668 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-g497d" Jul 13 22:00:19.309795 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.309777 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 13 22:00:19.309795 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.309782 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 13 22:00:19.309917 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.309826 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-hc7lf\"" Jul 13 22:00:19.364633 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.364613 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-gwtpb"] Jul 13 22:00:19.385518 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.385495 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qhgw4"] Jul 13 22:00:19.385678 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.385644 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-gwtpb" Jul 13 22:00:19.388369 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.388350 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-jgmw6\"" Jul 13 22:00:19.409656 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.409629 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-5vww5"] Jul 13 22:00:19.409805 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.409748 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qhgw4" Jul 13 22:00:19.412368 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.412341 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-dm4wb\"" Jul 13 22:00:19.412466 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.412425 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 13 22:00:19.412509 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.412491 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 13 22:00:19.412767 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.412491 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 13 22:00:19.413305 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.413279 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 13 22:00:19.425648 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.425602 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5dbcf998b4-7qqv9"] Jul 13 22:00:19.425720 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.425707 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-5vww5" Jul 13 22:00:19.428514 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.428496 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 13 22:00:19.429538 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.429522 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 13 22:00:19.429845 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.429821 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 13 22:00:19.430439 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.430423 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xfmv6\"" Jul 13 22:00:19.430600 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.430587 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 13 22:00:19.436510 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.436493 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 13 22:00:19.447864 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.447840 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-x5qhq"] Jul 13 22:00:19.447966 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.447950 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:19.451110 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.451090 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 13 22:00:19.451110 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.451102 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 13 22:00:19.451261 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.451112 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 13 22:00:19.451261 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.451244 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xxrfv\"" Jul 13 22:00:19.451353 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.451339 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 13 22:00:19.451401 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.451355 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 13 22:00:19.451695 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.451682 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 13 22:00:19.464119 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.464091 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e9588e99-3cae-44f7-82b6-101ae1fbd604-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-dlj2d\" (UID: \"e9588e99-3cae-44f7-82b6-101ae1fbd604\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-dlj2d" Jul 13 22:00:19.464218 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.464132 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7brn9\" (UniqueName: \"kubernetes.io/projected/b9117cd6-d8c5-43a8-9a5d-8b5a349a5943-kube-api-access-7brn9\") pod \"volume-data-source-validator-78495597f6-g497d\" (UID: \"b9117cd6-d8c5-43a8-9a5d-8b5a349a5943\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-g497d" Jul 13 22:00:19.464218 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.464202 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f9s4\" (UniqueName: \"kubernetes.io/projected/e9588e99-3cae-44f7-82b6-101ae1fbd604-kube-api-access-7f9s4\") pod \"cluster-monitoring-operator-5f7f494dd4-dlj2d\" (UID: \"e9588e99-3cae-44f7-82b6-101ae1fbd604\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-dlj2d" Jul 13 22:00:19.464303 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.464260 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e9588e99-3cae-44f7-82b6-101ae1fbd604-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-dlj2d\" (UID: \"e9588e99-3cae-44f7-82b6-101ae1fbd604\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-dlj2d" Jul 13 22:00:19.468076 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.468056 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-gwtpb"] Jul 13 22:00:19.468147 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.468089 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qhgw4"] Jul 13 22:00:19.468147 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.468104 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-5vww5"] Jul 13 22:00:19.468147 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.468118 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5dbcf998b4-7qqv9"] Jul 13 22:00:19.468147 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.468133 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-x5qhq"] Jul 13 22:00:19.468273 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.468161 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c6cfc"] Jul 13 22:00:19.468273 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.468174 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-x5qhq" Jul 13 22:00:19.470469 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.470451 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 13 22:00:19.471056 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.471038 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 13 22:00:19.471191 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.471047 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-hxfpk\"" Jul 13 22:00:19.472225 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.472209 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 13 22:00:19.472637 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.472625 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 13 22:00:19.481059 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.481038 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 13 22:00:19.490036 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.490016 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c6cfc"] Jul 13 22:00:19.490129 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.490065 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c6cfc" Jul 13 22:00:19.493001 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.492984 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zfb5v\"" Jul 13 22:00:19.493102 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.492984 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 13 22:00:19.493280 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.493265 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 13 22:00:19.493347 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.493280 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 13 22:00:19.563387 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.563354 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rcbqx"] Jul 13 22:00:19.565382 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.565363 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f90dbf0b-63cf-41a1-971c-76d47f6d859b-config\") pod \"service-ca-operator-7695fc6c8c-qhgw4\" (UID: \"f90dbf0b-63cf-41a1-971c-76d47f6d859b\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qhgw4" Jul 13 22:00:19.565444 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.565411 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e9588e99-3cae-44f7-82b6-101ae1fbd604-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-dlj2d\" (UID: \"e9588e99-3cae-44f7-82b6-101ae1fbd604\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-dlj2d" Jul 13 22:00:19.565444 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.565437 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3815ceb2-a3b8-43bc-bf04-b1dd4adc4412-tmp\") pod \"insights-operator-68d9dd94cf-5vww5\" (UID: \"3815ceb2-a3b8-43bc-bf04-b1dd4adc4412\") " pod="openshift-insights/insights-operator-68d9dd94cf-5vww5" Jul 13 22:00:19.565509 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.565454 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3815ceb2-a3b8-43bc-bf04-b1dd4adc4412-serving-cert\") pod \"insights-operator-68d9dd94cf-5vww5\" (UID: \"3815ceb2-a3b8-43bc-bf04-b1dd4adc4412\") " pod="openshift-insights/insights-operator-68d9dd94cf-5vww5" Jul 13 22:00:19.565509 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.565472 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e9588e99-3cae-44f7-82b6-101ae1fbd604-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-dlj2d\" (UID: \"e9588e99-3cae-44f7-82b6-101ae1fbd604\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-dlj2d" Jul 13 22:00:19.565509 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.565499 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f9s4\" (UniqueName: \"kubernetes.io/projected/e9588e99-3cae-44f7-82b6-101ae1fbd604-kube-api-access-7f9s4\") pod \"cluster-monitoring-operator-5f7f494dd4-dlj2d\" (UID: \"e9588e99-3cae-44f7-82b6-101ae1fbd604\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-dlj2d" Jul 13 22:00:19.565653 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.565526 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dbedfa07-690d-4c79-92d3-dc85b88124d7-stats-auth\") pod \"router-default-5dbcf998b4-7qqv9\" (UID: \"dbedfa07-690d-4c79-92d3-dc85b88124d7\") " pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:19.565653 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.565541 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b1fa75b-f29d-45a1-af98-fa947c2058b2-trusted-ca\") pod \"console-operator-7866497494-x5qhq\" (UID: \"7b1fa75b-f29d-45a1-af98-fa947c2058b2\") " pod="openshift-console-operator/console-operator-7866497494-x5qhq" Jul 13 22:00:19.565653 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.565558 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f90dbf0b-63cf-41a1-971c-76d47f6d859b-serving-cert\") pod \"service-ca-operator-7695fc6c8c-qhgw4\" (UID: \"f90dbf0b-63cf-41a1-971c-76d47f6d859b\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qhgw4" Jul 13 22:00:19.565653 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.565592 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pnxr\" (UniqueName: \"kubernetes.io/projected/f90dbf0b-63cf-41a1-971c-76d47f6d859b-kube-api-access-6pnxr\") pod \"service-ca-operator-7695fc6c8c-qhgw4\" (UID: \"f90dbf0b-63cf-41a1-971c-76d47f6d859b\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qhgw4" Jul 13 22:00:19.565653 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.565608 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b1fa75b-f29d-45a1-af98-fa947c2058b2-serving-cert\") pod \"console-operator-7866497494-x5qhq\" (UID: \"7b1fa75b-f29d-45a1-af98-fa947c2058b2\") " pod="openshift-console-operator/console-operator-7866497494-x5qhq" Jul 13 22:00:19.565653 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:19.565641 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 22:00:19.565911 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.565660 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3815ceb2-a3b8-43bc-bf04-b1dd4adc4412-snapshots\") pod \"insights-operator-68d9dd94cf-5vww5\" (UID: \"3815ceb2-a3b8-43bc-bf04-b1dd4adc4412\") " pod="openshift-insights/insights-operator-68d9dd94cf-5vww5" Jul 13 22:00:19.565911 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.565710 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3815ceb2-a3b8-43bc-bf04-b1dd4adc4412-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-5vww5\" (UID: \"3815ceb2-a3b8-43bc-bf04-b1dd4adc4412\") " pod="openshift-insights/insights-operator-68d9dd94cf-5vww5" Jul 13 22:00:19.565911 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:19.565753 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9588e99-3cae-44f7-82b6-101ae1fbd604-cluster-monitoring-operator-tls podName:e9588e99-3cae-44f7-82b6-101ae1fbd604 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:20.065733306 +0000 UTC m=+41.516288824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e9588e99-3cae-44f7-82b6-101ae1fbd604-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-dlj2d" (UID: "e9588e99-3cae-44f7-82b6-101ae1fbd604") : secret "cluster-monitoring-operator-tls" not found Jul 13 22:00:19.565911 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.565822 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtt6h\" (UniqueName: \"kubernetes.io/projected/7b1fa75b-f29d-45a1-af98-fa947c2058b2-kube-api-access-xtt6h\") pod \"console-operator-7866497494-x5qhq\" (UID: \"7b1fa75b-f29d-45a1-af98-fa947c2058b2\") " pod="openshift-console-operator/console-operator-7866497494-x5qhq" Jul 13 22:00:19.565911 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.565846 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gm7p\" (UniqueName: \"kubernetes.io/projected/aca2745d-9fd1-4064-9448-abcdb92715af-kube-api-access-4gm7p\") pod \"network-check-source-869dcfcdcd-gwtpb\" (UID: \"aca2745d-9fd1-4064-9448-abcdb92715af\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-gwtpb" Jul 13 22:00:19.565911 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.565865 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbedfa07-690d-4c79-92d3-dc85b88124d7-metrics-certs\") pod \"router-default-5dbcf998b4-7qqv9\" (UID: \"dbedfa07-690d-4c79-92d3-dc85b88124d7\") " pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:19.565911 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.565888 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b1fa75b-f29d-45a1-af98-fa947c2058b2-config\") pod \"console-operator-7866497494-x5qhq\" (UID: \"7b1fa75b-f29d-45a1-af98-fa947c2058b2\") " pod="openshift-console-operator/console-operator-7866497494-x5qhq" Jul 13 22:00:19.565911 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.565913 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7brn9\" (UniqueName: \"kubernetes.io/projected/b9117cd6-d8c5-43a8-9a5d-8b5a349a5943-kube-api-access-7brn9\") pod \"volume-data-source-validator-78495597f6-g497d\" (UID: \"b9117cd6-d8c5-43a8-9a5d-8b5a349a5943\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-g497d" Jul 13 22:00:19.566293 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.565928 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbedfa07-690d-4c79-92d3-dc85b88124d7-service-ca-bundle\") pod \"router-default-5dbcf998b4-7qqv9\" (UID: \"dbedfa07-690d-4c79-92d3-dc85b88124d7\") " pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:19.566293 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.565955 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbrzl\" (UniqueName: \"kubernetes.io/projected/dbedfa07-690d-4c79-92d3-dc85b88124d7-kube-api-access-vbrzl\") pod \"router-default-5dbcf998b4-7qqv9\" (UID: \"dbedfa07-690d-4c79-92d3-dc85b88124d7\") " pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:19.566293 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.565994 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3815ceb2-a3b8-43bc-bf04-b1dd4adc4412-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-5vww5\" (UID: \"3815ceb2-a3b8-43bc-bf04-b1dd4adc4412\") " pod="openshift-insights/insights-operator-68d9dd94cf-5vww5" Jul 13 22:00:19.566293 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.566022 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dbedfa07-690d-4c79-92d3-dc85b88124d7-default-certificate\") pod \"router-default-5dbcf998b4-7qqv9\" (UID: \"dbedfa07-690d-4c79-92d3-dc85b88124d7\") " pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:19.566293 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.566069 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkvk7\" (UniqueName: \"kubernetes.io/projected/3815ceb2-a3b8-43bc-bf04-b1dd4adc4412-kube-api-access-kkvk7\") pod \"insights-operator-68d9dd94cf-5vww5\" (UID: \"3815ceb2-a3b8-43bc-bf04-b1dd4adc4412\") " pod="openshift-insights/insights-operator-68d9dd94cf-5vww5" Jul 13 22:00:19.566293 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.566206 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e9588e99-3cae-44f7-82b6-101ae1fbd604-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-dlj2d\" (UID: \"e9588e99-3cae-44f7-82b6-101ae1fbd604\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-dlj2d" Jul 13 22:00:19.581635 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.581607 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f9s4\" (UniqueName: \"kubernetes.io/projected/e9588e99-3cae-44f7-82b6-101ae1fbd604-kube-api-access-7f9s4\") pod \"cluster-monitoring-operator-5f7f494dd4-dlj2d\" (UID: \"e9588e99-3cae-44f7-82b6-101ae1fbd604\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-dlj2d" Jul 13 22:00:19.581749 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.581658 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7brn9\" (UniqueName: \"kubernetes.io/projected/b9117cd6-d8c5-43a8-9a5d-8b5a349a5943-kube-api-access-7brn9\") pod \"volume-data-source-validator-78495597f6-g497d\" (UID: \"b9117cd6-d8c5-43a8-9a5d-8b5a349a5943\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-g497d" Jul 13 22:00:19.584179 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.584161 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rcbqx"] Jul 13 22:00:19.584253 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.584243 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rcbqx" Jul 13 22:00:19.586492 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.586447 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 13 22:00:19.586647 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.586493 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 13 22:00:19.586647 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.586526 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 13 22:00:19.586647 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.586554 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-w9nc5\"" Jul 13 22:00:19.586647 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.586636 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 13 22:00:19.615225 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.615196 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-g497d" Jul 13 22:00:19.666646 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.666603 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dbedfa07-690d-4c79-92d3-dc85b88124d7-stats-auth\") pod \"router-default-5dbcf998b4-7qqv9\" (UID: \"dbedfa07-690d-4c79-92d3-dc85b88124d7\") " pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:19.666646 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.666648 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b1fa75b-f29d-45a1-af98-fa947c2058b2-trusted-ca\") pod \"console-operator-7866497494-x5qhq\" (UID: \"7b1fa75b-f29d-45a1-af98-fa947c2058b2\") " pod="openshift-console-operator/console-operator-7866497494-x5qhq" Jul 13 22:00:19.666839 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.666667 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f90dbf0b-63cf-41a1-971c-76d47f6d859b-serving-cert\") pod \"service-ca-operator-7695fc6c8c-qhgw4\" (UID: \"f90dbf0b-63cf-41a1-971c-76d47f6d859b\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qhgw4" Jul 13 22:00:19.666931 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.666904 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh6dl\" (UniqueName: \"kubernetes.io/projected/555d9e2d-52a7-4024-aa67-991a477e5ab7-kube-api-access-hh6dl\") pod \"cluster-samples-operator-65db858894-c6cfc\" (UID: \"555d9e2d-52a7-4024-aa67-991a477e5ab7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c6cfc" Jul 13 22:00:19.667057 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.667026 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pnxr\" (UniqueName: \"kubernetes.io/projected/f90dbf0b-63cf-41a1-971c-76d47f6d859b-kube-api-access-6pnxr\") pod \"service-ca-operator-7695fc6c8c-qhgw4\" (UID: \"f90dbf0b-63cf-41a1-971c-76d47f6d859b\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qhgw4" Jul 13 22:00:19.667137 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.667105 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b1fa75b-f29d-45a1-af98-fa947c2058b2-serving-cert\") pod \"console-operator-7866497494-x5qhq\" (UID: \"7b1fa75b-f29d-45a1-af98-fa947c2058b2\") " pod="openshift-console-operator/console-operator-7866497494-x5qhq" Jul 13 22:00:19.667195 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.667149 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3815ceb2-a3b8-43bc-bf04-b1dd4adc4412-snapshots\") pod \"insights-operator-68d9dd94cf-5vww5\" (UID: \"3815ceb2-a3b8-43bc-bf04-b1dd4adc4412\") " pod="openshift-insights/insights-operator-68d9dd94cf-5vww5" Jul 13 22:00:19.667195 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.667187 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3815ceb2-a3b8-43bc-bf04-b1dd4adc4412-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-5vww5\" (UID: \"3815ceb2-a3b8-43bc-bf04-b1dd4adc4412\") " pod="openshift-insights/insights-operator-68d9dd94cf-5vww5" Jul 13 22:00:19.667289 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.667221 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtt6h\" (UniqueName: \"kubernetes.io/projected/7b1fa75b-f29d-45a1-af98-fa947c2058b2-kube-api-access-xtt6h\") pod \"console-operator-7866497494-x5qhq\" (UID: \"7b1fa75b-f29d-45a1-af98-fa947c2058b2\") " pod="openshift-console-operator/console-operator-7866497494-x5qhq" Jul 13 22:00:19.667289 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.667258 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gm7p\" (UniqueName: \"kubernetes.io/projected/aca2745d-9fd1-4064-9448-abcdb92715af-kube-api-access-4gm7p\") pod \"network-check-source-869dcfcdcd-gwtpb\" (UID: \"aca2745d-9fd1-4064-9448-abcdb92715af\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-gwtpb" Jul 13 22:00:19.667289 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.667282 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbedfa07-690d-4c79-92d3-dc85b88124d7-metrics-certs\") pod \"router-default-5dbcf998b4-7qqv9\" (UID: \"dbedfa07-690d-4c79-92d3-dc85b88124d7\") " pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:19.667430 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.667312 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b1fa75b-f29d-45a1-af98-fa947c2058b2-config\") pod \"console-operator-7866497494-x5qhq\" (UID: \"7b1fa75b-f29d-45a1-af98-fa947c2058b2\") " pod="openshift-console-operator/console-operator-7866497494-x5qhq" Jul 13 22:00:19.667430 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.667346 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbedfa07-690d-4c79-92d3-dc85b88124d7-service-ca-bundle\") pod \"router-default-5dbcf998b4-7qqv9\" (UID: \"dbedfa07-690d-4c79-92d3-dc85b88124d7\") " pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:19.667430 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.667379 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbrzl\" (UniqueName: \"kubernetes.io/projected/dbedfa07-690d-4c79-92d3-dc85b88124d7-kube-api-access-vbrzl\") pod \"router-default-5dbcf998b4-7qqv9\" (UID: \"dbedfa07-690d-4c79-92d3-dc85b88124d7\") " pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:19.667430 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.667403 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3815ceb2-a3b8-43bc-bf04-b1dd4adc4412-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-5vww5\" (UID: \"3815ceb2-a3b8-43bc-bf04-b1dd4adc4412\") " pod="openshift-insights/insights-operator-68d9dd94cf-5vww5" Jul 13 22:00:19.667430 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.667427 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dbedfa07-690d-4c79-92d3-dc85b88124d7-default-certificate\") pod \"router-default-5dbcf998b4-7qqv9\" (UID: \"dbedfa07-690d-4c79-92d3-dc85b88124d7\") " pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:19.667653 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.667606 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkvk7\" (UniqueName: \"kubernetes.io/projected/3815ceb2-a3b8-43bc-bf04-b1dd4adc4412-kube-api-access-kkvk7\") pod \"insights-operator-68d9dd94cf-5vww5\" (UID: \"3815ceb2-a3b8-43bc-bf04-b1dd4adc4412\") " pod="openshift-insights/insights-operator-68d9dd94cf-5vww5" Jul 13 22:00:19.667653 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.667643 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f90dbf0b-63cf-41a1-971c-76d47f6d859b-config\") pod \"service-ca-operator-7695fc6c8c-qhgw4\" (UID: \"f90dbf0b-63cf-41a1-971c-76d47f6d859b\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qhgw4" Jul 13 22:00:19.667759 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.667708 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3815ceb2-a3b8-43bc-bf04-b1dd4adc4412-tmp\") pod \"insights-operator-68d9dd94cf-5vww5\" (UID: \"3815ceb2-a3b8-43bc-bf04-b1dd4adc4412\") " pod="openshift-insights/insights-operator-68d9dd94cf-5vww5" Jul 13 22:00:19.667759 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.667735 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3815ceb2-a3b8-43bc-bf04-b1dd4adc4412-serving-cert\") pod \"insights-operator-68d9dd94cf-5vww5\" (UID: \"3815ceb2-a3b8-43bc-bf04-b1dd4adc4412\") " pod="openshift-insights/insights-operator-68d9dd94cf-5vww5" Jul 13 22:00:19.667855 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.667808 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/555d9e2d-52a7-4024-aa67-991a477e5ab7-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-c6cfc\" (UID: \"555d9e2d-52a7-4024-aa67-991a477e5ab7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c6cfc" Jul 13 22:00:19.668026 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.667992 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b1fa75b-f29d-45a1-af98-fa947c2058b2-trusted-ca\") pod \"console-operator-7866497494-x5qhq\" (UID: \"7b1fa75b-f29d-45a1-af98-fa947c2058b2\") " pod="openshift-console-operator/console-operator-7866497494-x5qhq" Jul 13 22:00:19.669187 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:19.668245 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dbedfa07-690d-4c79-92d3-dc85b88124d7-service-ca-bundle podName:dbedfa07-690d-4c79-92d3-dc85b88124d7 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:20.168224025 +0000 UTC m=+41.618779561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dbedfa07-690d-4c79-92d3-dc85b88124d7-service-ca-bundle") pod "router-default-5dbcf998b4-7qqv9" (UID: "dbedfa07-690d-4c79-92d3-dc85b88124d7") : configmap references non-existent config key: service-ca.crt Jul 13 22:00:19.669187 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.668536 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f90dbf0b-63cf-41a1-971c-76d47f6d859b-config\") pod \"service-ca-operator-7695fc6c8c-qhgw4\" (UID: \"f90dbf0b-63cf-41a1-971c-76d47f6d859b\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qhgw4" Jul 13 22:00:19.669187 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.668542 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3815ceb2-a3b8-43bc-bf04-b1dd4adc4412-snapshots\") pod \"insights-operator-68d9dd94cf-5vww5\" (UID: \"3815ceb2-a3b8-43bc-bf04-b1dd4adc4412\") " pod="openshift-insights/insights-operator-68d9dd94cf-5vww5" Jul 13 22:00:19.669187 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.668757 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b1fa75b-f29d-45a1-af98-fa947c2058b2-config\") pod \"console-operator-7866497494-x5qhq\" (UID: \"7b1fa75b-f29d-45a1-af98-fa947c2058b2\") " pod="openshift-console-operator/console-operator-7866497494-x5qhq" Jul 13 22:00:19.669187 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.668825 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3815ceb2-a3b8-43bc-bf04-b1dd4adc4412-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-5vww5\" (UID: \"3815ceb2-a3b8-43bc-bf04-b1dd4adc4412\") " pod="openshift-insights/insights-operator-68d9dd94cf-5vww5" Jul 13 22:00:19.669187 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:19.668855 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 22:00:19.669187 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:19.668907 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbedfa07-690d-4c79-92d3-dc85b88124d7-metrics-certs podName:dbedfa07-690d-4c79-92d3-dc85b88124d7 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:20.16889023 +0000 UTC m=+41.619445745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbedfa07-690d-4c79-92d3-dc85b88124d7-metrics-certs") pod "router-default-5dbcf998b4-7qqv9" (UID: "dbedfa07-690d-4c79-92d3-dc85b88124d7") : secret "router-metrics-certs-default" not found Jul 13 22:00:19.669187 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.668995 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3815ceb2-a3b8-43bc-bf04-b1dd4adc4412-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-5vww5\" (UID: \"3815ceb2-a3b8-43bc-bf04-b1dd4adc4412\") " pod="openshift-insights/insights-operator-68d9dd94cf-5vww5" Jul 13 22:00:19.669187 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.669175 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3815ceb2-a3b8-43bc-bf04-b1dd4adc4412-tmp\") pod \"insights-operator-68d9dd94cf-5vww5\" (UID: \"3815ceb2-a3b8-43bc-bf04-b1dd4adc4412\") " pod="openshift-insights/insights-operator-68d9dd94cf-5vww5" Jul 13 22:00:19.670222 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.670199 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dbedfa07-690d-4c79-92d3-dc85b88124d7-stats-auth\") pod \"router-default-5dbcf998b4-7qqv9\" (UID: \"dbedfa07-690d-4c79-92d3-dc85b88124d7\") " pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:19.670445 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.670424 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f90dbf0b-63cf-41a1-971c-76d47f6d859b-serving-cert\") pod \"service-ca-operator-7695fc6c8c-qhgw4\" (UID: \"f90dbf0b-63cf-41a1-971c-76d47f6d859b\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qhgw4" Jul 13 22:00:19.670536 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.670465 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b1fa75b-f29d-45a1-af98-fa947c2058b2-serving-cert\") pod \"console-operator-7866497494-x5qhq\" (UID: \"7b1fa75b-f29d-45a1-af98-fa947c2058b2\") " pod="openshift-console-operator/console-operator-7866497494-x5qhq" Jul 13 22:00:19.671288 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.671269 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dbedfa07-690d-4c79-92d3-dc85b88124d7-default-certificate\") pod \"router-default-5dbcf998b4-7qqv9\" (UID: \"dbedfa07-690d-4c79-92d3-dc85b88124d7\") " pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:19.671505 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.671484 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3815ceb2-a3b8-43bc-bf04-b1dd4adc4412-serving-cert\") pod \"insights-operator-68d9dd94cf-5vww5\" (UID: \"3815ceb2-a3b8-43bc-bf04-b1dd4adc4412\") " pod="openshift-insights/insights-operator-68d9dd94cf-5vww5" Jul 13 22:00:19.677699 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.677471 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtt6h\" (UniqueName: \"kubernetes.io/projected/7b1fa75b-f29d-45a1-af98-fa947c2058b2-kube-api-access-xtt6h\") pod \"console-operator-7866497494-x5qhq\" (UID: \"7b1fa75b-f29d-45a1-af98-fa947c2058b2\") " pod="openshift-console-operator/console-operator-7866497494-x5qhq" Jul 13 22:00:19.678767 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.677838 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pnxr\" (UniqueName: \"kubernetes.io/projected/f90dbf0b-63cf-41a1-971c-76d47f6d859b-kube-api-access-6pnxr\") pod \"service-ca-operator-7695fc6c8c-qhgw4\" (UID: \"f90dbf0b-63cf-41a1-971c-76d47f6d859b\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qhgw4" Jul 13 22:00:19.679907 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.679564 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gm7p\" (UniqueName: \"kubernetes.io/projected/aca2745d-9fd1-4064-9448-abcdb92715af-kube-api-access-4gm7p\") pod \"network-check-source-869dcfcdcd-gwtpb\" (UID: \"aca2745d-9fd1-4064-9448-abcdb92715af\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-gwtpb" Jul 13 22:00:19.681372 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.681350 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkvk7\" (UniqueName: \"kubernetes.io/projected/3815ceb2-a3b8-43bc-bf04-b1dd4adc4412-kube-api-access-kkvk7\") pod \"insights-operator-68d9dd94cf-5vww5\" (UID: \"3815ceb2-a3b8-43bc-bf04-b1dd4adc4412\") " pod="openshift-insights/insights-operator-68d9dd94cf-5vww5" Jul 13 22:00:19.681540 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.681520 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbrzl\" (UniqueName: \"kubernetes.io/projected/dbedfa07-690d-4c79-92d3-dc85b88124d7-kube-api-access-vbrzl\") pod \"router-default-5dbcf998b4-7qqv9\" (UID: \"dbedfa07-690d-4c79-92d3-dc85b88124d7\") " pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:19.696409 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.696323 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-gwtpb" Jul 13 22:00:19.721397 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.721370 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qhgw4" Jul 13 22:00:19.733993 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.733956 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-5vww5" Jul 13 22:00:19.750358 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.750329 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-g497d"] Jul 13 22:00:19.752714 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:00:19.752681 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9117cd6_d8c5_43a8_9a5d_8b5a349a5943.slice/crio-944df706f19c8ad01c20c05b15da9cde4963b2e131e861dfd7c075ccc2df81ab WatchSource:0}: Error finding container 944df706f19c8ad01c20c05b15da9cde4963b2e131e861dfd7c075ccc2df81ab: Status 404 returned error can't find the container with id 944df706f19c8ad01c20c05b15da9cde4963b2e131e861dfd7c075ccc2df81ab Jul 13 22:00:19.768702 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.768675 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh6dl\" (UniqueName: \"kubernetes.io/projected/555d9e2d-52a7-4024-aa67-991a477e5ab7-kube-api-access-hh6dl\") pod \"cluster-samples-operator-65db858894-c6cfc\" (UID: \"555d9e2d-52a7-4024-aa67-991a477e5ab7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c6cfc" Jul 13 22:00:19.768849 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.768731 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/db233478-500b-472c-919f-510266682def-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-rcbqx\" (UID: \"db233478-500b-472c-919f-510266682def\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rcbqx" Jul 13 22:00:19.768849 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.768798 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn5ds\" (UniqueName: \"kubernetes.io/projected/db233478-500b-472c-919f-510266682def-kube-api-access-tn5ds\") pod \"kube-storage-version-migrator-operator-799cb7f958-rcbqx\" (UID: \"db233478-500b-472c-919f-510266682def\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rcbqx" Jul 13 22:00:19.768849 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.768840 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/db233478-500b-472c-919f-510266682def-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-rcbqx\" (UID: \"db233478-500b-472c-919f-510266682def\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rcbqx" Jul 13 22:00:19.769040 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.768943 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/555d9e2d-52a7-4024-aa67-991a477e5ab7-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-c6cfc\" (UID: \"555d9e2d-52a7-4024-aa67-991a477e5ab7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c6cfc" Jul 13 22:00:19.769077 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:19.769053 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 13 22:00:19.769135 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:19.769120 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/555d9e2d-52a7-4024-aa67-991a477e5ab7-samples-operator-tls podName:555d9e2d-52a7-4024-aa67-991a477e5ab7 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:20.269100153 +0000 UTC m=+41.719655689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/555d9e2d-52a7-4024-aa67-991a477e5ab7-samples-operator-tls") pod "cluster-samples-operator-65db858894-c6cfc" (UID: "555d9e2d-52a7-4024-aa67-991a477e5ab7") : secret "samples-operator-tls" not found Jul 13 22:00:19.777467 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.777407 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-x5qhq" Jul 13 22:00:19.778741 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.778673 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh6dl\" (UniqueName: \"kubernetes.io/projected/555d9e2d-52a7-4024-aa67-991a477e5ab7-kube-api-access-hh6dl\") pod \"cluster-samples-operator-65db858894-c6cfc\" (UID: \"555d9e2d-52a7-4024-aa67-991a477e5ab7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c6cfc" Jul 13 22:00:19.869474 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.869443 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/db233478-500b-472c-919f-510266682def-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-rcbqx\" (UID: \"db233478-500b-472c-919f-510266682def\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rcbqx" Jul 13 22:00:19.869634 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.869484 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn5ds\" (UniqueName: \"kubernetes.io/projected/db233478-500b-472c-919f-510266682def-kube-api-access-tn5ds\") pod \"kube-storage-version-migrator-operator-799cb7f958-rcbqx\" (UID: \"db233478-500b-472c-919f-510266682def\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rcbqx" Jul 13 22:00:19.869634 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.869513 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/db233478-500b-472c-919f-510266682def-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-rcbqx\" (UID: \"db233478-500b-472c-919f-510266682def\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rcbqx" Jul 13 22:00:19.870284 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.870242 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/db233478-500b-472c-919f-510266682def-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-rcbqx\" (UID: \"db233478-500b-472c-919f-510266682def\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rcbqx" Jul 13 22:00:19.871998 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.871949 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/db233478-500b-472c-919f-510266682def-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-rcbqx\" (UID: \"db233478-500b-472c-919f-510266682def\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rcbqx" Jul 13 22:00:19.880466 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.880348 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn5ds\" (UniqueName: \"kubernetes.io/projected/db233478-500b-472c-919f-510266682def-kube-api-access-tn5ds\") pod \"kube-storage-version-migrator-operator-799cb7f958-rcbqx\" (UID: \"db233478-500b-472c-919f-510266682def\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rcbqx" Jul 13 22:00:19.886848 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.886569 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-gwtpb"] Jul 13 22:00:19.890243 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:00:19.890193 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca2745d_9fd1_4064_9448_abcdb92715af.slice/crio-9fd22848f53c5ef90424ab0a020d9f1a5478a93a79b8178f1df190becb03c632 WatchSource:0}: Error finding container 9fd22848f53c5ef90424ab0a020d9f1a5478a93a79b8178f1df190becb03c632: Status 404 returned error can't find the container with id 9fd22848f53c5ef90424ab0a020d9f1a5478a93a79b8178f1df190becb03c632 Jul 13 22:00:19.897608 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.897409 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rcbqx" Jul 13 22:00:19.920279 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.920252 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qhgw4"] Jul 13 22:00:19.927116 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:00:19.927089 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90dbf0b_63cf_41a1_971c_76d47f6d859b.slice/crio-7dabefeb6cb6448c3c1908908e038c3f703e7d65af64cae2b125b111d54326f8 WatchSource:0}: Error finding container 7dabefeb6cb6448c3c1908908e038c3f703e7d65af64cae2b125b111d54326f8: Status 404 returned error can't find the container with id 7dabefeb6cb6448c3c1908908e038c3f703e7d65af64cae2b125b111d54326f8 Jul 13 22:00:19.965337 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.965304 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-5vww5"] Jul 13 22:00:19.968616 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:00:19.968587 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3815ceb2_a3b8_43bc_bf04_b1dd4adc4412.slice/crio-850ba52acd82bf082917d93a4f62d3c8a9ce51ee8f7bbe9d6c438598d9c85ef4 WatchSource:0}: Error finding container 850ba52acd82bf082917d93a4f62d3c8a9ce51ee8f7bbe9d6c438598d9c85ef4: Status 404 returned error can't find the container with id 850ba52acd82bf082917d93a4f62d3c8a9ce51ee8f7bbe9d6c438598d9c85ef4 Jul 13 22:00:19.982692 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:19.982611 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-x5qhq"] Jul 13 22:00:19.986358 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:00:19.986333 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1fa75b_f29d_45a1_af98_fa947c2058b2.slice/crio-7a92eac83862161c813ab499a5a3675fb3f613509445be6f41b765b3832b0f26 WatchSource:0}: Error finding container 7a92eac83862161c813ab499a5a3675fb3f613509445be6f41b765b3832b0f26: Status 404 returned error can't find the container with id 7a92eac83862161c813ab499a5a3675fb3f613509445be6f41b765b3832b0f26 Jul 13 22:00:20.052952 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:20.052919 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rcbqx"] Jul 13 22:00:20.055697 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:00:20.055674 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb233478_500b_472c_919f_510266682def.slice/crio-b7ecf90074e74a194e24cf224c879ce97af2e0e3ad41b3c15e072c4374e67b35 WatchSource:0}: Error finding container b7ecf90074e74a194e24cf224c879ce97af2e0e3ad41b3c15e072c4374e67b35: Status 404 returned error can't find the container with id b7ecf90074e74a194e24cf224c879ce97af2e0e3ad41b3c15e072c4374e67b35 Jul 13 22:00:20.070624 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:20.070606 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e9588e99-3cae-44f7-82b6-101ae1fbd604-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-dlj2d\" (UID: \"e9588e99-3cae-44f7-82b6-101ae1fbd604\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-dlj2d" Jul 13 22:00:20.070766 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:20.070748 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 22:00:20.070818 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:20.070809 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9588e99-3cae-44f7-82b6-101ae1fbd604-cluster-monitoring-operator-tls podName:e9588e99-3cae-44f7-82b6-101ae1fbd604 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:21.070794463 +0000 UTC m=+42.521349978 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e9588e99-3cae-44f7-82b6-101ae1fbd604-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-dlj2d" (UID: "e9588e99-3cae-44f7-82b6-101ae1fbd604") : secret "cluster-monitoring-operator-tls" not found Jul 13 22:00:20.171423 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:20.171394 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbedfa07-690d-4c79-92d3-dc85b88124d7-metrics-certs\") pod \"router-default-5dbcf998b4-7qqv9\" (UID: \"dbedfa07-690d-4c79-92d3-dc85b88124d7\") " pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:20.171594 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:20.171434 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbedfa07-690d-4c79-92d3-dc85b88124d7-service-ca-bundle\") pod \"router-default-5dbcf998b4-7qqv9\" (UID: \"dbedfa07-690d-4c79-92d3-dc85b88124d7\") " pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:20.171594 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:20.171505 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 22:00:20.171594 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:20.171571 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbedfa07-690d-4c79-92d3-dc85b88124d7-metrics-certs podName:dbedfa07-690d-4c79-92d3-dc85b88124d7 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:21.171551385 +0000 UTC m=+42.622106901 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbedfa07-690d-4c79-92d3-dc85b88124d7-metrics-certs") pod "router-default-5dbcf998b4-7qqv9" (UID: "dbedfa07-690d-4c79-92d3-dc85b88124d7") : secret "router-metrics-certs-default" not found Jul 13 22:00:20.171594 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:20.171591 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dbedfa07-690d-4c79-92d3-dc85b88124d7-service-ca-bundle podName:dbedfa07-690d-4c79-92d3-dc85b88124d7 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:21.171582841 +0000 UTC m=+42.622138357 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dbedfa07-690d-4c79-92d3-dc85b88124d7-service-ca-bundle") pod "router-default-5dbcf998b4-7qqv9" (UID: "dbedfa07-690d-4c79-92d3-dc85b88124d7") : configmap references non-existent config key: service-ca.crt Jul 13 22:00:20.174497 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:20.174466 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rcbqx" event={"ID":"db233478-500b-472c-919f-510266682def","Type":"ContainerStarted","Data":"b7ecf90074e74a194e24cf224c879ce97af2e0e3ad41b3c15e072c4374e67b35"} Jul 13 22:00:20.175652 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:20.175628 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-x5qhq" event={"ID":"7b1fa75b-f29d-45a1-af98-fa947c2058b2","Type":"ContainerStarted","Data":"7a92eac83862161c813ab499a5a3675fb3f613509445be6f41b765b3832b0f26"} Jul 13 22:00:20.176647 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:20.176619 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-5vww5" event={"ID":"3815ceb2-a3b8-43bc-bf04-b1dd4adc4412","Type":"ContainerStarted","Data":"850ba52acd82bf082917d93a4f62d3c8a9ce51ee8f7bbe9d6c438598d9c85ef4"} Jul 13 22:00:20.177811 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:20.177790 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-g497d" event={"ID":"b9117cd6-d8c5-43a8-9a5d-8b5a349a5943","Type":"ContainerStarted","Data":"944df706f19c8ad01c20c05b15da9cde4963b2e131e861dfd7c075ccc2df81ab"} Jul 13 22:00:20.178835 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:20.178783 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-gwtpb" event={"ID":"aca2745d-9fd1-4064-9448-abcdb92715af","Type":"ContainerStarted","Data":"9fd22848f53c5ef90424ab0a020d9f1a5478a93a79b8178f1df190becb03c632"} Jul 13 22:00:20.179871 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:20.179851 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qhgw4" event={"ID":"f90dbf0b-63cf-41a1-971c-76d47f6d859b","Type":"ContainerStarted","Data":"7dabefeb6cb6448c3c1908908e038c3f703e7d65af64cae2b125b111d54326f8"} Jul 13 22:00:20.272670 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:20.272642 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/555d9e2d-52a7-4024-aa67-991a477e5ab7-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-c6cfc\" (UID: \"555d9e2d-52a7-4024-aa67-991a477e5ab7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c6cfc" Jul 13 22:00:20.272786 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:20.272750 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 13 22:00:20.272822 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:20.272799 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/555d9e2d-52a7-4024-aa67-991a477e5ab7-samples-operator-tls podName:555d9e2d-52a7-4024-aa67-991a477e5ab7 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:21.272787613 +0000 UTC m=+42.723343128 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/555d9e2d-52a7-4024-aa67-991a477e5ab7-samples-operator-tls") pod "cluster-samples-operator-65db858894-c6cfc" (UID: "555d9e2d-52a7-4024-aa67-991a477e5ab7") : secret "samples-operator-tls" not found Jul 13 22:00:21.080288 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:21.079521 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e9588e99-3cae-44f7-82b6-101ae1fbd604-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-dlj2d\" (UID: \"e9588e99-3cae-44f7-82b6-101ae1fbd604\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-dlj2d" Jul 13 22:00:21.080288 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:21.079791 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 22:00:21.080288 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:21.079858 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9588e99-3cae-44f7-82b6-101ae1fbd604-cluster-monitoring-operator-tls podName:e9588e99-3cae-44f7-82b6-101ae1fbd604 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:23.079836977 +0000 UTC m=+44.530392506 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e9588e99-3cae-44f7-82b6-101ae1fbd604-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-dlj2d" (UID: "e9588e99-3cae-44f7-82b6-101ae1fbd604") : secret "cluster-monitoring-operator-tls" not found Jul 13 22:00:21.181691 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:21.181170 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbedfa07-690d-4c79-92d3-dc85b88124d7-metrics-certs\") pod \"router-default-5dbcf998b4-7qqv9\" (UID: \"dbedfa07-690d-4c79-92d3-dc85b88124d7\") " pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:21.181691 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:21.181228 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbedfa07-690d-4c79-92d3-dc85b88124d7-service-ca-bundle\") pod \"router-default-5dbcf998b4-7qqv9\" (UID: \"dbedfa07-690d-4c79-92d3-dc85b88124d7\") " pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:21.181691 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:21.181290 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 22:00:21.181691 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:21.181360 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbedfa07-690d-4c79-92d3-dc85b88124d7-metrics-certs podName:dbedfa07-690d-4c79-92d3-dc85b88124d7 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:23.181339782 +0000 UTC m=+44.631895300 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbedfa07-690d-4c79-92d3-dc85b88124d7-metrics-certs") pod "router-default-5dbcf998b4-7qqv9" (UID: "dbedfa07-690d-4c79-92d3-dc85b88124d7") : secret "router-metrics-certs-default" not found Jul 13 22:00:21.181691 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:21.181379 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dbedfa07-690d-4c79-92d3-dc85b88124d7-service-ca-bundle podName:dbedfa07-690d-4c79-92d3-dc85b88124d7 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:23.181369025 +0000 UTC m=+44.631924547 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dbedfa07-690d-4c79-92d3-dc85b88124d7-service-ca-bundle") pod "router-default-5dbcf998b4-7qqv9" (UID: "dbedfa07-690d-4c79-92d3-dc85b88124d7") : configmap references non-existent config key: service-ca.crt Jul 13 22:00:21.282178 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:21.282071 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/555d9e2d-52a7-4024-aa67-991a477e5ab7-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-c6cfc\" (UID: \"555d9e2d-52a7-4024-aa67-991a477e5ab7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c6cfc" Jul 13 22:00:21.282368 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:21.282268 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 13 22:00:21.282368 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:21.282332 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/555d9e2d-52a7-4024-aa67-991a477e5ab7-samples-operator-tls podName:555d9e2d-52a7-4024-aa67-991a477e5ab7 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:23.28231429 +0000 UTC m=+44.732869807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/555d9e2d-52a7-4024-aa67-991a477e5ab7-samples-operator-tls") pod "cluster-samples-operator-65db858894-c6cfc" (UID: "555d9e2d-52a7-4024-aa67-991a477e5ab7") : secret "samples-operator-tls" not found Jul 13 22:00:22.294561 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:22.294398 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-tls\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:22.294561 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:22.294556 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 22:00:22.295301 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:22.294581 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f7bbf7c5-9h5nl: secret "image-registry-tls" not found Jul 13 22:00:22.295301 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:22.294655 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-tls podName:505f1712-2250-43ff-a8ab-270f7cc2b859 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:30.294633814 +0000 UTC m=+51.745189341 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-tls") pod "image-registry-86f7bbf7c5-9h5nl" (UID: "505f1712-2250-43ff-a8ab-270f7cc2b859") : secret "image-registry-tls" not found Jul 13 22:00:22.394927 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:22.394890 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06450a5e-03fe-4a9e-a6e8-722e0fc36664-metrics-tls\") pod \"dns-default-qzmt4\" (UID: \"06450a5e-03fe-4a9e-a6e8-722e0fc36664\") " pod="openshift-dns/dns-default-qzmt4" Jul 13 22:00:22.395130 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:22.395114 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 22:00:22.395235 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:22.395192 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06450a5e-03fe-4a9e-a6e8-722e0fc36664-metrics-tls podName:06450a5e-03fe-4a9e-a6e8-722e0fc36664 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:30.395167974 +0000 UTC m=+51.845723489 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06450a5e-03fe-4a9e-a6e8-722e0fc36664-metrics-tls") pod "dns-default-qzmt4" (UID: "06450a5e-03fe-4a9e-a6e8-722e0fc36664") : secret "dns-default-metrics-tls" not found Jul 13 22:00:23.100356 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:23.100328 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e9588e99-3cae-44f7-82b6-101ae1fbd604-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-dlj2d\" (UID: \"e9588e99-3cae-44f7-82b6-101ae1fbd604\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-dlj2d" Jul 13 22:00:23.100554 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:23.100531 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 22:00:23.100671 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:23.100617 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9588e99-3cae-44f7-82b6-101ae1fbd604-cluster-monitoring-operator-tls podName:e9588e99-3cae-44f7-82b6-101ae1fbd604 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:27.100596016 +0000 UTC m=+48.551151544 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e9588e99-3cae-44f7-82b6-101ae1fbd604-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-dlj2d" (UID: "e9588e99-3cae-44f7-82b6-101ae1fbd604") : secret "cluster-monitoring-operator-tls" not found Jul 13 22:00:23.201044 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:23.201011 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbedfa07-690d-4c79-92d3-dc85b88124d7-metrics-certs\") pod \"router-default-5dbcf998b4-7qqv9\" (UID: \"dbedfa07-690d-4c79-92d3-dc85b88124d7\") " pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:23.201044 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:23.201054 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbedfa07-690d-4c79-92d3-dc85b88124d7-service-ca-bundle\") pod \"router-default-5dbcf998b4-7qqv9\" (UID: \"dbedfa07-690d-4c79-92d3-dc85b88124d7\") " pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:23.201235 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:23.201170 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 22:00:23.201276 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:23.201236 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbedfa07-690d-4c79-92d3-dc85b88124d7-metrics-certs podName:dbedfa07-690d-4c79-92d3-dc85b88124d7 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:27.201217827 +0000 UTC m=+48.651773352 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbedfa07-690d-4c79-92d3-dc85b88124d7-metrics-certs") pod "router-default-5dbcf998b4-7qqv9" (UID: "dbedfa07-690d-4c79-92d3-dc85b88124d7") : secret "router-metrics-certs-default" not found Jul 13 22:00:23.201621 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:23.201595 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dbedfa07-690d-4c79-92d3-dc85b88124d7-service-ca-bundle podName:dbedfa07-690d-4c79-92d3-dc85b88124d7 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:27.201246201 +0000 UTC m=+48.651801720 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dbedfa07-690d-4c79-92d3-dc85b88124d7-service-ca-bundle") pod "router-default-5dbcf998b4-7qqv9" (UID: "dbedfa07-690d-4c79-92d3-dc85b88124d7") : configmap references non-existent config key: service-ca.crt Jul 13 22:00:23.302449 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:23.302414 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/555d9e2d-52a7-4024-aa67-991a477e5ab7-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-c6cfc\" (UID: \"555d9e2d-52a7-4024-aa67-991a477e5ab7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c6cfc" Jul 13 22:00:23.302819 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:23.302549 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 13 22:00:23.302819 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:23.302632 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/555d9e2d-52a7-4024-aa67-991a477e5ab7-samples-operator-tls podName:555d9e2d-52a7-4024-aa67-991a477e5ab7 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:27.302615673 +0000 UTC m=+48.753171188 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/555d9e2d-52a7-4024-aa67-991a477e5ab7-samples-operator-tls") pod "cluster-samples-operator-65db858894-c6cfc" (UID: "555d9e2d-52a7-4024-aa67-991a477e5ab7") : secret "samples-operator-tls" not found Jul 13 22:00:27.137307 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:27.137269 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e9588e99-3cae-44f7-82b6-101ae1fbd604-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-dlj2d\" (UID: \"e9588e99-3cae-44f7-82b6-101ae1fbd604\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-dlj2d" Jul 13 22:00:27.137772 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:27.137434 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 22:00:27.137772 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:27.137510 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9588e99-3cae-44f7-82b6-101ae1fbd604-cluster-monitoring-operator-tls podName:e9588e99-3cae-44f7-82b6-101ae1fbd604 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:35.137489172 +0000 UTC m=+56.588044691 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e9588e99-3cae-44f7-82b6-101ae1fbd604-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-dlj2d" (UID: "e9588e99-3cae-44f7-82b6-101ae1fbd604") : secret "cluster-monitoring-operator-tls" not found Jul 13 22:00:27.205550 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:27.205506 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rcbqx" event={"ID":"db233478-500b-472c-919f-510266682def","Type":"ContainerStarted","Data":"d23c07c56436a3b3118ebfe1d37c5d5e4b3ccd09b8a5a36e4925139d0d4fbc65"} Jul 13 22:00:27.207477 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:27.207453 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x5qhq_7b1fa75b-f29d-45a1-af98-fa947c2058b2/console-operator/0.log" Jul 13 22:00:27.207618 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:27.207496 2562 generic.go:356] "Generic (PLEG): container finished" podID="7b1fa75b-f29d-45a1-af98-fa947c2058b2" containerID="6e76cb20974c9d2650e95934e4fba7e8e4a654b8ae5df90e012e959752ebc6a0" exitCode=255 Jul 13 22:00:27.207618 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:27.207574 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-x5qhq" event={"ID":"7b1fa75b-f29d-45a1-af98-fa947c2058b2","Type":"ContainerDied","Data":"6e76cb20974c9d2650e95934e4fba7e8e4a654b8ae5df90e012e959752ebc6a0"} Jul 13 22:00:27.207811 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:27.207783 2562 scope.go:117] "RemoveContainer" containerID="6e76cb20974c9d2650e95934e4fba7e8e4a654b8ae5df90e012e959752ebc6a0" Jul 13 22:00:27.209453 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:27.209406 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-5vww5" event={"ID":"3815ceb2-a3b8-43bc-bf04-b1dd4adc4412","Type":"ContainerStarted","Data":"2ec787aa4a14bc7ce4544a7c29cd0481a547438d3c0e347becc165285ebef020"} Jul 13 22:00:27.211396 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:27.211368 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-g497d" event={"ID":"b9117cd6-d8c5-43a8-9a5d-8b5a349a5943","Type":"ContainerStarted","Data":"dd9235423909448996a2787534c0105e5a2cd6d15d3d0a069df257ae33b25484"} Jul 13 22:00:27.212947 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:27.212922 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-gwtpb" event={"ID":"aca2745d-9fd1-4064-9448-abcdb92715af","Type":"ContainerStarted","Data":"cd0ba5dd753cc05f63fcd731f0e9e50afafd80497e2d5f3bec65781b8057a3de"} Jul 13 22:00:27.214526 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:27.214501 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qhgw4" event={"ID":"f90dbf0b-63cf-41a1-971c-76d47f6d859b","Type":"ContainerStarted","Data":"709669d00b0b5767bcd82ef5b279ccecc0a078d7dfc7d9ba67a72055c310f390"} Jul 13 22:00:27.220651 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:27.220608 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rcbqx" podStartSLOduration=1.584831535 podStartE2EDuration="8.220592595s" totalImagesPullingTime="6.63576106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:27.219930445 +0000 UTC m=+48.670485984" watchObservedRunningTime="2026-07-13 22:00:27.220592595 +0000 UTC m=+48.671148134" Jul 13 22:00:27.235118 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:27.235070 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qhgw4" podStartSLOduration=1.471737481 podStartE2EDuration="8.235052299s" totalImagesPullingTime="6.763314818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:27.234311608 +0000 UTC m=+48.684867145" watchObservedRunningTime="2026-07-13 22:00:27.235052299 +0000 UTC m=+48.685607840" Jul 13 22:00:27.237834 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:27.237810 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbedfa07-690d-4c79-92d3-dc85b88124d7-service-ca-bundle\") pod \"router-default-5dbcf998b4-7qqv9\" (UID: \"dbedfa07-690d-4c79-92d3-dc85b88124d7\") " pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:27.238342 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:27.238322 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbedfa07-690d-4c79-92d3-dc85b88124d7-metrics-certs\") pod \"router-default-5dbcf998b4-7qqv9\" (UID: \"dbedfa07-690d-4c79-92d3-dc85b88124d7\") " pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:27.239505 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:27.239479 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dbedfa07-690d-4c79-92d3-dc85b88124d7-service-ca-bundle podName:dbedfa07-690d-4c79-92d3-dc85b88124d7 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:35.239460275 +0000 UTC m=+56.690015791 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dbedfa07-690d-4c79-92d3-dc85b88124d7-service-ca-bundle") pod "router-default-5dbcf998b4-7qqv9" (UID: "dbedfa07-690d-4c79-92d3-dc85b88124d7") : configmap references non-existent config key: service-ca.crt Jul 13 22:00:27.239833 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:27.239816 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 22:00:27.239959 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:27.239946 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbedfa07-690d-4c79-92d3-dc85b88124d7-metrics-certs podName:dbedfa07-690d-4c79-92d3-dc85b88124d7 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:35.239929236 +0000 UTC m=+56.690484763 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbedfa07-690d-4c79-92d3-dc85b88124d7-metrics-certs") pod "router-default-5dbcf998b4-7qqv9" (UID: "dbedfa07-690d-4c79-92d3-dc85b88124d7") : secret "router-metrics-certs-default" not found Jul 13 22:00:27.271489 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:27.271429 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-g497d" podStartSLOduration=1.384930956 podStartE2EDuration="8.271408896s" totalImagesPullingTime="6.88647794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:27.271327024 +0000 UTC m=+48.721882562" watchObservedRunningTime="2026-07-13 22:00:27.271408896 +0000 UTC m=+48.721964435" Jul 13 22:00:27.289079 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:27.289001 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-5vww5" podStartSLOduration=1.566664246 podStartE2EDuration="8.288964979s" totalImagesPullingTime="6.722300733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:27.288564283 +0000 UTC m=+48.739119821" watchObservedRunningTime="2026-07-13 22:00:27.288964979 +0000 UTC m=+48.739520516" Jul 13 22:00:27.304928 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:27.304882 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-gwtpb" podStartSLOduration=1.50359218 podStartE2EDuration="8.304863542s" totalImagesPullingTime="6.801271362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:27.303748648 +0000 UTC m=+48.754304188" watchObservedRunningTime="2026-07-13 22:00:27.304863542 +0000 UTC m=+48.755419077" Jul 13 22:00:27.340013 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:27.339926 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/555d9e2d-52a7-4024-aa67-991a477e5ab7-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-c6cfc\" (UID: \"555d9e2d-52a7-4024-aa67-991a477e5ab7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c6cfc" Jul 13 22:00:27.340182 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:27.340163 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 13 22:00:27.340402 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:27.340386 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/555d9e2d-52a7-4024-aa67-991a477e5ab7-samples-operator-tls podName:555d9e2d-52a7-4024-aa67-991a477e5ab7 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:35.340365288 +0000 UTC m=+56.790920807 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/555d9e2d-52a7-4024-aa67-991a477e5ab7-samples-operator-tls") pod "cluster-samples-operator-65db858894-c6cfc" (UID: "555d9e2d-52a7-4024-aa67-991a477e5ab7") : secret "samples-operator-tls" not found Jul 13 22:00:28.220201 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:28.220172 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x5qhq_7b1fa75b-f29d-45a1-af98-fa947c2058b2/console-operator/1.log" Jul 13 22:00:28.220661 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:28.220644 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x5qhq_7b1fa75b-f29d-45a1-af98-fa947c2058b2/console-operator/0.log" Jul 13 22:00:28.220740 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:28.220686 2562 generic.go:356] "Generic (PLEG): container finished" podID="7b1fa75b-f29d-45a1-af98-fa947c2058b2" containerID="be02e46854fa77ac3ba5d61a7a8797219f8eca65bd0be6011443e18d5453997e" exitCode=255 Jul 13 22:00:28.220810 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:28.220787 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-x5qhq" event={"ID":"7b1fa75b-f29d-45a1-af98-fa947c2058b2","Type":"ContainerDied","Data":"be02e46854fa77ac3ba5d61a7a8797219f8eca65bd0be6011443e18d5453997e"} Jul 13 22:00:28.220907 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:28.220893 2562 scope.go:117] "RemoveContainer" containerID="6e76cb20974c9d2650e95934e4fba7e8e4a654b8ae5df90e012e959752ebc6a0" Jul 13 22:00:28.221208 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:28.221177 2562 scope.go:117] "RemoveContainer" containerID="be02e46854fa77ac3ba5d61a7a8797219f8eca65bd0be6011443e18d5453997e" Jul 13 22:00:28.221358 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:28.221333 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-x5qhq_openshift-console-operator(7b1fa75b-f29d-45a1-af98-fa947c2058b2)\"" pod="openshift-console-operator/console-operator-7866497494-x5qhq" podUID="7b1fa75b-f29d-45a1-af98-fa947c2058b2" Jul 13 22:00:28.733482 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:28.733443 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vhlb7"] Jul 13 22:00:28.762120 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:28.762078 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vhlb7"] Jul 13 22:00:28.762279 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:28.762225 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vhlb7" Jul 13 22:00:28.764600 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:28.764578 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 13 22:00:28.764743 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:28.764607 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xvmjd\"" Jul 13 22:00:28.764743 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:28.764609 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 13 22:00:28.853174 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:28.853139 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwmhp\" (UniqueName: \"kubernetes.io/projected/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-kube-api-access-zwmhp\") pod \"insights-runtime-extractor-vhlb7\" (UID: \"9d1a0995-48f0-46b7-a75d-6c10a0860e7b\") " pod="openshift-insights/insights-runtime-extractor-vhlb7" Jul 13 22:00:28.853318 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:28.853211 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhlb7\" (UID: \"9d1a0995-48f0-46b7-a75d-6c10a0860e7b\") " pod="openshift-insights/insights-runtime-extractor-vhlb7" Jul 13 22:00:28.853397 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:28.853337 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-crio-socket\") pod \"insights-runtime-extractor-vhlb7\" (UID: \"9d1a0995-48f0-46b7-a75d-6c10a0860e7b\") " pod="openshift-insights/insights-runtime-extractor-vhlb7" Jul 13 22:00:28.853397 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:28.853369 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhlb7\" (UID: \"9d1a0995-48f0-46b7-a75d-6c10a0860e7b\") " pod="openshift-insights/insights-runtime-extractor-vhlb7" Jul 13 22:00:28.853473 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:28.853415 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-data-volume\") pod \"insights-runtime-extractor-vhlb7\" (UID: \"9d1a0995-48f0-46b7-a75d-6c10a0860e7b\") " pod="openshift-insights/insights-runtime-extractor-vhlb7" Jul 13 22:00:28.954654 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:28.954620 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-crio-socket\") pod \"insights-runtime-extractor-vhlb7\" (UID: \"9d1a0995-48f0-46b7-a75d-6c10a0860e7b\") " pod="openshift-insights/insights-runtime-extractor-vhlb7" Jul 13 22:00:28.954654 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:28.954656 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhlb7\" (UID: \"9d1a0995-48f0-46b7-a75d-6c10a0860e7b\") " pod="openshift-insights/insights-runtime-extractor-vhlb7" Jul 13 22:00:28.954861 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:28.954759 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 22:00:28.954861 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:28.954787 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-data-volume\") pod \"insights-runtime-extractor-vhlb7\" (UID: \"9d1a0995-48f0-46b7-a75d-6c10a0860e7b\") " pod="openshift-insights/insights-runtime-extractor-vhlb7" Jul 13 22:00:28.954861 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:28.954804 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-insights-runtime-extractor-tls podName:9d1a0995-48f0-46b7-a75d-6c10a0860e7b nodeName:}" failed. No retries permitted until 2026-07-13 22:00:29.454791011 +0000 UTC m=+50.905346526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhlb7" (UID: "9d1a0995-48f0-46b7-a75d-6c10a0860e7b") : secret "insights-runtime-extractor-tls" not found Jul 13 22:00:28.954861 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:28.954827 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-crio-socket\") pod \"insights-runtime-extractor-vhlb7\" (UID: \"9d1a0995-48f0-46b7-a75d-6c10a0860e7b\") " pod="openshift-insights/insights-runtime-extractor-vhlb7" Jul 13 22:00:28.954861 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:28.954847 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwmhp\" (UniqueName: \"kubernetes.io/projected/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-kube-api-access-zwmhp\") pod \"insights-runtime-extractor-vhlb7\" (UID: \"9d1a0995-48f0-46b7-a75d-6c10a0860e7b\") " pod="openshift-insights/insights-runtime-extractor-vhlb7" Jul 13 22:00:28.955198 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:28.954871 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhlb7\" (UID: \"9d1a0995-48f0-46b7-a75d-6c10a0860e7b\") " pod="openshift-insights/insights-runtime-extractor-vhlb7" Jul 13 22:00:28.955198 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:28.954958 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 22:00:28.955198 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:28.955016 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-kube-rbac-proxy-cm podName:9d1a0995-48f0-46b7-a75d-6c10a0860e7b nodeName:}" failed. No retries permitted until 2026-07-13 22:00:29.455006667 +0000 UTC m=+50.905562182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vhlb7" (UID: "9d1a0995-48f0-46b7-a75d-6c10a0860e7b") : configmap "kube-rbac-proxy" not found Jul 13 22:00:28.955198 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:28.955068 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-data-volume\") pod \"insights-runtime-extractor-vhlb7\" (UID: \"9d1a0995-48f0-46b7-a75d-6c10a0860e7b\") " pod="openshift-insights/insights-runtime-extractor-vhlb7" Jul 13 22:00:28.966888 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:28.966856 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwmhp\" (UniqueName: \"kubernetes.io/projected/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-kube-api-access-zwmhp\") pod \"insights-runtime-extractor-vhlb7\" (UID: \"9d1a0995-48f0-46b7-a75d-6c10a0860e7b\") " pod="openshift-insights/insights-runtime-extractor-vhlb7" Jul 13 22:00:29.225745 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:29.225720 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x5qhq_7b1fa75b-f29d-45a1-af98-fa947c2058b2/console-operator/1.log" Jul 13 22:00:29.226117 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:29.226035 2562 scope.go:117] "RemoveContainer" containerID="be02e46854fa77ac3ba5d61a7a8797219f8eca65bd0be6011443e18d5453997e" Jul 13 22:00:29.226216 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:29.226188 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-x5qhq_openshift-console-operator(7b1fa75b-f29d-45a1-af98-fa947c2058b2)\"" pod="openshift-console-operator/console-operator-7866497494-x5qhq" podUID="7b1fa75b-f29d-45a1-af98-fa947c2058b2" Jul 13 22:00:29.459641 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:29.459604 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhlb7\" (UID: \"9d1a0995-48f0-46b7-a75d-6c10a0860e7b\") " pod="openshift-insights/insights-runtime-extractor-vhlb7" Jul 13 22:00:29.459795 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:29.459689 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhlb7\" (UID: \"9d1a0995-48f0-46b7-a75d-6c10a0860e7b\") " pod="openshift-insights/insights-runtime-extractor-vhlb7" Jul 13 22:00:29.459795 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:29.459739 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 13 22:00:29.459870 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:29.459818 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-kube-rbac-proxy-cm podName:9d1a0995-48f0-46b7-a75d-6c10a0860e7b nodeName:}" failed. No retries permitted until 2026-07-13 22:00:30.4598015 +0000 UTC m=+51.910357015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vhlb7" (UID: "9d1a0995-48f0-46b7-a75d-6c10a0860e7b") : configmap "kube-rbac-proxy" not found Jul 13 22:00:29.459870 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:29.459842 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 22:00:29.459938 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:29.459895 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-insights-runtime-extractor-tls podName:9d1a0995-48f0-46b7-a75d-6c10a0860e7b nodeName:}" failed. No retries permitted until 2026-07-13 22:00:30.459881642 +0000 UTC m=+51.910437162 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhlb7" (UID: "9d1a0995-48f0-46b7-a75d-6c10a0860e7b") : secret "insights-runtime-extractor-tls" not found Jul 13 22:00:29.777965 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:29.777932 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-x5qhq" Jul 13 22:00:29.778138 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:29.778001 2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-x5qhq" Jul 13 22:00:30.229516 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:30.229493 2562 scope.go:117] "RemoveContainer" containerID="be02e46854fa77ac3ba5d61a7a8797219f8eca65bd0be6011443e18d5453997e" Jul 13 22:00:30.229858 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:30.229673 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-x5qhq_openshift-console-operator(7b1fa75b-f29d-45a1-af98-fa947c2058b2)\"" pod="openshift-console-operator/console-operator-7866497494-x5qhq" podUID="7b1fa75b-f29d-45a1-af98-fa947c2058b2" Jul 13 22:00:30.368546 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:30.368514 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-tls\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:30.368687 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:30.368667 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 13 22:00:30.368738 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:30.368689 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-86f7bbf7c5-9h5nl: secret "image-registry-tls" not found Jul 13 22:00:30.368770 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:30.368753 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-tls podName:505f1712-2250-43ff-a8ab-270f7cc2b859 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:46.368737692 +0000 UTC m=+67.819293206 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-tls") pod "image-registry-86f7bbf7c5-9h5nl" (UID: "505f1712-2250-43ff-a8ab-270f7cc2b859") : secret "image-registry-tls" not found Jul 13 22:00:30.469568 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:30.469530 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhlb7\" (UID: \"9d1a0995-48f0-46b7-a75d-6c10a0860e7b\") " pod="openshift-insights/insights-runtime-extractor-vhlb7" Jul 13 22:00:30.469568 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:30.469573 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06450a5e-03fe-4a9e-a6e8-722e0fc36664-metrics-tls\") pod \"dns-default-qzmt4\" (UID: \"06450a5e-03fe-4a9e-a6e8-722e0fc36664\") " pod="openshift-dns/dns-default-qzmt4" Jul 13 22:00:30.469747 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:30.469628 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhlb7\" (UID: \"9d1a0995-48f0-46b7-a75d-6c10a0860e7b\") " pod="openshift-insights/insights-runtime-extractor-vhlb7" Jul 13 22:00:30.469747 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:30.469678 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 22:00:30.469747 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:30.469739 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-insights-runtime-extractor-tls podName:9d1a0995-48f0-46b7-a75d-6c10a0860e7b nodeName:}" failed. No retries permitted until 2026-07-13 22:00:32.469723414 +0000 UTC m=+53.920278928 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhlb7" (UID: "9d1a0995-48f0-46b7-a75d-6c10a0860e7b") : secret "insights-runtime-extractor-tls" not found Jul 13 22:00:30.469851 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:30.469748 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 13 22:00:30.469851 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:30.469795 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06450a5e-03fe-4a9e-a6e8-722e0fc36664-metrics-tls podName:06450a5e-03fe-4a9e-a6e8-722e0fc36664 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:46.469779416 +0000 UTC m=+67.920334944 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06450a5e-03fe-4a9e-a6e8-722e0fc36664-metrics-tls") pod "dns-default-qzmt4" (UID: "06450a5e-03fe-4a9e-a6e8-722e0fc36664") : secret "dns-default-metrics-tls" not found Jul 13 22:00:30.470104 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:30.470090 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vhlb7\" (UID: \"9d1a0995-48f0-46b7-a75d-6c10a0860e7b\") " pod="openshift-insights/insights-runtime-extractor-vhlb7" Jul 13 22:00:31.338696 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:31.338667 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ccs2h_3f77cde0-c7bd-4803-8c7b-b5e4071ace68/dns-node-resolver/0.log" Jul 13 22:00:32.139988 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:32.139942 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7dpl6_348d04ea-a0a4-45da-ba4b-480288fd701b/node-ca/0.log" Jul 13 22:00:32.488562 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:32.488527 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhlb7\" (UID: \"9d1a0995-48f0-46b7-a75d-6c10a0860e7b\") " pod="openshift-insights/insights-runtime-extractor-vhlb7" Jul 13 22:00:32.488898 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:32.488685 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 22:00:32.488898 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:32.488755 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-insights-runtime-extractor-tls podName:9d1a0995-48f0-46b7-a75d-6c10a0860e7b nodeName:}" failed. No retries permitted until 2026-07-13 22:00:36.488739087 +0000 UTC m=+57.939294602 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhlb7" (UID: "9d1a0995-48f0-46b7-a75d-6c10a0860e7b") : secret "insights-runtime-extractor-tls" not found Jul 13 22:00:32.939141 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:32.939054 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hj2nl_b64f1d4a-95b8-4abb-8a05-8b246fe11842/serve-healthcheck-canary/0.log" Jul 13 22:00:33.540873 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:33.540842 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-rcbqx_db233478-500b-472c-919f-510266682def/kube-storage-version-migrator-operator/0.log" Jul 13 22:00:35.212652 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:35.212610 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e9588e99-3cae-44f7-82b6-101ae1fbd604-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-dlj2d\" (UID: \"e9588e99-3cae-44f7-82b6-101ae1fbd604\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-dlj2d" Jul 13 22:00:35.213066 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:35.212761 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 13 22:00:35.213066 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:35.212825 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e9588e99-3cae-44f7-82b6-101ae1fbd604-cluster-monitoring-operator-tls podName:e9588e99-3cae-44f7-82b6-101ae1fbd604 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:51.212810328 +0000 UTC m=+72.663365843 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e9588e99-3cae-44f7-82b6-101ae1fbd604-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-dlj2d" (UID: "e9588e99-3cae-44f7-82b6-101ae1fbd604") : secret "cluster-monitoring-operator-tls" not found Jul 13 22:00:35.314049 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:35.314020 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbedfa07-690d-4c79-92d3-dc85b88124d7-metrics-certs\") pod \"router-default-5dbcf998b4-7qqv9\" (UID: \"dbedfa07-690d-4c79-92d3-dc85b88124d7\") " pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:35.314049 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:35.314052 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbedfa07-690d-4c79-92d3-dc85b88124d7-service-ca-bundle\") pod \"router-default-5dbcf998b4-7qqv9\" (UID: \"dbedfa07-690d-4c79-92d3-dc85b88124d7\") " pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:35.314241 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:35.314167 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 13 22:00:35.314241 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:35.314229 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbedfa07-690d-4c79-92d3-dc85b88124d7-metrics-certs podName:dbedfa07-690d-4c79-92d3-dc85b88124d7 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:51.314206043 +0000 UTC m=+72.764761560 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dbedfa07-690d-4c79-92d3-dc85b88124d7-metrics-certs") pod "router-default-5dbcf998b4-7qqv9" (UID: "dbedfa07-690d-4c79-92d3-dc85b88124d7") : secret "router-metrics-certs-default" not found Jul 13 22:00:35.314312 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:35.314281 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dbedfa07-690d-4c79-92d3-dc85b88124d7-service-ca-bundle podName:dbedfa07-690d-4c79-92d3-dc85b88124d7 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:51.314270212 +0000 UTC m=+72.764825728 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dbedfa07-690d-4c79-92d3-dc85b88124d7-service-ca-bundle") pod "router-default-5dbcf998b4-7qqv9" (UID: "dbedfa07-690d-4c79-92d3-dc85b88124d7") : configmap references non-existent config key: service-ca.crt Jul 13 22:00:35.415447 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:35.415405 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/555d9e2d-52a7-4024-aa67-991a477e5ab7-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-c6cfc\" (UID: \"555d9e2d-52a7-4024-aa67-991a477e5ab7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c6cfc" Jul 13 22:00:35.415627 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:35.415554 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 13 22:00:35.415627 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:35.415616 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/555d9e2d-52a7-4024-aa67-991a477e5ab7-samples-operator-tls podName:555d9e2d-52a7-4024-aa67-991a477e5ab7 nodeName:}" failed. No retries permitted until 2026-07-13 22:00:51.4156008 +0000 UTC m=+72.866156315 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/555d9e2d-52a7-4024-aa67-991a477e5ab7-samples-operator-tls") pod "cluster-samples-operator-65db858894-c6cfc" (UID: "555d9e2d-52a7-4024-aa67-991a477e5ab7") : secret "samples-operator-tls" not found Jul 13 22:00:36.526420 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:36.526373 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhlb7\" (UID: \"9d1a0995-48f0-46b7-a75d-6c10a0860e7b\") " pod="openshift-insights/insights-runtime-extractor-vhlb7" Jul 13 22:00:36.526840 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:36.526602 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 13 22:00:36.526840 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:00:36.526671 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-insights-runtime-extractor-tls podName:9d1a0995-48f0-46b7-a75d-6c10a0860e7b nodeName:}" failed. No retries permitted until 2026-07-13 22:00:44.526656007 +0000 UTC m=+65.977211521 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vhlb7" (UID: "9d1a0995-48f0-46b7-a75d-6c10a0860e7b") : secret "insights-runtime-extractor-tls" not found Jul 13 22:00:39.139751 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:39.139724 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zw4h7" Jul 13 22:00:44.598419 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:44.598383 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhlb7\" (UID: \"9d1a0995-48f0-46b7-a75d-6c10a0860e7b\") " pod="openshift-insights/insights-runtime-extractor-vhlb7" Jul 13 22:00:44.600750 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:44.600723 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9d1a0995-48f0-46b7-a75d-6c10a0860e7b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vhlb7\" (UID: \"9d1a0995-48f0-46b7-a75d-6c10a0860e7b\") " pod="openshift-insights/insights-runtime-extractor-vhlb7" Jul 13 22:00:44.674057 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:44.674019 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xvmjd\"" Jul 13 22:00:44.682743 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:44.682720 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vhlb7" Jul 13 22:00:44.699722 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:44.699696 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg6gq\" (UniqueName: \"kubernetes.io/projected/71a3685e-3328-41cb-8b35-c8e8e1efe021-kube-api-access-pg6gq\") pod \"network-check-target-nbdz9\" (UID: \"71a3685e-3328-41cb-8b35-c8e8e1efe021\") " pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 22:00:44.702031 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:44.702005 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg6gq\" (UniqueName: \"kubernetes.io/projected/71a3685e-3328-41cb-8b35-c8e8e1efe021-kube-api-access-pg6gq\") pod \"network-check-target-nbdz9\" (UID: \"71a3685e-3328-41cb-8b35-c8e8e1efe021\") " pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 22:00:44.772651 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:44.772626 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gmvbg\"" Jul 13 22:00:44.780643 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:44.780622 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 22:00:44.801256 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:44.801229 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09fdf756-6037-48d0-a284-719f7770ade3-original-pull-secret\") pod \"global-pull-secret-syncer-zkrhd\" (UID: \"09fdf756-6037-48d0-a284-719f7770ade3\") " pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 22:00:44.801383 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:44.801309 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a48e3e9b-b015-44af-89b9-42c2977ae9a8-metrics-certs\") pod \"network-metrics-daemon-w8fht\" (UID: \"a48e3e9b-b015-44af-89b9-42c2977ae9a8\") " pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 22:00:44.803512 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:44.803493 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 13 22:00:44.803610 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:44.803495 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 13 22:00:44.808505 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:44.808485 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vhlb7"] Jul 13 22:00:44.811549 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:00:44.811525 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1a0995_48f0_46b7_a75d_6c10a0860e7b.slice/crio-ac8f067396175b6886026e9674d91e5a56e39d051e9dfe5037555507779008cf WatchSource:0}: Error finding container ac8f067396175b6886026e9674d91e5a56e39d051e9dfe5037555507779008cf: Status 404 returned error can't find the container with id ac8f067396175b6886026e9674d91e5a56e39d051e9dfe5037555507779008cf Jul 13 22:00:44.813703 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:44.813666 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/09fdf756-6037-48d0-a284-719f7770ade3-original-pull-secret\") pod \"global-pull-secret-syncer-zkrhd\" (UID: \"09fdf756-6037-48d0-a284-719f7770ade3\") " pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 22:00:44.813882 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:44.813856 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a48e3e9b-b015-44af-89b9-42c2977ae9a8-metrics-certs\") pod \"network-metrics-daemon-w8fht\" (UID: \"a48e3e9b-b015-44af-89b9-42c2977ae9a8\") " pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 22:00:44.900961 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:44.900930 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nbdz9"] Jul 13 22:00:44.904034 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:00:44.904009 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a3685e_3328_41cb_8b35_c8e8e1efe021.slice/crio-c31489d39453a9ecf097a451d46696ee803f99f65f7cd33c6ad540e531031198 WatchSource:0}: Error finding container c31489d39453a9ecf097a451d46696ee803f99f65f7cd33c6ad540e531031198: Status 404 returned error can't find the container with id c31489d39453a9ecf097a451d46696ee803f99f65f7cd33c6ad540e531031198 Jul 13 22:00:45.076279 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:45.076246 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zkrhd" Jul 13 22:00:45.081861 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:45.081840 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b6tsw\"" Jul 13 22:00:45.090441 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:45.090421 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-w8fht" Jul 13 22:00:45.210158 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:45.210127 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zkrhd"] Jul 13 22:00:45.212855 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:00:45.212817 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09fdf756_6037_48d0_a284_719f7770ade3.slice/crio-666b37b14707d7b11c29a57b9a852cc0d67c9d634ef1f8dd79fae10f62011039 WatchSource:0}: Error finding container 666b37b14707d7b11c29a57b9a852cc0d67c9d634ef1f8dd79fae10f62011039: Status 404 returned error can't find the container with id 666b37b14707d7b11c29a57b9a852cc0d67c9d634ef1f8dd79fae10f62011039 Jul 13 22:00:45.223942 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:45.223919 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-w8fht"] Jul 13 22:00:45.226873 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:00:45.226837 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda48e3e9b_b015_44af_89b9_42c2977ae9a8.slice/crio-b339affaaab05c1fa555d6e0e806faa071061566206a0ccc24e244299002411d WatchSource:0}: Error finding container b339affaaab05c1fa555d6e0e806faa071061566206a0ccc24e244299002411d: Status 404 returned error can't find the container with id b339affaaab05c1fa555d6e0e806faa071061566206a0ccc24e244299002411d Jul 13 22:00:45.276308 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:45.276265 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zkrhd" event={"ID":"09fdf756-6037-48d0-a284-719f7770ade3","Type":"ContainerStarted","Data":"666b37b14707d7b11c29a57b9a852cc0d67c9d634ef1f8dd79fae10f62011039"} Jul 13 22:00:45.277700 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:45.277675 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nbdz9" event={"ID":"71a3685e-3328-41cb-8b35-c8e8e1efe021","Type":"ContainerStarted","Data":"54bef700f6031ecd787c75126d8ee6f352888295eb6c7d98bcadcdfc1546a76e"} Jul 13 22:00:45.277827 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:45.277718 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nbdz9" event={"ID":"71a3685e-3328-41cb-8b35-c8e8e1efe021","Type":"ContainerStarted","Data":"c31489d39453a9ecf097a451d46696ee803f99f65f7cd33c6ad540e531031198"} Jul 13 22:00:45.277827 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:45.277789 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 22:00:45.278800 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:45.278767 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vhlb7" event={"ID":"9d1a0995-48f0-46b7-a75d-6c10a0860e7b","Type":"ContainerStarted","Data":"ac8f067396175b6886026e9674d91e5a56e39d051e9dfe5037555507779008cf"} Jul 13 22:00:45.279730 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:45.279709 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w8fht" event={"ID":"a48e3e9b-b015-44af-89b9-42c2977ae9a8","Type":"ContainerStarted","Data":"b339affaaab05c1fa555d6e0e806faa071061566206a0ccc24e244299002411d"} Jul 13 22:00:45.291762 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:45.291721 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nbdz9" podStartSLOduration=66.291708248 podStartE2EDuration="1m6.291708248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:45.291199315 +0000 UTC m=+66.741754854" watchObservedRunningTime="2026-07-13 22:00:45.291708248 +0000 UTC m=+66.742263779" Jul 13 22:00:45.961055 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:45.961023 2562 scope.go:117] "RemoveContainer" containerID="be02e46854fa77ac3ba5d61a7a8797219f8eca65bd0be6011443e18d5453997e" Jul 13 22:00:46.295435 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:46.294248 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x5qhq_7b1fa75b-f29d-45a1-af98-fa947c2058b2/console-operator/1.log" Jul 13 22:00:46.295435 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:46.294620 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-x5qhq" event={"ID":"7b1fa75b-f29d-45a1-af98-fa947c2058b2","Type":"ContainerStarted","Data":"e97122e2a52c1dc6fb9ad909a96ad55387dbcabb6da0facfbf90fe7694f9783a"} Jul 13 22:00:46.295435 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:46.295029 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-x5qhq" Jul 13 22:00:46.427222 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:46.424963 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-tls\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:46.430302 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:46.430199 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-tls\") pod \"image-registry-86f7bbf7c5-9h5nl\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:46.526606 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:46.526375 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06450a5e-03fe-4a9e-a6e8-722e0fc36664-metrics-tls\") pod \"dns-default-qzmt4\" (UID: \"06450a5e-03fe-4a9e-a6e8-722e0fc36664\") " pod="openshift-dns/dns-default-qzmt4" Jul 13 22:00:46.532649 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:46.532590 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06450a5e-03fe-4a9e-a6e8-722e0fc36664-metrics-tls\") pod \"dns-default-qzmt4\" (UID: \"06450a5e-03fe-4a9e-a6e8-722e0fc36664\") " pod="openshift-dns/dns-default-qzmt4" Jul 13 22:00:46.551954 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:46.551691 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s74j5\"" Jul 13 22:00:46.560636 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:46.560359 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:46.661287 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:46.661203 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lvbhm\"" Jul 13 22:00:46.669265 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:46.669016 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qzmt4" Jul 13 22:00:46.934870 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:46.934457 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-x5qhq" podStartSLOduration=21.223604442 podStartE2EDuration="27.934432295s" totalImagesPullingTime="6.710827853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:46.315457447 +0000 UTC m=+67.766012985" watchObservedRunningTime="2026-07-13 22:00:46.934432295 +0000 UTC m=+68.384987832" Jul 13 22:00:46.934870 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:46.934782 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86f7bbf7c5-9h5nl"] Jul 13 22:00:46.939530 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:00:46.939497 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod505f1712_2250_43ff_a8ab_270f7cc2b859.slice/crio-14427a49793df1e61bd38b88949e4da45fd5b9b7842d11981117bb8d85f8b9f8 WatchSource:0}: Error finding container 14427a49793df1e61bd38b88949e4da45fd5b9b7842d11981117bb8d85f8b9f8: Status 404 returned error can't find the container with id 14427a49793df1e61bd38b88949e4da45fd5b9b7842d11981117bb8d85f8b9f8 Jul 13 22:00:46.956901 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:46.956727 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qzmt4"] Jul 13 22:00:46.963951 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:00:46.963921 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06450a5e_03fe_4a9e_a6e8_722e0fc36664.slice/crio-ee81c431bfb6da4cf05deda9df8a5d75b62d50aa788fb78c494fd1f84ca3234e WatchSource:0}: Error finding container ee81c431bfb6da4cf05deda9df8a5d75b62d50aa788fb78c494fd1f84ca3234e: Status 404 returned error can't find the container with id ee81c431bfb6da4cf05deda9df8a5d75b62d50aa788fb78c494fd1f84ca3234e Jul 13 22:00:47.295539 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:47.295207 2562 patch_prober.go:28] interesting pod/console-operator-7866497494-x5qhq container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.15:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 13 22:00:47.295539 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:47.295280 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7866497494-x5qhq" podUID="7b1fa75b-f29d-45a1-af98-fa947c2058b2" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.15:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 13 22:00:47.302692 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:47.302646 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w8fht" event={"ID":"a48e3e9b-b015-44af-89b9-42c2977ae9a8","Type":"ContainerStarted","Data":"a2d0fb7f78c1bf5d494fd172df5fa562c47fdfd28b2efa0b57e8dc1026868f87"} Jul 13 22:00:47.302844 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:47.302706 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-w8fht" event={"ID":"a48e3e9b-b015-44af-89b9-42c2977ae9a8","Type":"ContainerStarted","Data":"b731ced6b5cec79d56481cacad862c3f7a641972ac51bb304dcef99cb360612b"} Jul 13 22:00:47.305377 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:47.305343 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" event={"ID":"505f1712-2250-43ff-a8ab-270f7cc2b859","Type":"ContainerStarted","Data":"ed7b099c38ed08880247b1b1aad6b325f0b34f669f9132a5b478a45671b3e3d8"} Jul 13 22:00:47.305497 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:47.305398 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" event={"ID":"505f1712-2250-43ff-a8ab-270f7cc2b859","Type":"ContainerStarted","Data":"14427a49793df1e61bd38b88949e4da45fd5b9b7842d11981117bb8d85f8b9f8"} Jul 13 22:00:47.305497 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:47.305480 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:00:47.308369 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:47.308339 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qzmt4" event={"ID":"06450a5e-03fe-4a9e-a6e8-722e0fc36664","Type":"ContainerStarted","Data":"ee81c431bfb6da4cf05deda9df8a5d75b62d50aa788fb78c494fd1f84ca3234e"} Jul 13 22:00:47.318827 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:47.318458 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-w8fht" podStartSLOduration=66.758822905 podStartE2EDuration="1m8.318439543s" totalImagesPullingTime="1.559616638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:59:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:47.316917751 +0000 UTC m=+68.767473301" watchObservedRunningTime="2026-07-13 22:00:47.318439543 +0000 UTC m=+68.768995121" Jul 13 22:00:47.336246 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:47.336196 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" podStartSLOduration=68.3361784 podStartE2EDuration="1m8.3361784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 21:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:47.334644403 +0000 UTC m=+68.785199966" watchObservedRunningTime="2026-07-13 22:00:47.3361784 +0000 UTC m=+68.786733940" Jul 13 22:00:48.307960 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:48.307832 2562 patch_prober.go:28] interesting pod/console-operator-7866497494-x5qhq container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.15:8443/readyz\": context deadline exceeded" start-of-body= Jul 13 22:00:48.307960 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:48.307899 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7866497494-x5qhq" podUID="7b1fa75b-f29d-45a1-af98-fa947c2058b2" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.15:8443/readyz\": context deadline exceeded" Jul 13 22:00:51.271829 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:51.271796 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e9588e99-3cae-44f7-82b6-101ae1fbd604-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-dlj2d\" (UID: \"e9588e99-3cae-44f7-82b6-101ae1fbd604\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-dlj2d" Jul 13 22:00:51.274575 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:51.274549 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e9588e99-3cae-44f7-82b6-101ae1fbd604-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-dlj2d\" (UID: \"e9588e99-3cae-44f7-82b6-101ae1fbd604\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-dlj2d" Jul 13 22:00:51.373250 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:51.373212 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbedfa07-690d-4c79-92d3-dc85b88124d7-metrics-certs\") pod \"router-default-5dbcf998b4-7qqv9\" (UID: \"dbedfa07-690d-4c79-92d3-dc85b88124d7\") " pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:51.373250 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:51.373256 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbedfa07-690d-4c79-92d3-dc85b88124d7-service-ca-bundle\") pod \"router-default-5dbcf998b4-7qqv9\" (UID: \"dbedfa07-690d-4c79-92d3-dc85b88124d7\") " pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:51.374012 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:51.373962 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbedfa07-690d-4c79-92d3-dc85b88124d7-service-ca-bundle\") pod \"router-default-5dbcf998b4-7qqv9\" (UID: \"dbedfa07-690d-4c79-92d3-dc85b88124d7\") " pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:51.375642 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:51.375618 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbedfa07-690d-4c79-92d3-dc85b88124d7-metrics-certs\") pod \"router-default-5dbcf998b4-7qqv9\" (UID: \"dbedfa07-690d-4c79-92d3-dc85b88124d7\") " pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:51.393946 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:51.393920 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wwdv5\"" Jul 13 22:00:51.402697 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:51.402665 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-dlj2d" Jul 13 22:00:51.474386 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:51.474349 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/555d9e2d-52a7-4024-aa67-991a477e5ab7-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-c6cfc\" (UID: \"555d9e2d-52a7-4024-aa67-991a477e5ab7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c6cfc" Jul 13 22:00:51.477312 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:51.477261 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/555d9e2d-52a7-4024-aa67-991a477e5ab7-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-c6cfc\" (UID: \"555d9e2d-52a7-4024-aa67-991a477e5ab7\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c6cfc" Jul 13 22:00:51.558150 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:51.558127 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xxrfv\"" Jul 13 22:00:51.566899 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:51.566869 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:51.600850 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:51.600821 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-zfb5v\"" Jul 13 22:00:51.608680 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:51.608660 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c6cfc" Jul 13 22:00:52.180690 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:52.178455 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-dlj2d"] Jul 13 22:00:52.187436 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:00:52.187392 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9588e99_3cae_44f7_82b6_101ae1fbd604.slice/crio-9a023854c6191609180619aa7ab7e77d5662b47d0a711c73e79054814afa8523 WatchSource:0}: Error finding container 9a023854c6191609180619aa7ab7e77d5662b47d0a711c73e79054814afa8523: Status 404 returned error can't find the container with id 9a023854c6191609180619aa7ab7e77d5662b47d0a711c73e79054814afa8523 Jul 13 22:00:52.219030 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:52.218921 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c6cfc"] Jul 13 22:00:52.243333 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:52.243301 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5dbcf998b4-7qqv9"] Jul 13 22:00:52.248437 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:00:52.248405 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbedfa07_690d_4c79_92d3_dc85b88124d7.slice/crio-eb85a19036d4f9fe327d7e318c5f74435efd930c57b8798bfe4777b41806087a WatchSource:0}: Error finding container eb85a19036d4f9fe327d7e318c5f74435efd930c57b8798bfe4777b41806087a: Status 404 returned error can't find the container with id eb85a19036d4f9fe327d7e318c5f74435efd930c57b8798bfe4777b41806087a Jul 13 22:00:52.329595 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:52.329555 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qzmt4" event={"ID":"06450a5e-03fe-4a9e-a6e8-722e0fc36664","Type":"ContainerStarted","Data":"03e351d32fe6993702d2a0971c7bef5245f5c41a934f9004ae632f3dbb879c0a"} Jul 13 22:00:52.331093 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:52.331065 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" event={"ID":"dbedfa07-690d-4c79-92d3-dc85b88124d7","Type":"ContainerStarted","Data":"eb85a19036d4f9fe327d7e318c5f74435efd930c57b8798bfe4777b41806087a"} Jul 13 22:00:52.332497 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:52.332472 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-dlj2d" event={"ID":"e9588e99-3cae-44f7-82b6-101ae1fbd604","Type":"ContainerStarted","Data":"9a023854c6191609180619aa7ab7e77d5662b47d0a711c73e79054814afa8523"} Jul 13 22:00:52.334592 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:52.334556 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zkrhd" event={"ID":"09fdf756-6037-48d0-a284-719f7770ade3","Type":"ContainerStarted","Data":"aa74c6fa715601c4631093cafd96bd141ad3e665b6ea4c5db244f7cb7449d7e3"} Jul 13 22:00:52.336622 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:52.336598 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vhlb7" event={"ID":"9d1a0995-48f0-46b7-a75d-6c10a0860e7b","Type":"ContainerStarted","Data":"b282214c42a3e033e0a07ad9550c956b019e4ba5de3b09e5eccef172840d1889"} Jul 13 22:00:52.338111 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:52.338068 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c6cfc" event={"ID":"555d9e2d-52a7-4024-aa67-991a477e5ab7","Type":"ContainerStarted","Data":"a08f02864096f66eabee87a618e9b88dbdf8371ac9f739ebb5b7da83189c49a4"} Jul 13 22:00:52.382801 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:52.382722 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zkrhd" podStartSLOduration=66.609483601 podStartE2EDuration="1m13.382699481s" totalImagesPullingTime="6.77321588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 21:59:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:52.379820583 +0000 UTC m=+73.830376119" watchObservedRunningTime="2026-07-13 22:00:52.382699481 +0000 UTC m=+73.833255016" Jul 13 22:00:53.349263 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:53.349160 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vhlb7" event={"ID":"9d1a0995-48f0-46b7-a75d-6c10a0860e7b","Type":"ContainerStarted","Data":"73dd2302e55eaf7088e940a60e2dbf204c235ea5648bcad609629a4024ab05a6"} Jul 13 22:00:53.351772 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:53.351729 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qzmt4" event={"ID":"06450a5e-03fe-4a9e-a6e8-722e0fc36664","Type":"ContainerStarted","Data":"bf0215016e21966f4247db0563fcb9a0cab7c7f6f18ab6e4f5578946f09f76e9"} Jul 13 22:00:53.351923 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:53.351879 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qzmt4" Jul 13 22:00:53.353737 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:53.353704 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" event={"ID":"dbedfa07-690d-4c79-92d3-dc85b88124d7","Type":"ContainerStarted","Data":"936a5d581eb2368098a2a1ce37b7860340f5ce2e06283f265c4e8e944e9182b1"} Jul 13 22:00:53.373822 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:53.373764 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qzmt4" podStartSLOduration=34.271576766 podStartE2EDuration="39.3737447s" totalImagesPullingTime="5.102167934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:53.372046036 +0000 UTC m=+74.822601575" watchObservedRunningTime="2026-07-13 22:00:53.3737447 +0000 UTC m=+74.824300238" Jul 13 22:00:53.395528 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:53.395475 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" podStartSLOduration=34.395455219 podStartE2EDuration="34.395455219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:53.393361486 +0000 UTC m=+74.843917016" watchObservedRunningTime="2026-07-13 22:00:53.395455219 +0000 UTC m=+74.846010757" Jul 13 22:00:53.568022 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:53.567967 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:53.571838 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:53.571793 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:54.359760 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:54.359729 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:54.361101 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:54.361079 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5dbcf998b4-7qqv9" Jul 13 22:00:55.466066 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.466035 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d5576bfb7-fdtzv"] Jul 13 22:00:55.474320 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.474296 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d5576bfb7-fdtzv" Jul 13 22:00:55.477871 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.477847 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 13 22:00:55.478100 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.477892 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 13 22:00:55.478100 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.478022 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 13 22:00:55.478606 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.478327 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 13 22:00:55.478606 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.478412 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-crbrs\"" Jul 13 22:00:55.478606 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.478448 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 13 22:00:55.478606 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.478318 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d5576bfb7-fdtzv"] Jul 13 22:00:55.478867 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.478666 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 13 22:00:55.478867 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.478772 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 13 22:00:55.613345 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.613308 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c8rr\" (UniqueName: \"kubernetes.io/projected/2774db86-b0f5-44d4-8188-285a38934e4d-kube-api-access-7c8rr\") pod \"console-6d5576bfb7-fdtzv\" (UID: \"2774db86-b0f5-44d4-8188-285a38934e4d\") " pod="openshift-console/console-6d5576bfb7-fdtzv" Jul 13 22:00:55.613537 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.613371 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2774db86-b0f5-44d4-8188-285a38934e4d-console-serving-cert\") pod \"console-6d5576bfb7-fdtzv\" (UID: \"2774db86-b0f5-44d4-8188-285a38934e4d\") " pod="openshift-console/console-6d5576bfb7-fdtzv" Jul 13 22:00:55.613537 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.613419 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2774db86-b0f5-44d4-8188-285a38934e4d-console-oauth-config\") pod \"console-6d5576bfb7-fdtzv\" (UID: \"2774db86-b0f5-44d4-8188-285a38934e4d\") " pod="openshift-console/console-6d5576bfb7-fdtzv" Jul 13 22:00:55.613537 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.613440 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2774db86-b0f5-44d4-8188-285a38934e4d-service-ca\") pod \"console-6d5576bfb7-fdtzv\" (UID: \"2774db86-b0f5-44d4-8188-285a38934e4d\") " pod="openshift-console/console-6d5576bfb7-fdtzv" Jul 13 22:00:55.613537 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.613458 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2774db86-b0f5-44d4-8188-285a38934e4d-oauth-serving-cert\") pod \"console-6d5576bfb7-fdtzv\" (UID: \"2774db86-b0f5-44d4-8188-285a38934e4d\") " pod="openshift-console/console-6d5576bfb7-fdtzv" Jul 13 22:00:55.613537 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.613514 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2774db86-b0f5-44d4-8188-285a38934e4d-console-config\") pod \"console-6d5576bfb7-fdtzv\" (UID: \"2774db86-b0f5-44d4-8188-285a38934e4d\") " pod="openshift-console/console-6d5576bfb7-fdtzv" Jul 13 22:00:55.714257 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.714215 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2774db86-b0f5-44d4-8188-285a38934e4d-console-oauth-config\") pod \"console-6d5576bfb7-fdtzv\" (UID: \"2774db86-b0f5-44d4-8188-285a38934e4d\") " pod="openshift-console/console-6d5576bfb7-fdtzv" Jul 13 22:00:55.714427 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.714265 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2774db86-b0f5-44d4-8188-285a38934e4d-service-ca\") pod \"console-6d5576bfb7-fdtzv\" (UID: \"2774db86-b0f5-44d4-8188-285a38934e4d\") " pod="openshift-console/console-6d5576bfb7-fdtzv" Jul 13 22:00:55.714427 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.714292 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2774db86-b0f5-44d4-8188-285a38934e4d-oauth-serving-cert\") pod \"console-6d5576bfb7-fdtzv\" (UID: \"2774db86-b0f5-44d4-8188-285a38934e4d\") " pod="openshift-console/console-6d5576bfb7-fdtzv" Jul 13 22:00:55.714427 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.714349 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2774db86-b0f5-44d4-8188-285a38934e4d-console-config\") pod \"console-6d5576bfb7-fdtzv\" (UID: \"2774db86-b0f5-44d4-8188-285a38934e4d\") " pod="openshift-console/console-6d5576bfb7-fdtzv" Jul 13 22:00:55.714427 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.714385 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c8rr\" (UniqueName: \"kubernetes.io/projected/2774db86-b0f5-44d4-8188-285a38934e4d-kube-api-access-7c8rr\") pod \"console-6d5576bfb7-fdtzv\" (UID: \"2774db86-b0f5-44d4-8188-285a38934e4d\") " pod="openshift-console/console-6d5576bfb7-fdtzv" Jul 13 22:00:55.714633 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.714432 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2774db86-b0f5-44d4-8188-285a38934e4d-console-serving-cert\") pod \"console-6d5576bfb7-fdtzv\" (UID: \"2774db86-b0f5-44d4-8188-285a38934e4d\") " pod="openshift-console/console-6d5576bfb7-fdtzv" Jul 13 22:00:55.715078 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.715053 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2774db86-b0f5-44d4-8188-285a38934e4d-console-config\") pod \"console-6d5576bfb7-fdtzv\" (UID: \"2774db86-b0f5-44d4-8188-285a38934e4d\") " pod="openshift-console/console-6d5576bfb7-fdtzv" Jul 13 22:00:55.715199 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.715104 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2774db86-b0f5-44d4-8188-285a38934e4d-oauth-serving-cert\") pod \"console-6d5576bfb7-fdtzv\" (UID: \"2774db86-b0f5-44d4-8188-285a38934e4d\") " pod="openshift-console/console-6d5576bfb7-fdtzv" Jul 13 22:00:55.715315 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.715295 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2774db86-b0f5-44d4-8188-285a38934e4d-service-ca\") pod \"console-6d5576bfb7-fdtzv\" (UID: \"2774db86-b0f5-44d4-8188-285a38934e4d\") " pod="openshift-console/console-6d5576bfb7-fdtzv" Jul 13 22:00:55.716822 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.716770 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2774db86-b0f5-44d4-8188-285a38934e4d-console-oauth-config\") pod \"console-6d5576bfb7-fdtzv\" (UID: \"2774db86-b0f5-44d4-8188-285a38934e4d\") " pod="openshift-console/console-6d5576bfb7-fdtzv" Jul 13 22:00:55.716915 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.716853 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2774db86-b0f5-44d4-8188-285a38934e4d-console-serving-cert\") pod \"console-6d5576bfb7-fdtzv\" (UID: \"2774db86-b0f5-44d4-8188-285a38934e4d\") " pod="openshift-console/console-6d5576bfb7-fdtzv" Jul 13 22:00:55.723408 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.723388 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c8rr\" (UniqueName: \"kubernetes.io/projected/2774db86-b0f5-44d4-8188-285a38934e4d-kube-api-access-7c8rr\") pod \"console-6d5576bfb7-fdtzv\" (UID: \"2774db86-b0f5-44d4-8188-285a38934e4d\") " pod="openshift-console/console-6d5576bfb7-fdtzv" Jul 13 22:00:55.786940 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:55.786901 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d5576bfb7-fdtzv" Jul 13 22:00:56.127732 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:56.127703 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d5576bfb7-fdtzv"] Jul 13 22:00:56.132317 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:00:56.132259 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2774db86_b0f5_44d4_8188_285a38934e4d.slice/crio-df9960904dcbc90c0764ab6b491f713a14b3ec61cf41b90c081a8cfc6e4d611f WatchSource:0}: Error finding container df9960904dcbc90c0764ab6b491f713a14b3ec61cf41b90c081a8cfc6e4d611f: Status 404 returned error can't find the container with id df9960904dcbc90c0764ab6b491f713a14b3ec61cf41b90c081a8cfc6e4d611f Jul 13 22:00:56.371653 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:56.371570 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vhlb7" event={"ID":"9d1a0995-48f0-46b7-a75d-6c10a0860e7b","Type":"ContainerStarted","Data":"2e592c46eb107301cc3f7ac85016168b969b71d0b7362658eef5fb66679c768d"} Jul 13 22:00:56.377634 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:56.374208 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c6cfc" event={"ID":"555d9e2d-52a7-4024-aa67-991a477e5ab7","Type":"ContainerStarted","Data":"150a8bd85247fe29397a924bc55afe67a57c0096933f85a65d04ea0101ace72e"} Jul 13 22:00:56.377634 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:56.374308 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c6cfc" event={"ID":"555d9e2d-52a7-4024-aa67-991a477e5ab7","Type":"ContainerStarted","Data":"d936e82686d15efbc8681218a9a97f915a618808956368aad43f0ebb5d272665"} Jul 13 22:00:56.378829 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:56.378799 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-dlj2d" event={"ID":"e9588e99-3cae-44f7-82b6-101ae1fbd604","Type":"ContainerStarted","Data":"364705d28ff6b3985c81ad35a193e83720fc4caaa10821d7b72f1eb4b3f0b52f"} Jul 13 22:00:56.380069 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:56.380041 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d5576bfb7-fdtzv" event={"ID":"2774db86-b0f5-44d4-8188-285a38934e4d","Type":"ContainerStarted","Data":"df9960904dcbc90c0764ab6b491f713a14b3ec61cf41b90c081a8cfc6e4d611f"} Jul 13 22:00:56.398577 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:56.398526 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vhlb7" podStartSLOduration=17.414925092 podStartE2EDuration="28.398508949s" totalImagesPullingTime="10.983583857s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 22:00:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:56.395858172 +0000 UTC m=+77.846413712" watchObservedRunningTime="2026-07-13 22:00:56.398508949 +0000 UTC m=+77.849064467" Jul 13 22:00:56.416631 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:56.416588 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-c6cfc" podStartSLOduration=33.71050517 podStartE2EDuration="37.416574901s" totalImagesPullingTime="3.706069731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:56.415755576 +0000 UTC m=+77.866311123" watchObservedRunningTime="2026-07-13 22:00:56.416574901 +0000 UTC m=+77.867130437" Jul 13 22:00:56.438047 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:56.438003 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-dlj2d" podStartSLOduration=33.642301601 podStartE2EDuration="37.437987935s" totalImagesPullingTime="3.795686334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:00:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:00:56.436664764 +0000 UTC m=+77.887220303" watchObservedRunningTime="2026-07-13 22:00:56.437987935 +0000 UTC m=+77.888543463" Jul 13 22:00:56.588855 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:56.588815 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gdkcd"] Jul 13 22:00:56.629455 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:56.629374 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gdkcd"] Jul 13 22:00:56.629593 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:56.629491 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gdkcd" Jul 13 22:00:56.631714 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:56.631692 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 13 22:00:56.631834 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:56.631773 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-4dsvb\"" Jul 13 22:00:56.722414 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:56.722372 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/623254a5-bbfa-4d83-bb75-85a15054f2cd-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-gdkcd\" (UID: \"623254a5-bbfa-4d83-bb75-85a15054f2cd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gdkcd" Jul 13 22:00:56.823314 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:56.823281 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/623254a5-bbfa-4d83-bb75-85a15054f2cd-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-gdkcd\" (UID: \"623254a5-bbfa-4d83-bb75-85a15054f2cd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gdkcd" Jul 13 22:00:56.825756 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:56.825734 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/623254a5-bbfa-4d83-bb75-85a15054f2cd-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-gdkcd\" (UID: \"623254a5-bbfa-4d83-bb75-85a15054f2cd\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gdkcd" Jul 13 22:00:56.939597 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:56.939515 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gdkcd" Jul 13 22:00:57.104262 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:57.104230 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gdkcd"] Jul 13 22:00:57.107621 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:00:57.107591 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623254a5_bbfa_4d83_bb75_85a15054f2cd.slice/crio-d4e4cb0b77b8a67877701158d0cb614eae121f0f3356d8de56133e02cb5227f5 WatchSource:0}: Error finding container d4e4cb0b77b8a67877701158d0cb614eae121f0f3356d8de56133e02cb5227f5: Status 404 returned error can't find the container with id d4e4cb0b77b8a67877701158d0cb614eae121f0f3356d8de56133e02cb5227f5 Jul 13 22:00:57.312437 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:57.312412 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-x5qhq" Jul 13 22:00:57.387144 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:00:57.387108 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gdkcd" event={"ID":"623254a5-bbfa-4d83-bb75-85a15054f2cd","Type":"ContainerStarted","Data":"d4e4cb0b77b8a67877701158d0cb614eae121f0f3356d8de56133e02cb5227f5"} Jul 13 22:01:00.401585 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:00.401548 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gdkcd" event={"ID":"623254a5-bbfa-4d83-bb75-85a15054f2cd","Type":"ContainerStarted","Data":"5bebea12d6bd969553aa5b94b2acda83821a1f4f9e82a2e22cbd40c5e1aaa171"} Jul 13 22:01:00.402053 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:00.401611 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gdkcd" Jul 13 22:01:00.403154 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:00.403130 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d5576bfb7-fdtzv" event={"ID":"2774db86-b0f5-44d4-8188-285a38934e4d","Type":"ContainerStarted","Data":"384415ea96f798da5c97986e8d7bb298f217f22f747f348b2b715a454187a798"} Jul 13 22:01:00.406477 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:00.406452 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gdkcd" Jul 13 22:01:00.419068 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:00.419026 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-gdkcd" podStartSLOduration=1.7690156940000001 podStartE2EDuration="4.419013234s" totalImagesPullingTime="2.64999754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:00:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:01:00.41815195 +0000 UTC m=+81.868707488" watchObservedRunningTime="2026-07-13 22:01:00.419013234 +0000 UTC m=+81.869568771" Jul 13 22:01:00.438638 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:00.438580 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d5576bfb7-fdtzv" podStartSLOduration=1.8076907850000001 podStartE2EDuration="5.438564659s" totalImagesPullingTime="3.630873874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:00:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:01:00.438237574 +0000 UTC m=+81.888793111" watchObservedRunningTime="2026-07-13 22:01:00.438564659 +0000 UTC m=+81.889120196" Jul 13 22:01:00.638266 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:00.638222 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-bbnk4"] Jul 13 22:01:00.663250 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:00.663179 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-bbnk4"] Jul 13 22:01:00.663374 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:00.663292 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-bbnk4" Jul 13 22:01:00.665620 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:00.665594 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 13 22:01:00.665620 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:00.665608 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 13 22:01:00.665620 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:00.665608 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-dfdwc\"" Jul 13 22:01:00.665799 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:00.665609 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 13 22:01:00.752787 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:00.752746 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa76e2d1-de61-45ff-824d-38d30db04a95-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-bbnk4\" (UID: \"fa76e2d1-de61-45ff-824d-38d30db04a95\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bbnk4" Jul 13 22:01:00.752787 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:00.752790 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa76e2d1-de61-45ff-824d-38d30db04a95-metrics-client-ca\") pod \"prometheus-operator-64785b6675-bbnk4\" (UID: \"fa76e2d1-de61-45ff-824d-38d30db04a95\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bbnk4" Jul 13 22:01:00.752962 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:00.752867 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xgvc\" (UniqueName: \"kubernetes.io/projected/fa76e2d1-de61-45ff-824d-38d30db04a95-kube-api-access-6xgvc\") pod \"prometheus-operator-64785b6675-bbnk4\" (UID: \"fa76e2d1-de61-45ff-824d-38d30db04a95\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bbnk4" Jul 13 22:01:00.752962 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:00.752937 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa76e2d1-de61-45ff-824d-38d30db04a95-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-bbnk4\" (UID: \"fa76e2d1-de61-45ff-824d-38d30db04a95\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bbnk4" Jul 13 22:01:00.853543 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:00.853420 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xgvc\" (UniqueName: \"kubernetes.io/projected/fa76e2d1-de61-45ff-824d-38d30db04a95-kube-api-access-6xgvc\") pod \"prometheus-operator-64785b6675-bbnk4\" (UID: \"fa76e2d1-de61-45ff-824d-38d30db04a95\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bbnk4" Jul 13 22:01:00.853543 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:00.853480 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa76e2d1-de61-45ff-824d-38d30db04a95-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-bbnk4\" (UID: \"fa76e2d1-de61-45ff-824d-38d30db04a95\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bbnk4" Jul 13 22:01:00.853747 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:00.853653 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa76e2d1-de61-45ff-824d-38d30db04a95-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-bbnk4\" (UID: \"fa76e2d1-de61-45ff-824d-38d30db04a95\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bbnk4" Jul 13 22:01:00.853747 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:00.853703 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa76e2d1-de61-45ff-824d-38d30db04a95-metrics-client-ca\") pod \"prometheus-operator-64785b6675-bbnk4\" (UID: \"fa76e2d1-de61-45ff-824d-38d30db04a95\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bbnk4" Jul 13 22:01:00.853833 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:01:00.853772 2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 13 22:01:00.853865 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:01:00.853838 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa76e2d1-de61-45ff-824d-38d30db04a95-prometheus-operator-tls podName:fa76e2d1-de61-45ff-824d-38d30db04a95 nodeName:}" failed. No retries permitted until 2026-07-13 22:01:01.35381974 +0000 UTC m=+82.804375259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/fa76e2d1-de61-45ff-824d-38d30db04a95-prometheus-operator-tls") pod "prometheus-operator-64785b6675-bbnk4" (UID: "fa76e2d1-de61-45ff-824d-38d30db04a95") : secret "prometheus-operator-tls" not found Jul 13 22:01:00.854366 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:00.854349 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa76e2d1-de61-45ff-824d-38d30db04a95-metrics-client-ca\") pod \"prometheus-operator-64785b6675-bbnk4\" (UID: \"fa76e2d1-de61-45ff-824d-38d30db04a95\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bbnk4" Jul 13 22:01:00.855870 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:00.855848 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa76e2d1-de61-45ff-824d-38d30db04a95-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-bbnk4\" (UID: \"fa76e2d1-de61-45ff-824d-38d30db04a95\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bbnk4" Jul 13 22:01:00.862894 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:00.862873 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xgvc\" (UniqueName: \"kubernetes.io/projected/fa76e2d1-de61-45ff-824d-38d30db04a95-kube-api-access-6xgvc\") pod \"prometheus-operator-64785b6675-bbnk4\" (UID: \"fa76e2d1-de61-45ff-824d-38d30db04a95\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bbnk4" Jul 13 22:01:01.358374 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:01.358332 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa76e2d1-de61-45ff-824d-38d30db04a95-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-bbnk4\" (UID: \"fa76e2d1-de61-45ff-824d-38d30db04a95\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bbnk4" Jul 13 22:01:01.358546 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:01:01.358484 2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 13 22:01:01.358594 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:01:01.358550 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa76e2d1-de61-45ff-824d-38d30db04a95-prometheus-operator-tls podName:fa76e2d1-de61-45ff-824d-38d30db04a95 nodeName:}" failed. No retries permitted until 2026-07-13 22:01:02.358535226 +0000 UTC m=+83.809090741 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/fa76e2d1-de61-45ff-824d-38d30db04a95-prometheus-operator-tls") pod "prometheus-operator-64785b6675-bbnk4" (UID: "fa76e2d1-de61-45ff-824d-38d30db04a95") : secret "prometheus-operator-tls" not found Jul 13 22:01:02.367298 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:02.367254 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa76e2d1-de61-45ff-824d-38d30db04a95-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-bbnk4\" (UID: \"fa76e2d1-de61-45ff-824d-38d30db04a95\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bbnk4" Jul 13 22:01:02.369621 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:02.369596 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa76e2d1-de61-45ff-824d-38d30db04a95-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-bbnk4\" (UID: \"fa76e2d1-de61-45ff-824d-38d30db04a95\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bbnk4" Jul 13 22:01:02.471780 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:02.471734 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-bbnk4" Jul 13 22:01:02.601718 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:02.601684 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-bbnk4"] Jul 13 22:01:02.604563 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:01:02.604537 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa76e2d1_de61_45ff_824d_38d30db04a95.slice/crio-2cb2abb797404d94e635f7f7336584d8e83f890ea37a713a4456a0dd8565529e WatchSource:0}: Error finding container 2cb2abb797404d94e635f7f7336584d8e83f890ea37a713a4456a0dd8565529e: Status 404 returned error can't find the container with id 2cb2abb797404d94e635f7f7336584d8e83f890ea37a713a4456a0dd8565529e Jul 13 22:01:03.361763 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:03.361731 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qzmt4" Jul 13 22:01:03.416949 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:03.416914 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-bbnk4" event={"ID":"fa76e2d1-de61-45ff-824d-38d30db04a95","Type":"ContainerStarted","Data":"2cb2abb797404d94e635f7f7336584d8e83f890ea37a713a4456a0dd8565529e"} Jul 13 22:01:04.422705 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:04.422630 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-bbnk4" event={"ID":"fa76e2d1-de61-45ff-824d-38d30db04a95","Type":"ContainerStarted","Data":"9a2b1d8cd282e7a21ef564258f84914199c4f4f4b6cf07dc2327a8d399422326"} Jul 13 22:01:04.422705 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:04.422691 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-bbnk4" event={"ID":"fa76e2d1-de61-45ff-824d-38d30db04a95","Type":"ContainerStarted","Data":"e96c5a845c690a241c38cdddf61283449e1783c1290d986f9d990db9ee0684a7"} Jul 13 22:01:04.442922 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:04.442872 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-bbnk4" podStartSLOduration=2.956424766 podStartE2EDuration="4.442851465s" totalImagesPullingTime="1.486426699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:01:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:01:04.440772317 +0000 UTC m=+85.891327853" watchObservedRunningTime="2026-07-13 22:01:04.442851465 +0000 UTC m=+85.893407002" Jul 13 22:01:05.787471 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:05.787440 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d5576bfb7-fdtzv" Jul 13 22:01:05.787847 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:05.787499 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d5576bfb7-fdtzv" Jul 13 22:01:05.792200 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:05.792176 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d5576bfb7-fdtzv" Jul 13 22:01:06.077385 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.077312 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cxxqb"] Jul 13 22:01:06.080928 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.080909 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.083595 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.083574 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 13 22:01:06.084019 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.083999 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-k57rd\"" Jul 13 22:01:06.084176 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.084052 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 13 22:01:06.084460 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.084444 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 13 22:01:06.095286 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.095268 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.095375 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.095304 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-root\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.095375 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.095329 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-node-exporter-tls\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.095375 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.095358 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-sys\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.095492 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.095414 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-node-exporter-wtmp\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.095492 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.095431 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-metrics-client-ca\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.095492 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.095484 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-node-exporter-accelerators-collector-config\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.095603 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.095501 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87frv\" (UniqueName: \"kubernetes.io/projected/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-kube-api-access-87frv\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.095603 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.095519 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-node-exporter-textfile\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.196147 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.196113 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-node-exporter-accelerators-collector-config\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.196147 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.196147 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87frv\" (UniqueName: \"kubernetes.io/projected/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-kube-api-access-87frv\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.196356 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.196165 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-node-exporter-textfile\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.196356 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.196228 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.196356 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.196292 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-root\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.196356 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.196323 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-node-exporter-tls\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.196558 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.196358 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-sys\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.196558 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.196411 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-node-exporter-wtmp\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.196558 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.196434 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-metrics-client-ca\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.196558 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.196429 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-root\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.196746 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.196556 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-node-exporter-wtmp\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.196746 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.196559 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-sys\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.196746 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:01:06.196612 2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 13 22:01:06.196746 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.196640 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-node-exporter-textfile\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.196746 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:01:06.196689 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-node-exporter-tls podName:fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8 nodeName:}" failed. No retries permitted until 2026-07-13 22:01:06.696667858 +0000 UTC m=+88.147223384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-node-exporter-tls") pod "node-exporter-cxxqb" (UID: "fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8") : secret "node-exporter-tls" not found Jul 13 22:01:06.196908 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.196834 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-node-exporter-accelerators-collector-config\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.196952 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.196930 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-metrics-client-ca\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.198676 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.198654 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.206459 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.206427 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87frv\" (UniqueName: \"kubernetes.io/projected/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-kube-api-access-87frv\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.439774 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.439691 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d5576bfb7-fdtzv" Jul 13 22:01:06.702134 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.702101 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-node-exporter-tls\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.704400 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.704371 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8-node-exporter-tls\") pod \"node-exporter-cxxqb\" (UID: \"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8\") " pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:06.989549 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:06.989469 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cxxqb" Jul 13 22:01:07.002844 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:01:07.002811 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc07e6f6_e23b_4c1d_8ad5_56c86e3bcca8.slice/crio-5389a6261e7100cc4cb4787c127c19f059b7f75f28fbe182e2e8f5bc06ca4249 WatchSource:0}: Error finding container 5389a6261e7100cc4cb4787c127c19f059b7f75f28fbe182e2e8f5bc06ca4249: Status 404 returned error can't find the container with id 5389a6261e7100cc4cb4787c127c19f059b7f75f28fbe182e2e8f5bc06ca4249 Jul 13 22:01:07.098714 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.098684 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 13 22:01:07.103476 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.103455 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.105928 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.105901 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 13 22:01:07.106084 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.105924 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 13 22:01:07.106084 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.106030 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 13 22:01:07.106288 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.106265 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 13 22:01:07.106401 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.106386 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 13 22:01:07.106657 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.106641 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 13 22:01:07.106790 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.106759 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 13 22:01:07.106891 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.106864 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 13 22:01:07.107009 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.106964 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-99ns8\"" Jul 13 22:01:07.107132 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.107117 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 13 22:01:07.119749 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.119729 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 13 22:01:07.205246 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.205215 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd8be815-5e2f-4987-ab66-f8a4d663ae49-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.205246 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.205248 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fd8be815-5e2f-4987-ab66-f8a4d663ae49-config-volume\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.205479 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.205269 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd8be815-5e2f-4987-ab66-f8a4d663ae49-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.205479 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.205288 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd8be815-5e2f-4987-ab66-f8a4d663ae49-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.205479 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.205305 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fd8be815-5e2f-4987-ab66-f8a4d663ae49-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.205479 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.205343 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd8be815-5e2f-4987-ab66-f8a4d663ae49-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.205479 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.205457 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd8be815-5e2f-4987-ab66-f8a4d663ae49-web-config\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.205714 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.205493 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd8be815-5e2f-4987-ab66-f8a4d663ae49-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.205714 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.205536 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fd8be815-5e2f-4987-ab66-f8a4d663ae49-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.205714 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.205571 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltsnr\" (UniqueName: \"kubernetes.io/projected/fd8be815-5e2f-4987-ab66-f8a4d663ae49-kube-api-access-ltsnr\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.205714 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.205626 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fd8be815-5e2f-4987-ab66-f8a4d663ae49-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.205714 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.205659 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fd8be815-5e2f-4987-ab66-f8a4d663ae49-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.205714 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.205700 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd8be815-5e2f-4987-ab66-f8a4d663ae49-config-out\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.306847 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.306759 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fd8be815-5e2f-4987-ab66-f8a4d663ae49-config-volume\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.306847 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.306801 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd8be815-5e2f-4987-ab66-f8a4d663ae49-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.306847 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.306828 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd8be815-5e2f-4987-ab66-f8a4d663ae49-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.307105 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.307080 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fd8be815-5e2f-4987-ab66-f8a4d663ae49-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.307162 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.307128 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd8be815-5e2f-4987-ab66-f8a4d663ae49-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.307213 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.307176 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd8be815-5e2f-4987-ab66-f8a4d663ae49-web-config\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.307213 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.307203 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd8be815-5e2f-4987-ab66-f8a4d663ae49-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.307314 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.307240 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fd8be815-5e2f-4987-ab66-f8a4d663ae49-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.307314 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.307266 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltsnr\" (UniqueName: \"kubernetes.io/projected/fd8be815-5e2f-4987-ab66-f8a4d663ae49-kube-api-access-ltsnr\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.307314 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.307299 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fd8be815-5e2f-4987-ab66-f8a4d663ae49-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.307471 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.307365 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fd8be815-5e2f-4987-ab66-f8a4d663ae49-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.307471 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.307402 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd8be815-5e2f-4987-ab66-f8a4d663ae49-config-out\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.307471 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.307429 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd8be815-5e2f-4987-ab66-f8a4d663ae49-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.308216 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.308191 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd8be815-5e2f-4987-ab66-f8a4d663ae49-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.309212 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.308817 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd8be815-5e2f-4987-ab66-f8a4d663ae49-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.309763 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.309733 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fd8be815-5e2f-4987-ab66-f8a4d663ae49-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.309873 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.309845 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fd8be815-5e2f-4987-ab66-f8a4d663ae49-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.310305 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.310283 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fd8be815-5e2f-4987-ab66-f8a4d663ae49-config-volume\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.310404 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.310334 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fd8be815-5e2f-4987-ab66-f8a4d663ae49-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.310770 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.310718 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fd8be815-5e2f-4987-ab66-f8a4d663ae49-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.311103 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.311080 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fd8be815-5e2f-4987-ab66-f8a4d663ae49-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.311195 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.311183 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fd8be815-5e2f-4987-ab66-f8a4d663ae49-web-config\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.313165 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.313123 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fd8be815-5e2f-4987-ab66-f8a4d663ae49-config-out\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.313280 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.313162 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fd8be815-5e2f-4987-ab66-f8a4d663ae49-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.313346 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.313332 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fd8be815-5e2f-4987-ab66-f8a4d663ae49-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.314914 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.314895 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltsnr\" (UniqueName: \"kubernetes.io/projected/fd8be815-5e2f-4987-ab66-f8a4d663ae49-kube-api-access-ltsnr\") pod \"alertmanager-main-0\" (UID: \"fd8be815-5e2f-4987-ab66-f8a4d663ae49\") " pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.415205 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.415174 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 13 22:01:07.438997 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.438957 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cxxqb" event={"ID":"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8","Type":"ContainerStarted","Data":"5389a6261e7100cc4cb4787c127c19f059b7f75f28fbe182e2e8f5bc06ca4249"} Jul 13 22:01:07.565699 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:07.565530 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 13 22:01:07.568565 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:01:07.568525 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd8be815_5e2f_4987_ab66_f8a4d663ae49.slice/crio-5188d075885548e450522b9c7663b8477fa49c5a8882c623f2cb47962cf435dc WatchSource:0}: Error finding container 5188d075885548e450522b9c7663b8477fa49c5a8882c623f2cb47962cf435dc: Status 404 returned error can't find the container with id 5188d075885548e450522b9c7663b8477fa49c5a8882c623f2cb47962cf435dc Jul 13 22:01:08.317679 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:08.317612 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:01:08.445208 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:08.445165 2562 generic.go:356] "Generic (PLEG): container finished" podID="fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8" containerID="899741a939113a2acb755ab088c1c34e8275cbd11ef5eeb4b0fd786103316264" exitCode=0 Jul 13 22:01:08.445386 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:08.445217 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cxxqb" event={"ID":"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8","Type":"ContainerDied","Data":"899741a939113a2acb755ab088c1c34e8275cbd11ef5eeb4b0fd786103316264"} Jul 13 22:01:08.446668 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:08.446636 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd8be815-5e2f-4987-ab66-f8a4d663ae49","Type":"ContainerStarted","Data":"5188d075885548e450522b9c7663b8477fa49c5a8882c623f2cb47962cf435dc"} Jul 13 22:01:09.452129 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:09.452092 2562 generic.go:356] "Generic (PLEG): container finished" podID="fd8be815-5e2f-4987-ab66-f8a4d663ae49" containerID="c009101289c94e7484cfd4e6303d8544029e11723402aa008e09fe191896f188" exitCode=0 Jul 13 22:01:09.452551 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:09.452194 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd8be815-5e2f-4987-ab66-f8a4d663ae49","Type":"ContainerDied","Data":"c009101289c94e7484cfd4e6303d8544029e11723402aa008e09fe191896f188"} Jul 13 22:01:09.454454 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:09.454433 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cxxqb" event={"ID":"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8","Type":"ContainerStarted","Data":"5f3220222259670cf3e22bd724191122334ed3ea8a58b49cf05376b4ce03b43e"} Jul 13 22:01:09.454555 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:09.454472 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cxxqb" event={"ID":"fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8","Type":"ContainerStarted","Data":"815dcbf7535577261692bd92172d487f81ad165cddb9f69ea03eb514c3ba74cf"} Jul 13 22:01:09.501072 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:09.499645 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cxxqb" podStartSLOduration=2.466864042 podStartE2EDuration="3.499626291s" totalImagesPullingTime="958.38905ms" totalInitContainerRuntime="74.373199ms" isStatefulPod=true podCreationTimestamp="2026-07-13 22:01:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:01:09.498452082 +0000 UTC m=+90.949007619" watchObservedRunningTime="2026-07-13 22:01:09.499626291 +0000 UTC m=+90.950181828" Jul 13 22:01:10.773145 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:10.773111 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-nlnfs"] Jul 13 22:01:10.776472 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:10.776451 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-nlnfs" Jul 13 22:01:10.778491 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:10.778468 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 13 22:01:10.778593 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:10.778506 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-nwc8k\"" Jul 13 22:01:10.783640 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:10.783615 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-nlnfs"] Jul 13 22:01:10.839704 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:10.839676 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2xtp\" (UniqueName: \"kubernetes.io/projected/90958798-39c1-49ab-9d43-e1ca66a4e00a-kube-api-access-l2xtp\") pod \"monitoring-plugin-5f68df7f58-nlnfs\" (UID: \"90958798-39c1-49ab-9d43-e1ca66a4e00a\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-nlnfs" Jul 13 22:01:10.839849 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:10.839776 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/90958798-39c1-49ab-9d43-e1ca66a4e00a-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-nlnfs\" (UID: \"90958798-39c1-49ab-9d43-e1ca66a4e00a\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-nlnfs" Jul 13 22:01:10.940712 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:10.940678 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/90958798-39c1-49ab-9d43-e1ca66a4e00a-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-nlnfs\" (UID: \"90958798-39c1-49ab-9d43-e1ca66a4e00a\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-nlnfs" Jul 13 22:01:10.940937 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:10.940752 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2xtp\" (UniqueName: \"kubernetes.io/projected/90958798-39c1-49ab-9d43-e1ca66a4e00a-kube-api-access-l2xtp\") pod \"monitoring-plugin-5f68df7f58-nlnfs\" (UID: \"90958798-39c1-49ab-9d43-e1ca66a4e00a\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-nlnfs" Jul 13 22:01:10.943615 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:10.943588 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/90958798-39c1-49ab-9d43-e1ca66a4e00a-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-nlnfs\" (UID: \"90958798-39c1-49ab-9d43-e1ca66a4e00a\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-nlnfs" Jul 13 22:01:10.952226 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:10.952187 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2xtp\" (UniqueName: \"kubernetes.io/projected/90958798-39c1-49ab-9d43-e1ca66a4e00a-kube-api-access-l2xtp\") pod \"monitoring-plugin-5f68df7f58-nlnfs\" (UID: \"90958798-39c1-49ab-9d43-e1ca66a4e00a\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-nlnfs" Jul 13 22:01:11.086536 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:11.086441 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-nlnfs" Jul 13 22:01:11.222792 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:11.222763 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-nlnfs"] Jul 13 22:01:11.225762 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:01:11.225736 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90958798_39c1_49ab_9d43_e1ca66a4e00a.slice/crio-b9f7bf2396162ee5bcd2d4a527bf43f05f6c995226f812f8d25175d1f4eba121 WatchSource:0}: Error finding container b9f7bf2396162ee5bcd2d4a527bf43f05f6c995226f812f8d25175d1f4eba121: Status 404 returned error can't find the container with id b9f7bf2396162ee5bcd2d4a527bf43f05f6c995226f812f8d25175d1f4eba121 Jul 13 22:01:11.463937 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:11.463902 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-nlnfs" event={"ID":"90958798-39c1-49ab-9d43-e1ca66a4e00a","Type":"ContainerStarted","Data":"b9f7bf2396162ee5bcd2d4a527bf43f05f6c995226f812f8d25175d1f4eba121"} Jul 13 22:01:12.272157 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.272004 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 22:01:12.276750 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.276719 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.282288 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.281503 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 13 22:01:12.282422 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.282379 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 13 22:01:12.282484 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.282430 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 13 22:01:12.282576 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.282552 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 13 22:01:12.283098 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.282899 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 13 22:01:12.283098 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.283084 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 13 22:01:12.283259 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.283104 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 13 22:01:12.283259 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.283147 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 13 22:01:12.283259 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.283224 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mmtht\"" Jul 13 22:01:12.283259 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.283233 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 13 22:01:12.283446 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.283339 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 13 22:01:12.283446 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.283368 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 13 22:01:12.283548 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.283505 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 13 22:01:12.283763 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.283653 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 13 22:01:12.284125 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.284015 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a6i8ghuqjb34e\"" Jul 13 22:01:12.285409 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.285338 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 13 22:01:12.288947 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.287644 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 13 22:01:12.299229 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.299208 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 22:01:12.353365 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.353330 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.353365 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.353368 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.353565 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.353386 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.353565 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.353402 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-web-config\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.353565 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.353461 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7s7n\" (UniqueName: \"kubernetes.io/projected/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-kube-api-access-j7s7n\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.353565 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.353525 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.353565 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.353564 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-config\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.353754 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.353596 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.353754 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.353635 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.353754 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.353662 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-config-out\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.353754 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.353692 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.353754 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.353733 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.353920 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.353791 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.353920 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.353817 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.353920 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.353840 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.353920 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.353902 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.354115 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.353933 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.354115 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.353957 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.354115 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.354049 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.354115 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.354087 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.455455 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.455402 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.455455 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.455458 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-config-out\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.455699 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.455604 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.455699 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.455654 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.455799 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.455709 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.455799 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.455737 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.455799 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.455763 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.455948 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.455816 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.455948 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.455847 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.455948 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.455874 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.455948 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.455920 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.455948 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.455945 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.456229 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.456006 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.456229 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.456033 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.456229 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.456058 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.456229 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.456081 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-web-config\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.456229 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.456107 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7s7n\" (UniqueName: \"kubernetes.io/projected/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-kube-api-access-j7s7n\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.456229 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.456150 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.456229 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.456195 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-config\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.456229 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.456229 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.456590 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.456303 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.456590 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.456406 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.457414 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.457382 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.458090 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.458064 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.458558 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.458532 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.459049 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.459028 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-config-out\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.460295 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.459933 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.460808 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.460464 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.461587 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.461563 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.464207 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.463380 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.464207 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.463389 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-web-config\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.464207 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.463788 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.464207 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.463855 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.464207 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.464158 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-config\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.464967 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.464581 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.464967 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.464929 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.466624 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.466575 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.466758 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.466734 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.467254 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.467230 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.469396 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.469355 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7s7n\" (UniqueName: \"kubernetes.io/projected/6ebde4e7-796f-4120-ba84-1a0b516a8d1e-kube-api-access-j7s7n\") pod \"prometheus-k8s-0\" (UID: \"6ebde4e7-796f-4120-ba84-1a0b516a8d1e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.474109 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.474017 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd8be815-5e2f-4987-ab66-f8a4d663ae49","Type":"ContainerStarted","Data":"1b1b81105556732a047b551c33d1424c89c8cfbb59738d7d73d879b648674042"} Jul 13 22:01:12.474109 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.474102 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd8be815-5e2f-4987-ab66-f8a4d663ae49","Type":"ContainerStarted","Data":"a4c6135b6b5ede09173346a84897a90228144c63c836530f6e841f68df7ff0d7"} Jul 13 22:01:12.474263 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.474148 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd8be815-5e2f-4987-ab66-f8a4d663ae49","Type":"ContainerStarted","Data":"d9e74bba29bcd0eb21b39674e571724476a76b9f6375adc08af8d57a39b2349d"} Jul 13 22:01:12.474263 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.474185 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd8be815-5e2f-4987-ab66-f8a4d663ae49","Type":"ContainerStarted","Data":"eb6e2c60df5b56038139bd536b60c7531a04412ea5bb52b3d70144943a869293"} Jul 13 22:01:12.474263 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.474222 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd8be815-5e2f-4987-ab66-f8a4d663ae49","Type":"ContainerStarted","Data":"b7be2de9d1b3fcfcc7b23782fd3060ad5b4e585f5c4d1c074f3eb1dd1335620c"} Jul 13 22:01:12.592732 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.592635 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:12.862360 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:12.862334 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 13 22:01:12.863858 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:01:12.863836 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebde4e7_796f_4120_ba84_1a0b516a8d1e.slice/crio-708ee6c433c538f5031e4d9db65c3f46a97fb03bc9f0e9fff566a36f6076dfd3 WatchSource:0}: Error finding container 708ee6c433c538f5031e4d9db65c3f46a97fb03bc9f0e9fff566a36f6076dfd3: Status 404 returned error can't find the container with id 708ee6c433c538f5031e4d9db65c3f46a97fb03bc9f0e9fff566a36f6076dfd3 Jul 13 22:01:13.480453 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:13.480415 2562 generic.go:356] "Generic (PLEG): container finished" podID="6ebde4e7-796f-4120-ba84-1a0b516a8d1e" containerID="ccf0e071dab9c0c8fdf2bb6ba9d13b5154b42a5fd34880a915bf6b597718a4d3" exitCode=0 Jul 13 22:01:13.480818 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:13.480506 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ebde4e7-796f-4120-ba84-1a0b516a8d1e","Type":"ContainerDied","Data":"ccf0e071dab9c0c8fdf2bb6ba9d13b5154b42a5fd34880a915bf6b597718a4d3"} Jul 13 22:01:13.480818 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:13.480590 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ebde4e7-796f-4120-ba84-1a0b516a8d1e","Type":"ContainerStarted","Data":"708ee6c433c538f5031e4d9db65c3f46a97fb03bc9f0e9fff566a36f6076dfd3"} Jul 13 22:01:13.482223 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:13.482187 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-nlnfs" event={"ID":"90958798-39c1-49ab-9d43-e1ca66a4e00a","Type":"ContainerStarted","Data":"4de9a3563f2170e96d8f629e38180320668e5e6ff879ce8a7fc2e26a3ea92d72"} Jul 13 22:01:13.482416 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:13.482342 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-nlnfs" Jul 13 22:01:13.486018 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:13.485969 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fd8be815-5e2f-4987-ab66-f8a4d663ae49","Type":"ContainerStarted","Data":"24316630d3e4e15df46c544346b6e4804e9e79c6b7548b0e84ac982af51bb741"} Jul 13 22:01:13.488606 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:13.488587 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-nlnfs" Jul 13 22:01:13.520643 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:13.520590 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-nlnfs" podStartSLOduration=1.971092179 podStartE2EDuration="3.520576639s" totalImagesPullingTime="1.54948446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:01:13.519305849 +0000 UTC m=+94.969861385" watchObservedRunningTime="2026-07-13 22:01:13.520576639 +0000 UTC m=+94.971132197" Jul 13 22:01:13.547523 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:13.547465 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.930075362 podStartE2EDuration="6.547447498s" totalImagesPullingTime="4.598358322s" totalInitContainerRuntime="19.013814ms" isStatefulPod=false podCreationTimestamp="2026-07-13 22:01:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:01:13.545159243 +0000 UTC m=+94.995714781" watchObservedRunningTime="2026-07-13 22:01:13.547447498 +0000 UTC m=+94.998003037" Jul 13 22:01:13.640631 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:13.640597 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-86f7bbf7c5-9h5nl"] Jul 13 22:01:14.548022 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:14.547984 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-b88478dd6-js7fb"] Jul 13 22:01:14.551314 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:14.551297 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:01:14.559858 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:14.559761 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 13 22:01:14.561701 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:14.561678 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b88478dd6-js7fb"] Jul 13 22:01:14.680024 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:14.679988 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-service-ca\") pod \"console-b88478dd6-js7fb\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:01:14.680024 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:14.680026 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-console-config\") pod \"console-b88478dd6-js7fb\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:01:14.680218 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:14.680056 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-trusted-ca-bundle\") pod \"console-b88478dd6-js7fb\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:01:14.680218 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:14.680096 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-oauth-serving-cert\") pod \"console-b88478dd6-js7fb\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:01:14.680218 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:14.680159 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d92rr\" (UniqueName: \"kubernetes.io/projected/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-kube-api-access-d92rr\") pod \"console-b88478dd6-js7fb\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:01:14.680361 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:14.680216 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-console-serving-cert\") pod \"console-b88478dd6-js7fb\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:01:14.680361 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:14.680332 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-console-oauth-config\") pod \"console-b88478dd6-js7fb\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:01:14.781225 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:14.781185 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d92rr\" (UniqueName: \"kubernetes.io/projected/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-kube-api-access-d92rr\") pod \"console-b88478dd6-js7fb\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:01:14.781411 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:14.781250 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-console-serving-cert\") pod \"console-b88478dd6-js7fb\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:01:14.781411 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:14.781333 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-console-oauth-config\") pod \"console-b88478dd6-js7fb\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:01:14.781411 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:14.781387 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-service-ca\") pod \"console-b88478dd6-js7fb\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:01:14.781546 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:14.781413 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-console-config\") pod \"console-b88478dd6-js7fb\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:01:14.781546 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:14.781437 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-trusted-ca-bundle\") pod \"console-b88478dd6-js7fb\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:01:14.781546 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:14.781469 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-oauth-serving-cert\") pod \"console-b88478dd6-js7fb\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:01:14.782377 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:14.782320 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-console-config\") pod \"console-b88478dd6-js7fb\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:01:14.782598 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:14.782530 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-trusted-ca-bundle\") pod \"console-b88478dd6-js7fb\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:01:14.782598 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:14.782546 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-service-ca\") pod \"console-b88478dd6-js7fb\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:01:14.782804 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:14.782786 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-oauth-serving-cert\") pod \"console-b88478dd6-js7fb\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:01:14.784139 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:14.784117 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-console-serving-cert\") pod \"console-b88478dd6-js7fb\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:01:14.784238 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:14.784155 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-console-oauth-config\") pod \"console-b88478dd6-js7fb\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:01:14.789593 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:14.789573 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d92rr\" (UniqueName: \"kubernetes.io/projected/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-kube-api-access-d92rr\") pod \"console-b88478dd6-js7fb\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:01:14.862389 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:14.862312 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:01:15.013257 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:15.013220 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b88478dd6-js7fb"] Jul 13 22:01:15.016521 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:01:15.016495 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f93cc6_544b_4a28_bc4c_cc5078f824c2.slice/crio-ea86db66d0eb8a98a612976bf19d39f7ef626cec051e875c6782303e4cc33820 WatchSource:0}: Error finding container ea86db66d0eb8a98a612976bf19d39f7ef626cec051e875c6782303e4cc33820: Status 404 returned error can't find the container with id ea86db66d0eb8a98a612976bf19d39f7ef626cec051e875c6782303e4cc33820 Jul 13 22:01:15.498614 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:15.498571 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b88478dd6-js7fb" event={"ID":"a8f93cc6-544b-4a28-bc4c-cc5078f824c2","Type":"ContainerStarted","Data":"bd28d723289a0606c579194c18f73967b6f7ca99d112e94a286aa7435250720e"} Jul 13 22:01:15.498805 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:15.498637 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b88478dd6-js7fb" event={"ID":"a8f93cc6-544b-4a28-bc4c-cc5078f824c2","Type":"ContainerStarted","Data":"ea86db66d0eb8a98a612976bf19d39f7ef626cec051e875c6782303e4cc33820"} Jul 13 22:01:15.517450 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:15.517393 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-b88478dd6-js7fb" podStartSLOduration=1.517373276 podStartE2EDuration="1.517373276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:01:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:01:15.516094513 +0000 UTC m=+96.966650051" watchObservedRunningTime="2026-07-13 22:01:15.517373276 +0000 UTC m=+96.967928814" Jul 13 22:01:16.297731 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:16.297699 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nbdz9" Jul 13 22:01:17.516287 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:17.516247 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ebde4e7-796f-4120-ba84-1a0b516a8d1e","Type":"ContainerStarted","Data":"4ee4816fcbbc5dc104cd1e63cd6308ffbd3bc6e1d15551abc4754cc612cede0f"} Jul 13 22:01:17.516696 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:17.516337 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ebde4e7-796f-4120-ba84-1a0b516a8d1e","Type":"ContainerStarted","Data":"40ca6db70c40d4f52157e9ca4f14c7f76ef0269f70618badba45a462d9b83822"} Jul 13 22:01:19.529594 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:19.529552 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ebde4e7-796f-4120-ba84-1a0b516a8d1e","Type":"ContainerStarted","Data":"4f7cfe3319c19da1359843088fe6158837efb58c67683b6f8971f97c38385884"} Jul 13 22:01:19.530139 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:19.529642 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ebde4e7-796f-4120-ba84-1a0b516a8d1e","Type":"ContainerStarted","Data":"699b44e15ddaa576c07ba5ed49bbf5318ae5d2307d9e1d6f97a50565e455e1c2"} Jul 13 22:01:19.530139 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:19.529686 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ebde4e7-796f-4120-ba84-1a0b516a8d1e","Type":"ContainerStarted","Data":"cb7e4a1f0ea8d9d738d35820c12c0f789c65296623868b6064736b27316c360f"} Jul 13 22:01:19.530139 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:19.529736 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6ebde4e7-796f-4120-ba84-1a0b516a8d1e","Type":"ContainerStarted","Data":"fa1b94b8136adafc431770fc8eee9f043aaad4626b2abdee9dacd139e62081eb"} Jul 13 22:01:19.563121 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:19.563065 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.508827644 podStartE2EDuration="7.563046734s" totalImagesPullingTime="5.033130702s" totalInitContainerRuntime="21.088388ms" isStatefulPod=false podCreationTimestamp="2026-07-13 22:01:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:01:19.56143658 +0000 UTC m=+101.011992116" watchObservedRunningTime="2026-07-13 22:01:19.563046734 +0000 UTC m=+101.013602272" Jul 13 22:01:22.593458 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:22.593419 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:01:24.862646 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:24.862608 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:01:24.863122 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:24.862699 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:01:24.867848 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:24.867829 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:01:25.563290 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:25.563262 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:01:25.633014 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:25.632968 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d5576bfb7-fdtzv"] Jul 13 22:01:38.668042 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:38.667961 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" podUID="505f1712-2250-43ff-a8ab-270f7cc2b859" containerName="registry" containerID="cri-o://ed7b099c38ed08880247b1b1aad6b325f0b34f669f9132a5b478a45671b3e3d8" gracePeriod=30 Jul 13 22:01:38.931651 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:38.931583 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:01:39.113865 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.113812 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/505f1712-2250-43ff-a8ab-270f7cc2b859-installation-pull-secrets\") pod \"505f1712-2250-43ff-a8ab-270f7cc2b859\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " Jul 13 22:01:39.113865 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.113873 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-tls\") pod \"505f1712-2250-43ff-a8ab-270f7cc2b859\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " Jul 13 22:01:39.114147 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.113939 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/505f1712-2250-43ff-a8ab-270f7cc2b859-image-registry-private-configuration\") pod \"505f1712-2250-43ff-a8ab-270f7cc2b859\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " Jul 13 22:01:39.114147 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.113964 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-certificates\") pod \"505f1712-2250-43ff-a8ab-270f7cc2b859\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " Jul 13 22:01:39.114147 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.114018 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gttvw\" (UniqueName: \"kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-kube-api-access-gttvw\") pod \"505f1712-2250-43ff-a8ab-270f7cc2b859\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " Jul 13 22:01:39.114147 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.114077 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-bound-sa-token\") pod \"505f1712-2250-43ff-a8ab-270f7cc2b859\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " Jul 13 22:01:39.114147 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.114110 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/505f1712-2250-43ff-a8ab-270f7cc2b859-ca-trust-extracted\") pod \"505f1712-2250-43ff-a8ab-270f7cc2b859\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " Jul 13 22:01:39.114147 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.114148 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/505f1712-2250-43ff-a8ab-270f7cc2b859-trusted-ca\") pod \"505f1712-2250-43ff-a8ab-270f7cc2b859\" (UID: \"505f1712-2250-43ff-a8ab-270f7cc2b859\") " Jul 13 22:01:39.114586 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.114555 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "505f1712-2250-43ff-a8ab-270f7cc2b859" (UID: "505f1712-2250-43ff-a8ab-270f7cc2b859"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:01:39.114784 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.114756 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/505f1712-2250-43ff-a8ab-270f7cc2b859-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "505f1712-2250-43ff-a8ab-270f7cc2b859" (UID: "505f1712-2250-43ff-a8ab-270f7cc2b859"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:01:39.116630 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.116599 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "505f1712-2250-43ff-a8ab-270f7cc2b859" (UID: "505f1712-2250-43ff-a8ab-270f7cc2b859"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:01:39.116731 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.116672 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/505f1712-2250-43ff-a8ab-270f7cc2b859-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "505f1712-2250-43ff-a8ab-270f7cc2b859" (UID: "505f1712-2250-43ff-a8ab-270f7cc2b859"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:01:39.116731 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.116724 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-kube-api-access-gttvw" (OuterVolumeSpecName: "kube-api-access-gttvw") pod "505f1712-2250-43ff-a8ab-270f7cc2b859" (UID: "505f1712-2250-43ff-a8ab-270f7cc2b859"). InnerVolumeSpecName "kube-api-access-gttvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:01:39.116886 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.116856 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "505f1712-2250-43ff-a8ab-270f7cc2b859" (UID: "505f1712-2250-43ff-a8ab-270f7cc2b859"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:01:39.116947 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.116916 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/505f1712-2250-43ff-a8ab-270f7cc2b859-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "505f1712-2250-43ff-a8ab-270f7cc2b859" (UID: "505f1712-2250-43ff-a8ab-270f7cc2b859"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:01:39.122724 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.122700 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/505f1712-2250-43ff-a8ab-270f7cc2b859-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "505f1712-2250-43ff-a8ab-270f7cc2b859" (UID: "505f1712-2250-43ff-a8ab-270f7cc2b859"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:01:39.215084 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.215056 2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/505f1712-2250-43ff-a8ab-270f7cc2b859-image-registry-private-configuration\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:01:39.215084 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.215085 2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-certificates\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:01:39.215263 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.215095 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gttvw\" (UniqueName: \"kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-kube-api-access-gttvw\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:01:39.215263 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.215104 2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-bound-sa-token\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:01:39.215263 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.215113 2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/505f1712-2250-43ff-a8ab-270f7cc2b859-ca-trust-extracted\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:01:39.215263 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.215121 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/505f1712-2250-43ff-a8ab-270f7cc2b859-trusted-ca\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:01:39.215263 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.215130 2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/505f1712-2250-43ff-a8ab-270f7cc2b859-installation-pull-secrets\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:01:39.215263 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.215138 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/505f1712-2250-43ff-a8ab-270f7cc2b859-registry-tls\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:01:39.622863 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.622771 2562 generic.go:356] "Generic (PLEG): container finished" podID="505f1712-2250-43ff-a8ab-270f7cc2b859" containerID="ed7b099c38ed08880247b1b1aad6b325f0b34f669f9132a5b478a45671b3e3d8" exitCode=0 Jul 13 22:01:39.622863 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.622815 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" event={"ID":"505f1712-2250-43ff-a8ab-270f7cc2b859","Type":"ContainerDied","Data":"ed7b099c38ed08880247b1b1aad6b325f0b34f669f9132a5b478a45671b3e3d8"} Jul 13 22:01:39.623071 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.622860 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" Jul 13 22:01:39.623071 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.622877 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86f7bbf7c5-9h5nl" event={"ID":"505f1712-2250-43ff-a8ab-270f7cc2b859","Type":"ContainerDied","Data":"14427a49793df1e61bd38b88949e4da45fd5b9b7842d11981117bb8d85f8b9f8"} Jul 13 22:01:39.623071 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.622896 2562 scope.go:117] "RemoveContainer" containerID="ed7b099c38ed08880247b1b1aad6b325f0b34f669f9132a5b478a45671b3e3d8" Jul 13 22:01:39.632403 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.632380 2562 scope.go:117] "RemoveContainer" containerID="ed7b099c38ed08880247b1b1aad6b325f0b34f669f9132a5b478a45671b3e3d8" Jul 13 22:01:39.632740 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:01:39.632713 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed7b099c38ed08880247b1b1aad6b325f0b34f669f9132a5b478a45671b3e3d8\": container with ID starting with ed7b099c38ed08880247b1b1aad6b325f0b34f669f9132a5b478a45671b3e3d8 not found: ID does not exist" containerID="ed7b099c38ed08880247b1b1aad6b325f0b34f669f9132a5b478a45671b3e3d8" Jul 13 22:01:39.632814 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.632746 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed7b099c38ed08880247b1b1aad6b325f0b34f669f9132a5b478a45671b3e3d8"} err="failed to get container status \"ed7b099c38ed08880247b1b1aad6b325f0b34f669f9132a5b478a45671b3e3d8\": rpc error: code = NotFound desc = could not find container \"ed7b099c38ed08880247b1b1aad6b325f0b34f669f9132a5b478a45671b3e3d8\": container with ID starting with ed7b099c38ed08880247b1b1aad6b325f0b34f669f9132a5b478a45671b3e3d8 not found: ID does not exist" Jul 13 22:01:39.645165 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.645133 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-86f7bbf7c5-9h5nl"] Jul 13 22:01:39.653469 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:39.653431 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-86f7bbf7c5-9h5nl"] Jul 13 22:01:40.965740 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:40.965707 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="505f1712-2250-43ff-a8ab-270f7cc2b859" path="/var/lib/kubelet/pods/505f1712-2250-43ff-a8ab-270f7cc2b859/volumes" Jul 13 22:01:47.661731 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:47.661638 2562 generic.go:356] "Generic (PLEG): container finished" podID="3815ceb2-a3b8-43bc-bf04-b1dd4adc4412" containerID="2ec787aa4a14bc7ce4544a7c29cd0481a547438d3c0e347becc165285ebef020" exitCode=0 Jul 13 22:01:47.661731 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:47.661680 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-5vww5" event={"ID":"3815ceb2-a3b8-43bc-bf04-b1dd4adc4412","Type":"ContainerDied","Data":"2ec787aa4a14bc7ce4544a7c29cd0481a547438d3c0e347becc165285ebef020"} Jul 13 22:01:47.662159 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:47.662049 2562 scope.go:117] "RemoveContainer" containerID="2ec787aa4a14bc7ce4544a7c29cd0481a547438d3c0e347becc165285ebef020" Jul 13 22:01:48.669854 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:48.669819 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-5vww5" event={"ID":"3815ceb2-a3b8-43bc-bf04-b1dd4adc4412","Type":"ContainerStarted","Data":"2f0afe7d52d8f6bb34f3101e02e4f52622030769480e27617afbe72996516be7"} Jul 13 22:01:50.657426 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:50.657358 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d5576bfb7-fdtzv" podUID="2774db86-b0f5-44d4-8188-285a38934e4d" containerName="console" containerID="cri-o://384415ea96f798da5c97986e8d7bb298f217f22f747f348b2b715a454187a798" gracePeriod=15 Jul 13 22:01:50.937914 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:50.937845 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d5576bfb7-fdtzv" Jul 13 22:01:51.021091 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:51.021064 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7c8rr\" (UniqueName: \"kubernetes.io/projected/2774db86-b0f5-44d4-8188-285a38934e4d-kube-api-access-7c8rr\") pod \"2774db86-b0f5-44d4-8188-285a38934e4d\" (UID: \"2774db86-b0f5-44d4-8188-285a38934e4d\") " Jul 13 22:01:51.021241 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:51.021103 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2774db86-b0f5-44d4-8188-285a38934e4d-service-ca\") pod \"2774db86-b0f5-44d4-8188-285a38934e4d\" (UID: \"2774db86-b0f5-44d4-8188-285a38934e4d\") " Jul 13 22:01:51.021241 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:51.021134 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2774db86-b0f5-44d4-8188-285a38934e4d-console-oauth-config\") pod \"2774db86-b0f5-44d4-8188-285a38934e4d\" (UID: \"2774db86-b0f5-44d4-8188-285a38934e4d\") " Jul 13 22:01:51.021241 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:51.021175 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2774db86-b0f5-44d4-8188-285a38934e4d-oauth-serving-cert\") pod \"2774db86-b0f5-44d4-8188-285a38934e4d\" (UID: \"2774db86-b0f5-44d4-8188-285a38934e4d\") " Jul 13 22:01:51.021386 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:51.021325 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2774db86-b0f5-44d4-8188-285a38934e4d-console-config\") pod \"2774db86-b0f5-44d4-8188-285a38934e4d\" (UID: \"2774db86-b0f5-44d4-8188-285a38934e4d\") " Jul 13 22:01:51.021438 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:51.021397 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2774db86-b0f5-44d4-8188-285a38934e4d-console-serving-cert\") pod \"2774db86-b0f5-44d4-8188-285a38934e4d\" (UID: \"2774db86-b0f5-44d4-8188-285a38934e4d\") " Jul 13 22:01:51.021604 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:51.021579 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2774db86-b0f5-44d4-8188-285a38934e4d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2774db86-b0f5-44d4-8188-285a38934e4d" (UID: "2774db86-b0f5-44d4-8188-285a38934e4d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:01:51.021726 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:51.021693 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2774db86-b0f5-44d4-8188-285a38934e4d-console-config" (OuterVolumeSpecName: "console-config") pod "2774db86-b0f5-44d4-8188-285a38934e4d" (UID: "2774db86-b0f5-44d4-8188-285a38934e4d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:01:51.021813 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:51.021792 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2774db86-b0f5-44d4-8188-285a38934e4d-service-ca" (OuterVolumeSpecName: "service-ca") pod "2774db86-b0f5-44d4-8188-285a38934e4d" (UID: "2774db86-b0f5-44d4-8188-285a38934e4d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:01:51.023420 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:51.023394 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2774db86-b0f5-44d4-8188-285a38934e4d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2774db86-b0f5-44d4-8188-285a38934e4d" (UID: "2774db86-b0f5-44d4-8188-285a38934e4d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:01:51.023513 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:51.023449 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2774db86-b0f5-44d4-8188-285a38934e4d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2774db86-b0f5-44d4-8188-285a38934e4d" (UID: "2774db86-b0f5-44d4-8188-285a38934e4d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:01:51.023561 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:51.023525 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2774db86-b0f5-44d4-8188-285a38934e4d-kube-api-access-7c8rr" (OuterVolumeSpecName: "kube-api-access-7c8rr") pod "2774db86-b0f5-44d4-8188-285a38934e4d" (UID: "2774db86-b0f5-44d4-8188-285a38934e4d"). InnerVolumeSpecName "kube-api-access-7c8rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:01:51.122755 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:51.122716 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c8rr\" (UniqueName: \"kubernetes.io/projected/2774db86-b0f5-44d4-8188-285a38934e4d-kube-api-access-7c8rr\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:01:51.122755 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:51.122742 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2774db86-b0f5-44d4-8188-285a38934e4d-service-ca\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:01:51.122755 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:51.122752 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2774db86-b0f5-44d4-8188-285a38934e4d-console-oauth-config\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:01:51.122755 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:51.122760 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2774db86-b0f5-44d4-8188-285a38934e4d-oauth-serving-cert\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:01:51.123043 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:51.122768 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2774db86-b0f5-44d4-8188-285a38934e4d-console-config\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:01:51.123043 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:51.122777 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2774db86-b0f5-44d4-8188-285a38934e4d-console-serving-cert\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:01:51.685852 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:51.685811 2562 generic.go:356] "Generic (PLEG): container finished" podID="2774db86-b0f5-44d4-8188-285a38934e4d" containerID="384415ea96f798da5c97986e8d7bb298f217f22f747f348b2b715a454187a798" exitCode=0 Jul 13 22:01:51.686256 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:51.685867 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d5576bfb7-fdtzv" Jul 13 22:01:51.686256 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:51.685922 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d5576bfb7-fdtzv" event={"ID":"2774db86-b0f5-44d4-8188-285a38934e4d","Type":"ContainerDied","Data":"384415ea96f798da5c97986e8d7bb298f217f22f747f348b2b715a454187a798"} Jul 13 22:01:51.686256 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:51.685970 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d5576bfb7-fdtzv" event={"ID":"2774db86-b0f5-44d4-8188-285a38934e4d","Type":"ContainerDied","Data":"df9960904dcbc90c0764ab6b491f713a14b3ec61cf41b90c081a8cfc6e4d611f"} Jul 13 22:01:51.686256 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:51.686015 2562 scope.go:117] "RemoveContainer" containerID="384415ea96f798da5c97986e8d7bb298f217f22f747f348b2b715a454187a798" Jul 13 22:01:51.695664 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:51.695647 2562 scope.go:117] "RemoveContainer" containerID="384415ea96f798da5c97986e8d7bb298f217f22f747f348b2b715a454187a798" Jul 13 22:01:51.695940 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:01:51.695924 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"384415ea96f798da5c97986e8d7bb298f217f22f747f348b2b715a454187a798\": container with ID starting with 384415ea96f798da5c97986e8d7bb298f217f22f747f348b2b715a454187a798 not found: ID does not exist" containerID="384415ea96f798da5c97986e8d7bb298f217f22f747f348b2b715a454187a798" Jul 13 22:01:51.696009 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:51.695948 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"384415ea96f798da5c97986e8d7bb298f217f22f747f348b2b715a454187a798"} err="failed to get container status \"384415ea96f798da5c97986e8d7bb298f217f22f747f348b2b715a454187a798\": rpc error: code = NotFound desc = could not find container \"384415ea96f798da5c97986e8d7bb298f217f22f747f348b2b715a454187a798\": container with ID starting with 384415ea96f798da5c97986e8d7bb298f217f22f747f348b2b715a454187a798 not found: ID does not exist" Jul 13 22:01:51.709666 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:51.709615 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d5576bfb7-fdtzv"] Jul 13 22:01:51.712507 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:51.712484 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d5576bfb7-fdtzv"] Jul 13 22:01:52.965245 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:52.965210 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2774db86-b0f5-44d4-8188-285a38934e4d" path="/var/lib/kubelet/pods/2774db86-b0f5-44d4-8188-285a38934e4d/volumes" Jul 13 22:01:53.702185 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:53.702150 2562 generic.go:356] "Generic (PLEG): container finished" podID="db233478-500b-472c-919f-510266682def" containerID="d23c07c56436a3b3118ebfe1d37c5d5e4b3ccd09b8a5a36e4925139d0d4fbc65" exitCode=0 Jul 13 22:01:53.702390 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:53.702226 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rcbqx" event={"ID":"db233478-500b-472c-919f-510266682def","Type":"ContainerDied","Data":"d23c07c56436a3b3118ebfe1d37c5d5e4b3ccd09b8a5a36e4925139d0d4fbc65"} Jul 13 22:01:53.702630 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:53.702613 2562 scope.go:117] "RemoveContainer" containerID="d23c07c56436a3b3118ebfe1d37c5d5e4b3ccd09b8a5a36e4925139d0d4fbc65" Jul 13 22:01:54.709723 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:54.709682 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rcbqx" event={"ID":"db233478-500b-472c-919f-510266682def","Type":"ContainerStarted","Data":"ef365f6613313f8aa689e20490893aad3a7bec0a6bd21f9b5667e9a7ec02b082"} Jul 13 22:01:57.723605 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:57.723575 2562 generic.go:356] "Generic (PLEG): container finished" podID="f90dbf0b-63cf-41a1-971c-76d47f6d859b" containerID="709669d00b0b5767bcd82ef5b279ccecc0a078d7dfc7d9ba67a72055c310f390" exitCode=0 Jul 13 22:01:57.723960 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:57.723635 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qhgw4" event={"ID":"f90dbf0b-63cf-41a1-971c-76d47f6d859b","Type":"ContainerDied","Data":"709669d00b0b5767bcd82ef5b279ccecc0a078d7dfc7d9ba67a72055c310f390"} Jul 13 22:01:57.724027 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:57.723957 2562 scope.go:117] "RemoveContainer" containerID="709669d00b0b5767bcd82ef5b279ccecc0a078d7dfc7d9ba67a72055c310f390" Jul 13 22:01:58.730868 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:01:58.730833 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qhgw4" event={"ID":"f90dbf0b-63cf-41a1-971c-76d47f6d859b","Type":"ContainerStarted","Data":"bee18a31ebd056331876293edbf1cf8485623a01c604493abef3f45b8d8a9504"} Jul 13 22:02:12.592795 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:02:12.592757 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:02:12.608451 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:02:12.608423 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:02:12.813006 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:02:12.812963 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 13 22:02:47.113028 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:02:47.112992 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b88478dd6-js7fb"] Jul 13 22:03:12.141814 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:12.141749 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-b88478dd6-js7fb" podUID="a8f93cc6-544b-4a28-bc4c-cc5078f824c2" containerName="console" containerID="cri-o://bd28d723289a0606c579194c18f73967b6f7ca99d112e94a286aa7435250720e" gracePeriod=15 Jul 13 22:03:12.397135 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:12.397077 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:03:12.433162 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:12.433132 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-console-oauth-config\") pod \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " Jul 13 22:03:12.433307 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:12.433172 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-oauth-serving-cert\") pod \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " Jul 13 22:03:12.433307 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:12.433199 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-trusted-ca-bundle\") pod \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " Jul 13 22:03:12.433307 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:12.433247 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-console-serving-cert\") pod \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " Jul 13 22:03:12.433307 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:12.433304 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-console-config\") pod \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " Jul 13 22:03:12.433525 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:12.433334 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-service-ca\") pod \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " Jul 13 22:03:12.433525 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:12.433373 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d92rr\" (UniqueName: \"kubernetes.io/projected/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-kube-api-access-d92rr\") pod \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\" (UID: \"a8f93cc6-544b-4a28-bc4c-cc5078f824c2\") " Jul 13 22:03:12.433631 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:12.433536 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a8f93cc6-544b-4a28-bc4c-cc5078f824c2" (UID: "a8f93cc6-544b-4a28-bc4c-cc5078f824c2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:03:12.433631 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:12.433612 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a8f93cc6-544b-4a28-bc4c-cc5078f824c2" (UID: "a8f93cc6-544b-4a28-bc4c-cc5078f824c2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:03:12.433737 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:12.433640 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-oauth-serving-cert\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:03:12.433898 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:12.433871 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-console-config" (OuterVolumeSpecName: "console-config") pod "a8f93cc6-544b-4a28-bc4c-cc5078f824c2" (UID: "a8f93cc6-544b-4a28-bc4c-cc5078f824c2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:03:12.433970 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:12.433908 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-service-ca" (OuterVolumeSpecName: "service-ca") pod "a8f93cc6-544b-4a28-bc4c-cc5078f824c2" (UID: "a8f93cc6-544b-4a28-bc4c-cc5078f824c2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 13 22:03:12.435441 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:12.435412 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a8f93cc6-544b-4a28-bc4c-cc5078f824c2" (UID: "a8f93cc6-544b-4a28-bc4c-cc5078f824c2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:03:12.435552 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:12.435438 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a8f93cc6-544b-4a28-bc4c-cc5078f824c2" (UID: "a8f93cc6-544b-4a28-bc4c-cc5078f824c2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:03:12.435657 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:12.435635 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-kube-api-access-d92rr" (OuterVolumeSpecName: "kube-api-access-d92rr") pod "a8f93cc6-544b-4a28-bc4c-cc5078f824c2" (UID: "a8f93cc6-544b-4a28-bc4c-cc5078f824c2"). InnerVolumeSpecName "kube-api-access-d92rr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:03:12.534881 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:12.534844 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d92rr\" (UniqueName: \"kubernetes.io/projected/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-kube-api-access-d92rr\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:03:12.534881 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:12.534878 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-console-oauth-config\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:03:12.534881 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:12.534888 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-trusted-ca-bundle\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:03:12.535085 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:12.534896 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-console-serving-cert\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:03:12.535085 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:12.534905 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-console-config\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:03:12.535085 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:12.534913 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a8f93cc6-544b-4a28-bc4c-cc5078f824c2-service-ca\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:03:13.058501 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:13.058469 2562 generic.go:356] "Generic (PLEG): container finished" podID="a8f93cc6-544b-4a28-bc4c-cc5078f824c2" containerID="bd28d723289a0606c579194c18f73967b6f7ca99d112e94a286aa7435250720e" exitCode=0 Jul 13 22:03:13.058665 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:13.058515 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b88478dd6-js7fb" Jul 13 22:03:13.058665 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:13.058531 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b88478dd6-js7fb" event={"ID":"a8f93cc6-544b-4a28-bc4c-cc5078f824c2","Type":"ContainerDied","Data":"bd28d723289a0606c579194c18f73967b6f7ca99d112e94a286aa7435250720e"} Jul 13 22:03:13.058665 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:13.058577 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b88478dd6-js7fb" event={"ID":"a8f93cc6-544b-4a28-bc4c-cc5078f824c2","Type":"ContainerDied","Data":"ea86db66d0eb8a98a612976bf19d39f7ef626cec051e875c6782303e4cc33820"} Jul 13 22:03:13.058665 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:13.058599 2562 scope.go:117] "RemoveContainer" containerID="bd28d723289a0606c579194c18f73967b6f7ca99d112e94a286aa7435250720e" Jul 13 22:03:13.067080 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:13.067060 2562 scope.go:117] "RemoveContainer" containerID="bd28d723289a0606c579194c18f73967b6f7ca99d112e94a286aa7435250720e" Jul 13 22:03:13.067330 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:03:13.067307 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd28d723289a0606c579194c18f73967b6f7ca99d112e94a286aa7435250720e\": container with ID starting with bd28d723289a0606c579194c18f73967b6f7ca99d112e94a286aa7435250720e not found: ID does not exist" containerID="bd28d723289a0606c579194c18f73967b6f7ca99d112e94a286aa7435250720e" Jul 13 22:03:13.067395 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:13.067337 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd28d723289a0606c579194c18f73967b6f7ca99d112e94a286aa7435250720e"} err="failed to get container status \"bd28d723289a0606c579194c18f73967b6f7ca99d112e94a286aa7435250720e\": rpc error: code = NotFound desc = could not find container \"bd28d723289a0606c579194c18f73967b6f7ca99d112e94a286aa7435250720e\": container with ID starting with bd28d723289a0606c579194c18f73967b6f7ca99d112e94a286aa7435250720e not found: ID does not exist" Jul 13 22:03:13.075291 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:13.075270 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b88478dd6-js7fb"] Jul 13 22:03:13.080284 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:13.080265 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b88478dd6-js7fb"] Jul 13 22:03:14.965732 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:03:14.965698 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8f93cc6-544b-4a28-bc4c-cc5078f824c2" path="/var/lib/kubelet/pods/a8f93cc6-544b-4a28-bc4c-cc5078f824c2/volumes" Jul 13 22:04:38.849123 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:38.849093 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x5qhq_7b1fa75b-f29d-45a1-af98-fa947c2058b2/console-operator/1.log" Jul 13 22:04:38.851292 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:38.851266 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x5qhq_7b1fa75b-f29d-45a1-af98-fa947c2058b2/console-operator/1.log" Jul 13 22:04:38.857478 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:38.857457 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw4h7_582b8d35-ee74-490e-aad6-e89ac36cb61c/ovn-acl-logging/0.log" Jul 13 22:04:38.859219 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:38.859199 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw4h7_582b8d35-ee74-490e-aad6-e89ac36cb61c/ovn-acl-logging/0.log" Jul 13 22:04:38.861339 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:38.861317 2562 kubelet.go:1648] "Image garbage collection succeeded" Jul 13 22:04:46.055117 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.055082 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67ddb65fc-tsrpc"] Jul 13 22:04:46.057479 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.055400 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2774db86-b0f5-44d4-8188-285a38934e4d" containerName="console" Jul 13 22:04:46.057479 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.055411 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2774db86-b0f5-44d4-8188-285a38934e4d" containerName="console" Jul 13 22:04:46.057479 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.055420 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8f93cc6-544b-4a28-bc4c-cc5078f824c2" containerName="console" Jul 13 22:04:46.057479 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.055425 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8f93cc6-544b-4a28-bc4c-cc5078f824c2" containerName="console" Jul 13 22:04:46.057479 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.055442 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="505f1712-2250-43ff-a8ab-270f7cc2b859" containerName="registry" Jul 13 22:04:46.057479 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.055448 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="505f1712-2250-43ff-a8ab-270f7cc2b859" containerName="registry" Jul 13 22:04:46.057479 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.055494 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8f93cc6-544b-4a28-bc4c-cc5078f824c2" containerName="console" Jul 13 22:04:46.057479 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.055506 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2774db86-b0f5-44d4-8188-285a38934e4d" containerName="console" Jul 13 22:04:46.057479 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.055512 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="505f1712-2250-43ff-a8ab-270f7cc2b859" containerName="registry" Jul 13 22:04:46.058427 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.058411 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-tsrpc" Jul 13 22:04:46.070114 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.070083 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 13 22:04:46.070227 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.070161 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 13 22:04:46.070227 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.070205 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 13 22:04:46.070363 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.070265 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 13 22:04:46.070436 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.070420 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-kwtxt\"" Jul 13 22:04:46.165969 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.165940 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67ddb65fc-tsrpc"] Jul 13 22:04:46.204792 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.204763 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/61805a0f-31ae-4a71-954e-be2de5d6a38b-manifests\") pod \"opendatahub-operator-controller-manager-67ddb65fc-tsrpc\" (UID: \"61805a0f-31ae-4a71-954e-be2de5d6a38b\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-tsrpc" Jul 13 22:04:46.204929 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.204802 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/61805a0f-31ae-4a71-954e-be2de5d6a38b-webhook-cert\") pod \"opendatahub-operator-controller-manager-67ddb65fc-tsrpc\" (UID: \"61805a0f-31ae-4a71-954e-be2de5d6a38b\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-tsrpc" Jul 13 22:04:46.204929 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.204835 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/61805a0f-31ae-4a71-954e-be2de5d6a38b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67ddb65fc-tsrpc\" (UID: \"61805a0f-31ae-4a71-954e-be2de5d6a38b\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-tsrpc" Jul 13 22:04:46.204929 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.204906 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpncp\" (UniqueName: \"kubernetes.io/projected/61805a0f-31ae-4a71-954e-be2de5d6a38b-kube-api-access-qpncp\") pod \"opendatahub-operator-controller-manager-67ddb65fc-tsrpc\" (UID: \"61805a0f-31ae-4a71-954e-be2de5d6a38b\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-tsrpc" Jul 13 22:04:46.205049 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.204932 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61805a0f-31ae-4a71-954e-be2de5d6a38b-tmp\") pod \"opendatahub-operator-controller-manager-67ddb65fc-tsrpc\" (UID: \"61805a0f-31ae-4a71-954e-be2de5d6a38b\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-tsrpc" Jul 13 22:04:46.305656 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.305581 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/61805a0f-31ae-4a71-954e-be2de5d6a38b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67ddb65fc-tsrpc\" (UID: \"61805a0f-31ae-4a71-954e-be2de5d6a38b\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-tsrpc" Jul 13 22:04:46.305656 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.305627 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpncp\" (UniqueName: \"kubernetes.io/projected/61805a0f-31ae-4a71-954e-be2de5d6a38b-kube-api-access-qpncp\") pod \"opendatahub-operator-controller-manager-67ddb65fc-tsrpc\" (UID: \"61805a0f-31ae-4a71-954e-be2de5d6a38b\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-tsrpc" Jul 13 22:04:46.305656 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.305647 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61805a0f-31ae-4a71-954e-be2de5d6a38b-tmp\") pod \"opendatahub-operator-controller-manager-67ddb65fc-tsrpc\" (UID: \"61805a0f-31ae-4a71-954e-be2de5d6a38b\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-tsrpc" Jul 13 22:04:46.305845 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.305689 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/61805a0f-31ae-4a71-954e-be2de5d6a38b-manifests\") pod \"opendatahub-operator-controller-manager-67ddb65fc-tsrpc\" (UID: \"61805a0f-31ae-4a71-954e-be2de5d6a38b\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-tsrpc" Jul 13 22:04:46.305845 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.305713 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/61805a0f-31ae-4a71-954e-be2de5d6a38b-webhook-cert\") pod \"opendatahub-operator-controller-manager-67ddb65fc-tsrpc\" (UID: \"61805a0f-31ae-4a71-954e-be2de5d6a38b\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-tsrpc" Jul 13 22:04:46.306157 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.306129 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/61805a0f-31ae-4a71-954e-be2de5d6a38b-manifests\") pod \"opendatahub-operator-controller-manager-67ddb65fc-tsrpc\" (UID: \"61805a0f-31ae-4a71-954e-be2de5d6a38b\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-tsrpc" Jul 13 22:04:46.308105 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.308072 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61805a0f-31ae-4a71-954e-be2de5d6a38b-tmp\") pod \"opendatahub-operator-controller-manager-67ddb65fc-tsrpc\" (UID: \"61805a0f-31ae-4a71-954e-be2de5d6a38b\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-tsrpc" Jul 13 22:04:46.308207 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.308193 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/61805a0f-31ae-4a71-954e-be2de5d6a38b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67ddb65fc-tsrpc\" (UID: \"61805a0f-31ae-4a71-954e-be2de5d6a38b\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-tsrpc" Jul 13 22:04:46.308291 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.308203 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/61805a0f-31ae-4a71-954e-be2de5d6a38b-webhook-cert\") pod \"opendatahub-operator-controller-manager-67ddb65fc-tsrpc\" (UID: \"61805a0f-31ae-4a71-954e-be2de5d6a38b\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-tsrpc" Jul 13 22:04:46.314282 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.314258 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpncp\" (UniqueName: \"kubernetes.io/projected/61805a0f-31ae-4a71-954e-be2de5d6a38b-kube-api-access-qpncp\") pod \"opendatahub-operator-controller-manager-67ddb65fc-tsrpc\" (UID: \"61805a0f-31ae-4a71-954e-be2de5d6a38b\") " pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-tsrpc" Jul 13 22:04:46.368430 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.368396 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-tsrpc" Jul 13 22:04:46.508956 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.508910 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67ddb65fc-tsrpc"] Jul 13 22:04:46.512073 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:04:46.512034 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61805a0f_31ae_4a71_954e_be2de5d6a38b.slice/crio-3397ae68e3894d706acb944494cdd516d402fba001864475c0367cf4863432a8 WatchSource:0}: Error finding container 3397ae68e3894d706acb944494cdd516d402fba001864475c0367cf4863432a8: Status 404 returned error can't find the container with id 3397ae68e3894d706acb944494cdd516d402fba001864475c0367cf4863432a8 Jul 13 22:04:46.513705 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:46.513688 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 22:04:47.425422 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:47.425366 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-tsrpc" event={"ID":"61805a0f-31ae-4a71-954e-be2de5d6a38b","Type":"ContainerStarted","Data":"3397ae68e3894d706acb944494cdd516d402fba001864475c0367cf4863432a8"} Jul 13 22:04:50.441636 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:50.441597 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-tsrpc" event={"ID":"61805a0f-31ae-4a71-954e-be2de5d6a38b","Type":"ContainerStarted","Data":"f10d71b8a40f99f09567b7d754e1f9febd7ee180898c6d751d6326ebeaedd626"} Jul 13 22:04:51.447088 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:51.447049 2562 generic.go:356] "Generic (PLEG): container finished" podID="61805a0f-31ae-4a71-954e-be2de5d6a38b" containerID="f10d71b8a40f99f09567b7d754e1f9febd7ee180898c6d751d6326ebeaedd626" exitCode=0 Jul 13 22:04:51.447465 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:51.447099 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-tsrpc" event={"ID":"61805a0f-31ae-4a71-954e-be2de5d6a38b","Type":"ContainerDied","Data":"f10d71b8a40f99f09567b7d754e1f9febd7ee180898c6d751d6326ebeaedd626"} Jul 13 22:04:52.454395 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:52.454361 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-tsrpc" event={"ID":"61805a0f-31ae-4a71-954e-be2de5d6a38b","Type":"ContainerStarted","Data":"08f6db5cd8467ca6ab60bd36cb433a7a4530c52fcf185791a30c4da1f6986e58"} Jul 13 22:04:52.454754 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:52.454493 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-tsrpc" Jul 13 22:04:52.477339 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:04:52.477233 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-tsrpc" podStartSLOduration=2.119676546 podStartE2EDuration="6.477215899s" totalImagesPullingTime="3.320627424s" totalInitContainerRuntime="1.036911929s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:04:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:04:52.475433593 +0000 UTC m=+313.925989130" watchObservedRunningTime="2026-07-13 22:04:52.477215899 +0000 UTC m=+313.927771436" Jul 13 22:05:03.461469 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:03.461431 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67ddb65fc-tsrpc" Jul 13 22:05:04.295432 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:04.295395 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-88cc59766-kkpsn"] Jul 13 22:05:04.302650 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:04.302627 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-88cc59766-kkpsn" Jul 13 22:05:04.305384 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:04.305356 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 13 22:05:04.305501 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:04.305426 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 13 22:05:04.305924 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:04.305907 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 13 22:05:04.305924 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:04.305915 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-kv7xp\"" Jul 13 22:05:04.306092 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:04.306075 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 13 22:05:04.306183 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:04.306153 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 13 22:05:04.314583 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:04.314558 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-88cc59766-kkpsn"] Jul 13 22:05:04.361305 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:04.361275 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgv25\" (UniqueName: \"kubernetes.io/projected/d7de19e7-aa27-4e7b-9035-1176e6aedff9-kube-api-access-lgv25\") pod \"lws-controller-manager-88cc59766-kkpsn\" (UID: \"d7de19e7-aa27-4e7b-9035-1176e6aedff9\") " pod="openshift-lws-operator/lws-controller-manager-88cc59766-kkpsn" Jul 13 22:05:04.361305 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:04.361322 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7de19e7-aa27-4e7b-9035-1176e6aedff9-cert\") pod \"lws-controller-manager-88cc59766-kkpsn\" (UID: \"d7de19e7-aa27-4e7b-9035-1176e6aedff9\") " pod="openshift-lws-operator/lws-controller-manager-88cc59766-kkpsn" Jul 13 22:05:04.361563 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:04.361344 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d7de19e7-aa27-4e7b-9035-1176e6aedff9-manager-config\") pod \"lws-controller-manager-88cc59766-kkpsn\" (UID: \"d7de19e7-aa27-4e7b-9035-1176e6aedff9\") " pod="openshift-lws-operator/lws-controller-manager-88cc59766-kkpsn" Jul 13 22:05:04.361563 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:04.361371 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7de19e7-aa27-4e7b-9035-1176e6aedff9-metrics-cert\") pod \"lws-controller-manager-88cc59766-kkpsn\" (UID: \"d7de19e7-aa27-4e7b-9035-1176e6aedff9\") " pod="openshift-lws-operator/lws-controller-manager-88cc59766-kkpsn" Jul 13 22:05:04.462500 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:04.462453 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgv25\" (UniqueName: \"kubernetes.io/projected/d7de19e7-aa27-4e7b-9035-1176e6aedff9-kube-api-access-lgv25\") pod \"lws-controller-manager-88cc59766-kkpsn\" (UID: \"d7de19e7-aa27-4e7b-9035-1176e6aedff9\") " pod="openshift-lws-operator/lws-controller-manager-88cc59766-kkpsn" Jul 13 22:05:04.462957 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:04.462522 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7de19e7-aa27-4e7b-9035-1176e6aedff9-cert\") pod \"lws-controller-manager-88cc59766-kkpsn\" (UID: \"d7de19e7-aa27-4e7b-9035-1176e6aedff9\") " pod="openshift-lws-operator/lws-controller-manager-88cc59766-kkpsn" Jul 13 22:05:04.462957 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:04.462553 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d7de19e7-aa27-4e7b-9035-1176e6aedff9-manager-config\") pod \"lws-controller-manager-88cc59766-kkpsn\" (UID: \"d7de19e7-aa27-4e7b-9035-1176e6aedff9\") " pod="openshift-lws-operator/lws-controller-manager-88cc59766-kkpsn" Jul 13 22:05:04.462957 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:04.462598 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7de19e7-aa27-4e7b-9035-1176e6aedff9-metrics-cert\") pod \"lws-controller-manager-88cc59766-kkpsn\" (UID: \"d7de19e7-aa27-4e7b-9035-1176e6aedff9\") " pod="openshift-lws-operator/lws-controller-manager-88cc59766-kkpsn" Jul 13 22:05:04.463341 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:04.463302 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d7de19e7-aa27-4e7b-9035-1176e6aedff9-manager-config\") pod \"lws-controller-manager-88cc59766-kkpsn\" (UID: \"d7de19e7-aa27-4e7b-9035-1176e6aedff9\") " pod="openshift-lws-operator/lws-controller-manager-88cc59766-kkpsn" Jul 13 22:05:04.465046 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:04.465026 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7de19e7-aa27-4e7b-9035-1176e6aedff9-cert\") pod \"lws-controller-manager-88cc59766-kkpsn\" (UID: \"d7de19e7-aa27-4e7b-9035-1176e6aedff9\") " pod="openshift-lws-operator/lws-controller-manager-88cc59766-kkpsn" Jul 13 22:05:04.465144 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:04.465092 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d7de19e7-aa27-4e7b-9035-1176e6aedff9-metrics-cert\") pod \"lws-controller-manager-88cc59766-kkpsn\" (UID: \"d7de19e7-aa27-4e7b-9035-1176e6aedff9\") " pod="openshift-lws-operator/lws-controller-manager-88cc59766-kkpsn" Jul 13 22:05:04.470679 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:04.470656 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgv25\" (UniqueName: \"kubernetes.io/projected/d7de19e7-aa27-4e7b-9035-1176e6aedff9-kube-api-access-lgv25\") pod \"lws-controller-manager-88cc59766-kkpsn\" (UID: \"d7de19e7-aa27-4e7b-9035-1176e6aedff9\") " pod="openshift-lws-operator/lws-controller-manager-88cc59766-kkpsn" Jul 13 22:05:04.611896 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:04.611814 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-88cc59766-kkpsn" Jul 13 22:05:04.749946 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:04.749921 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-88cc59766-kkpsn"] Jul 13 22:05:04.752481 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:05:04.752449 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7de19e7_aa27_4e7b_9035_1176e6aedff9.slice/crio-b904a93f8947c876daeeba410927d6105536eac8cca3e7f668288aff5ee6a530 WatchSource:0}: Error finding container b904a93f8947c876daeeba410927d6105536eac8cca3e7f668288aff5ee6a530: Status 404 returned error can't find the container with id b904a93f8947c876daeeba410927d6105536eac8cca3e7f668288aff5ee6a530 Jul 13 22:05:05.514043 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:05.514001 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-88cc59766-kkpsn" event={"ID":"d7de19e7-aa27-4e7b-9035-1176e6aedff9","Type":"ContainerStarted","Data":"b904a93f8947c876daeeba410927d6105536eac8cca3e7f668288aff5ee6a530"} Jul 13 22:05:05.738341 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:05.738310 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6bf5d9c687-mx6hg"] Jul 13 22:05:05.741750 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:05.741730 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-mx6hg" Jul 13 22:05:05.770209 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:05.770148 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 13 22:05:05.772481 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:05.772454 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd72b8d2-795c-4eb0-8086-6a685215395a-tls-certs\") pod \"kube-auth-proxy-6bf5d9c687-mx6hg\" (UID: \"fd72b8d2-795c-4eb0-8086-6a685215395a\") " pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-mx6hg" Jul 13 22:05:05.772618 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:05.772548 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd72b8d2-795c-4eb0-8086-6a685215395a-tmp\") pod \"kube-auth-proxy-6bf5d9c687-mx6hg\" (UID: \"fd72b8d2-795c-4eb0-8086-6a685215395a\") " pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-mx6hg" Jul 13 22:05:05.772668 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:05.772616 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqld4\" (UniqueName: \"kubernetes.io/projected/fd72b8d2-795c-4eb0-8086-6a685215395a-kube-api-access-qqld4\") pod \"kube-auth-proxy-6bf5d9c687-mx6hg\" (UID: \"fd72b8d2-795c-4eb0-8086-6a685215395a\") " pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-mx6hg" Jul 13 22:05:05.780477 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:05.780459 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vkctd\"" Jul 13 22:05:05.781294 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:05.781269 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6bf5d9c687-mx6hg"] Jul 13 22:05:05.783185 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:05.783167 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 13 22:05:05.874068 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:05.874029 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd72b8d2-795c-4eb0-8086-6a685215395a-tls-certs\") pod \"kube-auth-proxy-6bf5d9c687-mx6hg\" (UID: \"fd72b8d2-795c-4eb0-8086-6a685215395a\") " pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-mx6hg" Jul 13 22:05:05.874255 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:05.874102 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd72b8d2-795c-4eb0-8086-6a685215395a-tmp\") pod \"kube-auth-proxy-6bf5d9c687-mx6hg\" (UID: \"fd72b8d2-795c-4eb0-8086-6a685215395a\") " pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-mx6hg" Jul 13 22:05:05.874255 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:05.874163 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqld4\" (UniqueName: \"kubernetes.io/projected/fd72b8d2-795c-4eb0-8086-6a685215395a-kube-api-access-qqld4\") pod \"kube-auth-proxy-6bf5d9c687-mx6hg\" (UID: \"fd72b8d2-795c-4eb0-8086-6a685215395a\") " pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-mx6hg" Jul 13 22:05:05.876591 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:05.876563 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd72b8d2-795c-4eb0-8086-6a685215395a-tmp\") pod \"kube-auth-proxy-6bf5d9c687-mx6hg\" (UID: \"fd72b8d2-795c-4eb0-8086-6a685215395a\") " pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-mx6hg" Jul 13 22:05:05.876821 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:05.876797 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd72b8d2-795c-4eb0-8086-6a685215395a-tls-certs\") pod \"kube-auth-proxy-6bf5d9c687-mx6hg\" (UID: \"fd72b8d2-795c-4eb0-8086-6a685215395a\") " pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-mx6hg" Jul 13 22:05:05.912999 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:05.912943 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqld4\" (UniqueName: \"kubernetes.io/projected/fd72b8d2-795c-4eb0-8086-6a685215395a-kube-api-access-qqld4\") pod \"kube-auth-proxy-6bf5d9c687-mx6hg\" (UID: \"fd72b8d2-795c-4eb0-8086-6a685215395a\") " pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-mx6hg" Jul 13 22:05:06.051966 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:06.051881 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-mx6hg" Jul 13 22:05:06.871592 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:05:06.871568 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd72b8d2_795c_4eb0_8086_6a685215395a.slice/crio-3778a39a5cf068f992730d4c2a599a5aace8054bf348f65c5c62c1d5fc93b506 WatchSource:0}: Error finding container 3778a39a5cf068f992730d4c2a599a5aace8054bf348f65c5c62c1d5fc93b506: Status 404 returned error can't find the container with id 3778a39a5cf068f992730d4c2a599a5aace8054bf348f65c5c62c1d5fc93b506 Jul 13 22:05:06.871876 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:06.871600 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6bf5d9c687-mx6hg"] Jul 13 22:05:07.524157 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:07.524114 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-mx6hg" event={"ID":"fd72b8d2-795c-4eb0-8086-6a685215395a","Type":"ContainerStarted","Data":"3778a39a5cf068f992730d4c2a599a5aace8054bf348f65c5c62c1d5fc93b506"} Jul 13 22:05:07.525655 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:07.525633 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-88cc59766-kkpsn" event={"ID":"d7de19e7-aa27-4e7b-9035-1176e6aedff9","Type":"ContainerStarted","Data":"d576360624b007572c594f64cc8f178e96a5a98a877edd0a0986218f94e5582b"} Jul 13 22:05:07.525771 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:07.525759 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-88cc59766-kkpsn" Jul 13 22:05:10.540047 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:10.540015 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-mx6hg" event={"ID":"fd72b8d2-795c-4eb0-8086-6a685215395a","Type":"ContainerStarted","Data":"87c15e49c23f50a74640cbaed86a31e8063a869c6cc55ec7ee1bfe547912b639"} Jul 13 22:05:10.586417 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:10.586368 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-88cc59766-kkpsn" podStartSLOduration=4.550525725 podStartE2EDuration="6.586352329s" totalImagesPullingTime="2.035826604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:05:07.564664346 +0000 UTC m=+329.015219882" watchObservedRunningTime="2026-07-13 22:05:10.586352329 +0000 UTC m=+332.036907866" Jul 13 22:05:10.587159 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:10.587131 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6bf5d9c687-mx6hg" podStartSLOduration=2.887592429 podStartE2EDuration="5.587121085s" totalImagesPullingTime="2.699528656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:05:10.585820263 +0000 UTC m=+332.036375811" watchObservedRunningTime="2026-07-13 22:05:10.587121085 +0000 UTC m=+332.037676621" Jul 13 22:05:18.532528 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:05:18.532490 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-88cc59766-kkpsn" Jul 13 22:06:57.538106 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:06:57.538069 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-5nnfr"] Jul 13 22:06:57.541449 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:06:57.541434 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5nnfr" Jul 13 22:06:57.545694 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:06:57.545673 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 13 22:06:57.545864 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:06:57.545718 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 13 22:06:57.545864 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:06:57.545735 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 13 22:06:57.546346 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:06:57.546327 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xnhp8\"" Jul 13 22:06:57.546454 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:06:57.546371 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 13 22:06:57.553501 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:06:57.553479 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-5nnfr"] Jul 13 22:06:57.643784 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:06:57.643756 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c7f9792-825d-464c-ab40-16590978fd9c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-5nnfr\" (UID: \"2c7f9792-825d-464c-ab40-16590978fd9c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5nnfr" Jul 13 22:06:57.643927 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:06:57.643792 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjdfv\" (UniqueName: \"kubernetes.io/projected/2c7f9792-825d-464c-ab40-16590978fd9c-kube-api-access-vjdfv\") pod \"kuadrant-console-plugin-66b5774985-5nnfr\" (UID: \"2c7f9792-825d-464c-ab40-16590978fd9c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5nnfr" Jul 13 22:06:57.643927 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:06:57.643894 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2c7f9792-825d-464c-ab40-16590978fd9c-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-5nnfr\" (UID: \"2c7f9792-825d-464c-ab40-16590978fd9c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5nnfr" Jul 13 22:06:57.744740 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:06:57.744711 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2c7f9792-825d-464c-ab40-16590978fd9c-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-5nnfr\" (UID: \"2c7f9792-825d-464c-ab40-16590978fd9c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5nnfr" Jul 13 22:06:57.744896 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:06:57.744747 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c7f9792-825d-464c-ab40-16590978fd9c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-5nnfr\" (UID: \"2c7f9792-825d-464c-ab40-16590978fd9c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5nnfr" Jul 13 22:06:57.744896 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:06:57.744770 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjdfv\" (UniqueName: \"kubernetes.io/projected/2c7f9792-825d-464c-ab40-16590978fd9c-kube-api-access-vjdfv\") pod \"kuadrant-console-plugin-66b5774985-5nnfr\" (UID: \"2c7f9792-825d-464c-ab40-16590978fd9c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5nnfr" Jul 13 22:06:57.745450 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:06:57.745422 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2c7f9792-825d-464c-ab40-16590978fd9c-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-5nnfr\" (UID: \"2c7f9792-825d-464c-ab40-16590978fd9c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5nnfr" Jul 13 22:06:57.747063 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:06:57.747045 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2c7f9792-825d-464c-ab40-16590978fd9c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-5nnfr\" (UID: \"2c7f9792-825d-464c-ab40-16590978fd9c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5nnfr" Jul 13 22:06:57.753393 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:06:57.753367 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjdfv\" (UniqueName: \"kubernetes.io/projected/2c7f9792-825d-464c-ab40-16590978fd9c-kube-api-access-vjdfv\") pod \"kuadrant-console-plugin-66b5774985-5nnfr\" (UID: \"2c7f9792-825d-464c-ab40-16590978fd9c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5nnfr" Jul 13 22:06:57.861249 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:06:57.861164 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5nnfr" Jul 13 22:06:57.993486 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:06:57.993462 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-5nnfr"] Jul 13 22:06:57.996742 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:06:57.996714 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c7f9792_825d_464c_ab40_16590978fd9c.slice/crio-84a03736cd4fab6993fa0c7e9c8323390b797f502465f571f7dc3f90edec3fed WatchSource:0}: Error finding container 84a03736cd4fab6993fa0c7e9c8323390b797f502465f571f7dc3f90edec3fed: Status 404 returned error can't find the container with id 84a03736cd4fab6993fa0c7e9c8323390b797f502465f571f7dc3f90edec3fed Jul 13 22:06:58.012382 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:06:58.012353 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5nnfr" event={"ID":"2c7f9792-825d-464c-ab40-16590978fd9c","Type":"ContainerStarted","Data":"84a03736cd4fab6993fa0c7e9c8323390b797f502465f571f7dc3f90edec3fed"} Jul 13 22:07:22.140791 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:22.140757 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5nnfr" event={"ID":"2c7f9792-825d-464c-ab40-16590978fd9c","Type":"ContainerStarted","Data":"603704172f51367b97b2bbd3622dd7a2f5d20dfa7d705ecdd08441a4205269b0"} Jul 13 22:07:40.978202 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:40.978148 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5nnfr" podStartSLOduration=20.821731917 podStartE2EDuration="43.978132334s" totalImagesPullingTime="23.156400417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:06:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:07:22.16275554 +0000 UTC m=+463.613311077" watchObservedRunningTime="2026-07-13 22:07:40.978132334 +0000 UTC m=+482.428687871" Jul 13 22:07:40.978704 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:40.978258 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:07:40.997300 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:40.997272 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:07:40.997432 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:40.997356 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-gpfx4" Jul 13 22:07:40.999577 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:40.999547 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 13 22:07:41.008338 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:41.008317 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:07:41.020122 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:41.020094 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tft2m\" (UniqueName: \"kubernetes.io/projected/41182abb-5070-47d1-9920-52287ac3f65c-kube-api-access-tft2m\") pod \"limitador-limitador-54d6d988d5-gpfx4\" (UID: \"41182abb-5070-47d1-9920-52287ac3f65c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gpfx4" Jul 13 22:07:41.020214 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:41.020129 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/41182abb-5070-47d1-9920-52287ac3f65c-config-file\") pod \"limitador-limitador-54d6d988d5-gpfx4\" (UID: \"41182abb-5070-47d1-9920-52287ac3f65c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gpfx4" Jul 13 22:07:41.121421 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:41.121394 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tft2m\" (UniqueName: \"kubernetes.io/projected/41182abb-5070-47d1-9920-52287ac3f65c-kube-api-access-tft2m\") pod \"limitador-limitador-54d6d988d5-gpfx4\" (UID: \"41182abb-5070-47d1-9920-52287ac3f65c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gpfx4" Jul 13 22:07:41.121550 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:41.121428 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/41182abb-5070-47d1-9920-52287ac3f65c-config-file\") pod \"limitador-limitador-54d6d988d5-gpfx4\" (UID: \"41182abb-5070-47d1-9920-52287ac3f65c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gpfx4" Jul 13 22:07:41.122032 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:41.122014 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/41182abb-5070-47d1-9920-52287ac3f65c-config-file\") pod \"limitador-limitador-54d6d988d5-gpfx4\" (UID: \"41182abb-5070-47d1-9920-52287ac3f65c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gpfx4" Jul 13 22:07:41.130009 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:41.129966 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tft2m\" (UniqueName: \"kubernetes.io/projected/41182abb-5070-47d1-9920-52287ac3f65c-kube-api-access-tft2m\") pod \"limitador-limitador-54d6d988d5-gpfx4\" (UID: \"41182abb-5070-47d1-9920-52287ac3f65c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-gpfx4" Jul 13 22:07:41.309503 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:41.309423 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-gpfx4" Jul 13 22:07:41.451462 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:41.451432 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:07:41.453931 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:07:41.453903 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41182abb_5070_47d1_9920_52287ac3f65c.slice/crio-1205c6a565d41980aae1d4c158f7cca58cec54db88c0ff7ec47954da36545bae WatchSource:0}: Error finding container 1205c6a565d41980aae1d4c158f7cca58cec54db88c0ff7ec47954da36545bae: Status 404 returned error can't find the container with id 1205c6a565d41980aae1d4c158f7cca58cec54db88c0ff7ec47954da36545bae Jul 13 22:07:41.745669 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:41.745635 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-hcb5l"] Jul 13 22:07:41.750318 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:41.750301 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-hcb5l" Jul 13 22:07:41.753789 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:41.753764 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-s2kgp\"" Jul 13 22:07:41.765681 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:41.765659 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-hcb5l"] Jul 13 22:07:41.828319 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:41.828296 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hsbb\" (UniqueName: \"kubernetes.io/projected/ca8ab009-a28f-4f28-8b72-df9958a7c7d9-kube-api-access-5hsbb\") pod \"authorino-75d556bd77-hcb5l\" (UID: \"ca8ab009-a28f-4f28-8b72-df9958a7c7d9\") " pod="kuadrant-system/authorino-75d556bd77-hcb5l" Jul 13 22:07:41.930761 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:41.930011 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hsbb\" (UniqueName: \"kubernetes.io/projected/ca8ab009-a28f-4f28-8b72-df9958a7c7d9-kube-api-access-5hsbb\") pod \"authorino-75d556bd77-hcb5l\" (UID: \"ca8ab009-a28f-4f28-8b72-df9958a7c7d9\") " pod="kuadrant-system/authorino-75d556bd77-hcb5l" Jul 13 22:07:41.944690 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:41.944628 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hsbb\" (UniqueName: \"kubernetes.io/projected/ca8ab009-a28f-4f28-8b72-df9958a7c7d9-kube-api-access-5hsbb\") pod \"authorino-75d556bd77-hcb5l\" (UID: \"ca8ab009-a28f-4f28-8b72-df9958a7c7d9\") " pod="kuadrant-system/authorino-75d556bd77-hcb5l" Jul 13 22:07:42.059939 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:42.059854 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-hcb5l" Jul 13 22:07:42.239579 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:42.239540 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-gpfx4" event={"ID":"41182abb-5070-47d1-9920-52287ac3f65c","Type":"ContainerStarted","Data":"1205c6a565d41980aae1d4c158f7cca58cec54db88c0ff7ec47954da36545bae"} Jul 13 22:07:42.255890 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:42.255861 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-hcb5l"] Jul 13 22:07:42.259651 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:07:42.259603 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca8ab009_a28f_4f28_8b72_df9958a7c7d9.slice/crio-30e0aafdb346c1b8174b04a5769cb505ccefce1cb81c88c7675e4db9edd3e6de WatchSource:0}: Error finding container 30e0aafdb346c1b8174b04a5769cb505ccefce1cb81c88c7675e4db9edd3e6de: Status 404 returned error can't find the container with id 30e0aafdb346c1b8174b04a5769cb505ccefce1cb81c88c7675e4db9edd3e6de Jul 13 22:07:43.247792 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:43.247659 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-hcb5l" event={"ID":"ca8ab009-a28f-4f28-8b72-df9958a7c7d9","Type":"ContainerStarted","Data":"30e0aafdb346c1b8174b04a5769cb505ccefce1cb81c88c7675e4db9edd3e6de"} Jul 13 22:07:48.275272 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:48.275236 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-gpfx4" event={"ID":"41182abb-5070-47d1-9920-52287ac3f65c","Type":"ContainerStarted","Data":"e777d9df8cb8e469c0da3690a670ce29cd330cada816a729ea800fe0016cf6e2"} Jul 13 22:07:48.275698 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:48.275354 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-gpfx4" Jul 13 22:07:48.276857 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:48.276830 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-hcb5l" event={"ID":"ca8ab009-a28f-4f28-8b72-df9958a7c7d9","Type":"ContainerStarted","Data":"7b26f3880d846e89cb93b1611e3a5eb8f57b10050d44a940cc2210e2cbeb7de1"} Jul 13 22:07:48.300452 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:48.300410 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-gpfx4" podStartSLOduration=2.34831523 podStartE2EDuration="8.300397853s" totalImagesPullingTime="5.952082623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:07:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:07:48.295237412 +0000 UTC m=+489.745792971" watchObservedRunningTime="2026-07-13 22:07:48.300397853 +0000 UTC m=+489.750953431" Jul 13 22:07:48.317311 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:48.317269 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-hcb5l" podStartSLOduration=2.158464413 podStartE2EDuration="7.317256206s" totalImagesPullingTime="5.158791793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:07:48.315235044 +0000 UTC m=+489.765790581" watchObservedRunningTime="2026-07-13 22:07:48.317256206 +0000 UTC m=+489.767811742" Jul 13 22:07:59.285560 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:07:59.285524 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-gpfx4" Jul 13 22:08:17.869961 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:08:17.869887 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-hcb5l"] Jul 13 22:08:17.870444 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:08:17.870093 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-hcb5l" podUID="ca8ab009-a28f-4f28-8b72-df9958a7c7d9" containerName="authorino" containerID="cri-o://7b26f3880d846e89cb93b1611e3a5eb8f57b10050d44a940cc2210e2cbeb7de1" gracePeriod=30 Jul 13 22:08:18.121648 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:08:18.121595 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-hcb5l" Jul 13 22:08:18.153101 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:08:18.153076 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hsbb\" (UniqueName: \"kubernetes.io/projected/ca8ab009-a28f-4f28-8b72-df9958a7c7d9-kube-api-access-5hsbb\") pod \"ca8ab009-a28f-4f28-8b72-df9958a7c7d9\" (UID: \"ca8ab009-a28f-4f28-8b72-df9958a7c7d9\") " Jul 13 22:08:18.155703 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:08:18.155676 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca8ab009-a28f-4f28-8b72-df9958a7c7d9-kube-api-access-5hsbb" (OuterVolumeSpecName: "kube-api-access-5hsbb") pod "ca8ab009-a28f-4f28-8b72-df9958a7c7d9" (UID: "ca8ab009-a28f-4f28-8b72-df9958a7c7d9"). InnerVolumeSpecName "kube-api-access-5hsbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:08:18.254545 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:08:18.254514 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hsbb\" (UniqueName: \"kubernetes.io/projected/ca8ab009-a28f-4f28-8b72-df9958a7c7d9-kube-api-access-5hsbb\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:08:18.415761 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:08:18.415676 2562 generic.go:356] "Generic (PLEG): container finished" podID="ca8ab009-a28f-4f28-8b72-df9958a7c7d9" containerID="7b26f3880d846e89cb93b1611e3a5eb8f57b10050d44a940cc2210e2cbeb7de1" exitCode=0 Jul 13 22:08:18.415761 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:08:18.415719 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-hcb5l" Jul 13 22:08:18.416019 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:08:18.415762 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-hcb5l" event={"ID":"ca8ab009-a28f-4f28-8b72-df9958a7c7d9","Type":"ContainerDied","Data":"7b26f3880d846e89cb93b1611e3a5eb8f57b10050d44a940cc2210e2cbeb7de1"} Jul 13 22:08:18.416019 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:08:18.415802 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-hcb5l" event={"ID":"ca8ab009-a28f-4f28-8b72-df9958a7c7d9","Type":"ContainerDied","Data":"30e0aafdb346c1b8174b04a5769cb505ccefce1cb81c88c7675e4db9edd3e6de"} Jul 13 22:08:18.416019 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:08:18.415822 2562 scope.go:117] "RemoveContainer" containerID="7b26f3880d846e89cb93b1611e3a5eb8f57b10050d44a940cc2210e2cbeb7de1" Jul 13 22:08:18.424280 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:08:18.424263 2562 scope.go:117] "RemoveContainer" containerID="7b26f3880d846e89cb93b1611e3a5eb8f57b10050d44a940cc2210e2cbeb7de1" Jul 13 22:08:18.424542 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:08:18.424524 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b26f3880d846e89cb93b1611e3a5eb8f57b10050d44a940cc2210e2cbeb7de1\": container with ID starting with 7b26f3880d846e89cb93b1611e3a5eb8f57b10050d44a940cc2210e2cbeb7de1 not found: ID does not exist" containerID="7b26f3880d846e89cb93b1611e3a5eb8f57b10050d44a940cc2210e2cbeb7de1" Jul 13 22:08:18.424610 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:08:18.424554 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b26f3880d846e89cb93b1611e3a5eb8f57b10050d44a940cc2210e2cbeb7de1"} err="failed to get container status \"7b26f3880d846e89cb93b1611e3a5eb8f57b10050d44a940cc2210e2cbeb7de1\": rpc error: code = NotFound desc = could not find container \"7b26f3880d846e89cb93b1611e3a5eb8f57b10050d44a940cc2210e2cbeb7de1\": container with ID starting with 7b26f3880d846e89cb93b1611e3a5eb8f57b10050d44a940cc2210e2cbeb7de1 not found: ID does not exist" Jul 13 22:08:18.457318 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:08:18.457292 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-hcb5l"] Jul 13 22:08:18.464597 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:08:18.464572 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-hcb5l"] Jul 13 22:08:18.971808 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:08:18.971772 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca8ab009-a28f-4f28-8b72-df9958a7c7d9" path="/var/lib/kubelet/pods/ca8ab009-a28f-4f28-8b72-df9958a7c7d9/volumes" Jul 13 22:08:59.244423 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:08:59.244388 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:09:13.869327 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:13.869292 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:09:19.466734 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.466695 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l"] Jul 13 22:09:19.467183 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.467066 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca8ab009-a28f-4f28-8b72-df9958a7c7d9" containerName="authorino" Jul 13 22:09:19.467183 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.467078 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca8ab009-a28f-4f28-8b72-df9958a7c7d9" containerName="authorino" Jul 13 22:09:19.467183 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.467128 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca8ab009-a28f-4f28-8b72-df9958a7c7d9" containerName="authorino" Jul 13 22:09:19.470250 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.470234 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l" Jul 13 22:09:19.472715 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.472692 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 13 22:09:19.472849 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.472727 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 13 22:09:19.473021 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.473005 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-gfrsn\"" Jul 13 22:09:19.473408 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.473390 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 13 22:09:19.482193 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.482169 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l"] Jul 13 22:09:19.580622 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.580592 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35a2b15a-247b-4d8c-9114-49cd5a7f9174-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2fq6l\" (UID: \"35a2b15a-247b-4d8c-9114-49cd5a7f9174\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l" Jul 13 22:09:19.580813 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.580703 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35a2b15a-247b-4d8c-9114-49cd5a7f9174-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2fq6l\" (UID: \"35a2b15a-247b-4d8c-9114-49cd5a7f9174\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l" Jul 13 22:09:19.580813 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.580732 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35a2b15a-247b-4d8c-9114-49cd5a7f9174-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2fq6l\" (UID: \"35a2b15a-247b-4d8c-9114-49cd5a7f9174\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l" Jul 13 22:09:19.580813 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.580761 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68bm5\" (UniqueName: \"kubernetes.io/projected/35a2b15a-247b-4d8c-9114-49cd5a7f9174-kube-api-access-68bm5\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2fq6l\" (UID: \"35a2b15a-247b-4d8c-9114-49cd5a7f9174\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l" Jul 13 22:09:19.580813 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.580782 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35a2b15a-247b-4d8c-9114-49cd5a7f9174-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2fq6l\" (UID: \"35a2b15a-247b-4d8c-9114-49cd5a7f9174\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l" Jul 13 22:09:19.580813 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.580796 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a2b15a-247b-4d8c-9114-49cd5a7f9174-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2fq6l\" (UID: \"35a2b15a-247b-4d8c-9114-49cd5a7f9174\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l" Jul 13 22:09:19.681815 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.681768 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35a2b15a-247b-4d8c-9114-49cd5a7f9174-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2fq6l\" (UID: \"35a2b15a-247b-4d8c-9114-49cd5a7f9174\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l" Jul 13 22:09:19.681815 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.681813 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35a2b15a-247b-4d8c-9114-49cd5a7f9174-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2fq6l\" (UID: \"35a2b15a-247b-4d8c-9114-49cd5a7f9174\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l" Jul 13 22:09:19.682076 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.681841 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68bm5\" (UniqueName: \"kubernetes.io/projected/35a2b15a-247b-4d8c-9114-49cd5a7f9174-kube-api-access-68bm5\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2fq6l\" (UID: \"35a2b15a-247b-4d8c-9114-49cd5a7f9174\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l" Jul 13 22:09:19.682076 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.681966 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35a2b15a-247b-4d8c-9114-49cd5a7f9174-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2fq6l\" (UID: \"35a2b15a-247b-4d8c-9114-49cd5a7f9174\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l" Jul 13 22:09:19.682076 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.682019 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a2b15a-247b-4d8c-9114-49cd5a7f9174-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2fq6l\" (UID: \"35a2b15a-247b-4d8c-9114-49cd5a7f9174\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l" Jul 13 22:09:19.682076 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.682048 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35a2b15a-247b-4d8c-9114-49cd5a7f9174-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2fq6l\" (UID: \"35a2b15a-247b-4d8c-9114-49cd5a7f9174\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l" Jul 13 22:09:19.682300 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.682214 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35a2b15a-247b-4d8c-9114-49cd5a7f9174-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2fq6l\" (UID: \"35a2b15a-247b-4d8c-9114-49cd5a7f9174\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l" Jul 13 22:09:19.682300 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.682249 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35a2b15a-247b-4d8c-9114-49cd5a7f9174-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2fq6l\" (UID: \"35a2b15a-247b-4d8c-9114-49cd5a7f9174\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l" Jul 13 22:09:19.682389 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.682366 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35a2b15a-247b-4d8c-9114-49cd5a7f9174-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2fq6l\" (UID: \"35a2b15a-247b-4d8c-9114-49cd5a7f9174\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l" Jul 13 22:09:19.684119 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.684099 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35a2b15a-247b-4d8c-9114-49cd5a7f9174-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2fq6l\" (UID: \"35a2b15a-247b-4d8c-9114-49cd5a7f9174\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l" Jul 13 22:09:19.684487 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.684465 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a2b15a-247b-4d8c-9114-49cd5a7f9174-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2fq6l\" (UID: \"35a2b15a-247b-4d8c-9114-49cd5a7f9174\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l" Jul 13 22:09:19.689931 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.689907 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68bm5\" (UniqueName: \"kubernetes.io/projected/35a2b15a-247b-4d8c-9114-49cd5a7f9174-kube-api-access-68bm5\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2fq6l\" (UID: \"35a2b15a-247b-4d8c-9114-49cd5a7f9174\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l" Jul 13 22:09:19.780415 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.780331 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l" Jul 13 22:09:19.906327 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:19.906296 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l"] Jul 13 22:09:19.908750 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:09:19.908724 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a2b15a_247b_4d8c_9114_49cd5a7f9174.slice/crio-329d188b1da320dadf07794d7c0f08c8bb87e5d3bba699473fcac4bf8ac8dab0 WatchSource:0}: Error finding container 329d188b1da320dadf07794d7c0f08c8bb87e5d3bba699473fcac4bf8ac8dab0: Status 404 returned error can't find the container with id 329d188b1da320dadf07794d7c0f08c8bb87e5d3bba699473fcac4bf8ac8dab0 Jul 13 22:09:20.245738 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:20.245705 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:09:20.711048 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:20.711013 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l" event={"ID":"35a2b15a-247b-4d8c-9114-49cd5a7f9174","Type":"ContainerStarted","Data":"329d188b1da320dadf07794d7c0f08c8bb87e5d3bba699473fcac4bf8ac8dab0"} Jul 13 22:09:21.716826 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:21.716781 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l" event={"ID":"35a2b15a-247b-4d8c-9114-49cd5a7f9174","Type":"ContainerStarted","Data":"42b093d911e9a067010f4c993c042326671b994ddd365f563ccb4b880bfeb4ff"} Jul 13 22:09:21.736343 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:21.736292 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l" podStartSLOduration=1.715299729 podStartE2EDuration="2.736274862s" totalImagesPullingTime="1.020975133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:09:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:09:21.735820223 +0000 UTC m=+583.186375762" watchObservedRunningTime="2026-07-13 22:09:21.736274862 +0000 UTC m=+583.186830410" Jul 13 22:09:29.780560 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:29.780522 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l" Jul 13 22:09:29.781090 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:29.780579 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l" Jul 13 22:09:29.792844 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:29.792820 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l" Jul 13 22:09:30.766197 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:30.766170 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2fq6l" Jul 13 22:09:36.947096 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:36.947059 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:09:38.880484 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:38.880459 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x5qhq_7b1fa75b-f29d-45a1-af98-fa947c2058b2/console-operator/1.log" Jul 13 22:09:38.880917 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:38.880517 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x5qhq_7b1fa75b-f29d-45a1-af98-fa947c2058b2/console-operator/1.log" Jul 13 22:09:38.887628 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:38.887610 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw4h7_582b8d35-ee74-490e-aad6-e89ac36cb61c/ovn-acl-logging/0.log" Jul 13 22:09:38.887732 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:38.887691 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw4h7_582b8d35-ee74-490e-aad6-e89ac36cb61c/ovn-acl-logging/0.log" Jul 13 22:09:46.043377 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:46.043330 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:09:57.661895 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:57.661864 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh"] Jul 13 22:09:57.667782 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:57.667767 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh" Jul 13 22:09:57.669992 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:57.669954 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 13 22:09:57.676100 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:57.676076 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh"] Jul 13 22:09:57.719148 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:57.719121 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0caf-6078-4867-9d51-46cfd78b72a1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm8fh\" (UID: \"ca9b0caf-6078-4867-9d51-46cfd78b72a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh" Jul 13 22:09:57.719247 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:57.719149 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcr7q\" (UniqueName: \"kubernetes.io/projected/ca9b0caf-6078-4867-9d51-46cfd78b72a1-kube-api-access-pcr7q\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm8fh\" (UID: \"ca9b0caf-6078-4867-9d51-46cfd78b72a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh" Jul 13 22:09:57.719247 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:57.719196 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0caf-6078-4867-9d51-46cfd78b72a1-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm8fh\" (UID: \"ca9b0caf-6078-4867-9d51-46cfd78b72a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh" Jul 13 22:09:57.719247 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:57.719216 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0caf-6078-4867-9d51-46cfd78b72a1-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm8fh\" (UID: \"ca9b0caf-6078-4867-9d51-46cfd78b72a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh" Jul 13 22:09:57.719247 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:57.719235 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0caf-6078-4867-9d51-46cfd78b72a1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm8fh\" (UID: \"ca9b0caf-6078-4867-9d51-46cfd78b72a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh" Jul 13 22:09:57.719391 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:57.719257 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca9b0caf-6078-4867-9d51-46cfd78b72a1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm8fh\" (UID: \"ca9b0caf-6078-4867-9d51-46cfd78b72a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh" Jul 13 22:09:57.820714 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:57.820676 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0caf-6078-4867-9d51-46cfd78b72a1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm8fh\" (UID: \"ca9b0caf-6078-4867-9d51-46cfd78b72a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh" Jul 13 22:09:57.820714 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:57.820715 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcr7q\" (UniqueName: \"kubernetes.io/projected/ca9b0caf-6078-4867-9d51-46cfd78b72a1-kube-api-access-pcr7q\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm8fh\" (UID: \"ca9b0caf-6078-4867-9d51-46cfd78b72a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh" Jul 13 22:09:57.820869 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:57.820836 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0caf-6078-4867-9d51-46cfd78b72a1-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm8fh\" (UID: \"ca9b0caf-6078-4867-9d51-46cfd78b72a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh" Jul 13 22:09:57.820921 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:57.820880 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0caf-6078-4867-9d51-46cfd78b72a1-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm8fh\" (UID: \"ca9b0caf-6078-4867-9d51-46cfd78b72a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh" Jul 13 22:09:57.821003 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:57.820919 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0caf-6078-4867-9d51-46cfd78b72a1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm8fh\" (UID: \"ca9b0caf-6078-4867-9d51-46cfd78b72a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh" Jul 13 22:09:57.821003 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:57.820960 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca9b0caf-6078-4867-9d51-46cfd78b72a1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm8fh\" (UID: \"ca9b0caf-6078-4867-9d51-46cfd78b72a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh" Jul 13 22:09:57.821114 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:57.821097 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0caf-6078-4867-9d51-46cfd78b72a1-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm8fh\" (UID: \"ca9b0caf-6078-4867-9d51-46cfd78b72a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh" Jul 13 22:09:57.821272 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:57.821252 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0caf-6078-4867-9d51-46cfd78b72a1-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm8fh\" (UID: \"ca9b0caf-6078-4867-9d51-46cfd78b72a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh" Jul 13 22:09:57.821321 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:57.821308 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0caf-6078-4867-9d51-46cfd78b72a1-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm8fh\" (UID: \"ca9b0caf-6078-4867-9d51-46cfd78b72a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh" Jul 13 22:09:57.822876 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:57.822850 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0caf-6078-4867-9d51-46cfd78b72a1-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm8fh\" (UID: \"ca9b0caf-6078-4867-9d51-46cfd78b72a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh" Jul 13 22:09:57.823285 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:57.823267 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca9b0caf-6078-4867-9d51-46cfd78b72a1-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm8fh\" (UID: \"ca9b0caf-6078-4867-9d51-46cfd78b72a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh" Jul 13 22:09:57.828566 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:57.828548 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcr7q\" (UniqueName: \"kubernetes.io/projected/ca9b0caf-6078-4867-9d51-46cfd78b72a1-kube-api-access-pcr7q\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zm8fh\" (UID: \"ca9b0caf-6078-4867-9d51-46cfd78b72a1\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh" Jul 13 22:09:57.978016 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:57.977989 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh" Jul 13 22:09:58.111433 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:58.111409 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh"] Jul 13 22:09:58.113847 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:09:58.113824 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9b0caf_6078_4867_9d51_46cfd78b72a1.slice/crio-249244dcaae14601b67ab03d21eb26af3dd81a5f92faa55af3a06de70ece013f WatchSource:0}: Error finding container 249244dcaae14601b67ab03d21eb26af3dd81a5f92faa55af3a06de70ece013f: Status 404 returned error can't find the container with id 249244dcaae14601b67ab03d21eb26af3dd81a5f92faa55af3a06de70ece013f Jul 13 22:09:58.115272 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:58.115248 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 13 22:09:58.883677 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:58.883648 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh" event={"ID":"ca9b0caf-6078-4867-9d51-46cfd78b72a1","Type":"ContainerStarted","Data":"507a7f8482c6a3ecf4451ec3e561fa5ed1c3a8959005654f6f09db794be953a9"} Jul 13 22:09:58.884086 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:58.883697 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh" event={"ID":"ca9b0caf-6078-4867-9d51-46cfd78b72a1","Type":"ContainerStarted","Data":"249244dcaae14601b67ab03d21eb26af3dd81a5f92faa55af3a06de70ece013f"} Jul 13 22:09:58.905227 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:58.905173 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh" podStartSLOduration=1.7194150700000002 podStartE2EDuration="1.905157815s" totalImagesPullingTime="185.742745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:09:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:09:58.903408654 +0000 UTC m=+620.353964196" watchObservedRunningTime="2026-07-13 22:09:58.905157815 +0000 UTC m=+620.355713386" Jul 13 22:09:59.856012 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:09:59.855963 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:10:07.978892 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:07.978806 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh" Jul 13 22:10:07.978892 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:07.978866 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh" Jul 13 22:10:07.991312 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:07.991291 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh" Jul 13 22:10:08.941673 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:08.941639 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zm8fh" Jul 13 22:10:16.654913 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:16.654876 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24"] Jul 13 22:10:16.658204 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:16.658188 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24" Jul 13 22:10:16.660327 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:16.660305 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 13 22:10:16.666212 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:16.666188 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24"] Jul 13 22:10:16.801958 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:16.801926 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e7ab012-158d-4cc4-a435-8a58c5769268-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24\" (UID: \"2e7ab012-158d-4cc4-a435-8a58c5769268\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24" Jul 13 22:10:16.802186 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:16.801971 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e7ab012-158d-4cc4-a435-8a58c5769268-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24\" (UID: \"2e7ab012-158d-4cc4-a435-8a58c5769268\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24" Jul 13 22:10:16.802186 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:16.802078 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e7ab012-158d-4cc4-a435-8a58c5769268-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24\" (UID: \"2e7ab012-158d-4cc4-a435-8a58c5769268\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24" Jul 13 22:10:16.802186 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:16.802128 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e7ab012-158d-4cc4-a435-8a58c5769268-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24\" (UID: \"2e7ab012-158d-4cc4-a435-8a58c5769268\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24" Jul 13 22:10:16.802186 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:16.802164 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e7ab012-158d-4cc4-a435-8a58c5769268-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24\" (UID: \"2e7ab012-158d-4cc4-a435-8a58c5769268\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24" Jul 13 22:10:16.802359 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:16.802222 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsx5m\" (UniqueName: \"kubernetes.io/projected/2e7ab012-158d-4cc4-a435-8a58c5769268-kube-api-access-rsx5m\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24\" (UID: \"2e7ab012-158d-4cc4-a435-8a58c5769268\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24" Jul 13 22:10:16.903601 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:16.903565 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e7ab012-158d-4cc4-a435-8a58c5769268-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24\" (UID: \"2e7ab012-158d-4cc4-a435-8a58c5769268\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24" Jul 13 22:10:16.903855 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:16.903621 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e7ab012-158d-4cc4-a435-8a58c5769268-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24\" (UID: \"2e7ab012-158d-4cc4-a435-8a58c5769268\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24" Jul 13 22:10:16.903855 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:16.903667 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e7ab012-158d-4cc4-a435-8a58c5769268-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24\" (UID: \"2e7ab012-158d-4cc4-a435-8a58c5769268\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24" Jul 13 22:10:16.903855 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:16.903705 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e7ab012-158d-4cc4-a435-8a58c5769268-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24\" (UID: \"2e7ab012-158d-4cc4-a435-8a58c5769268\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24" Jul 13 22:10:16.903855 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:16.903825 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e7ab012-158d-4cc4-a435-8a58c5769268-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24\" (UID: \"2e7ab012-158d-4cc4-a435-8a58c5769268\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24" Jul 13 22:10:16.904092 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:16.903916 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsx5m\" (UniqueName: \"kubernetes.io/projected/2e7ab012-158d-4cc4-a435-8a58c5769268-kube-api-access-rsx5m\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24\" (UID: \"2e7ab012-158d-4cc4-a435-8a58c5769268\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24" Jul 13 22:10:16.904092 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:16.904074 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e7ab012-158d-4cc4-a435-8a58c5769268-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24\" (UID: \"2e7ab012-158d-4cc4-a435-8a58c5769268\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24" Jul 13 22:10:16.904159 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:16.904074 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e7ab012-158d-4cc4-a435-8a58c5769268-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24\" (UID: \"2e7ab012-158d-4cc4-a435-8a58c5769268\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24" Jul 13 22:10:16.904159 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:16.904138 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e7ab012-158d-4cc4-a435-8a58c5769268-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24\" (UID: \"2e7ab012-158d-4cc4-a435-8a58c5769268\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24" Jul 13 22:10:16.906082 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:16.906026 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e7ab012-158d-4cc4-a435-8a58c5769268-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24\" (UID: \"2e7ab012-158d-4cc4-a435-8a58c5769268\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24" Jul 13 22:10:16.906184 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:16.906121 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e7ab012-158d-4cc4-a435-8a58c5769268-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24\" (UID: \"2e7ab012-158d-4cc4-a435-8a58c5769268\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24" Jul 13 22:10:16.912104 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:16.912084 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsx5m\" (UniqueName: \"kubernetes.io/projected/2e7ab012-158d-4cc4-a435-8a58c5769268-kube-api-access-rsx5m\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24\" (UID: \"2e7ab012-158d-4cc4-a435-8a58c5769268\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24" Jul 13 22:10:16.969519 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:16.969488 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24" Jul 13 22:10:17.118569 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:17.118544 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24"] Jul 13 22:10:17.120245 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:10:17.120216 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e7ab012_158d_4cc4_a435_8a58c5769268.slice/crio-72e2ada8b2c687a009f32f024551378831b9be0688a8201824f0fa683279b3ec WatchSource:0}: Error finding container 72e2ada8b2c687a009f32f024551378831b9be0688a8201824f0fa683279b3ec: Status 404 returned error can't find the container with id 72e2ada8b2c687a009f32f024551378831b9be0688a8201824f0fa683279b3ec Jul 13 22:10:17.979436 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:17.979403 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24" event={"ID":"2e7ab012-158d-4cc4-a435-8a58c5769268","Type":"ContainerStarted","Data":"a70b074799a41bb8df997f7cb77476f7fe679ce60a3ddccc0690aaf93468adbb"} Jul 13 22:10:17.979436 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:17.979446 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24" event={"ID":"2e7ab012-158d-4cc4-a435-8a58c5769268","Type":"ContainerStarted","Data":"72e2ada8b2c687a009f32f024551378831b9be0688a8201824f0fa683279b3ec"} Jul 13 22:10:17.997036 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:17.996959 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24" podStartSLOduration=1.789908657 podStartE2EDuration="1.996939387s" totalImagesPullingTime="207.03073ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:10:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:10:17.996550657 +0000 UTC m=+639.447106194" watchObservedRunningTime="2026-07-13 22:10:17.996939387 +0000 UTC m=+639.447494925" Jul 13 22:10:19.072559 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:19.072526 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:10:26.970286 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:26.970253 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24" Jul 13 22:10:26.970767 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:26.970296 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24" Jul 13 22:10:26.982550 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:26.982522 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24" Jul 13 22:10:27.030274 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:10:27.030251 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24" Jul 13 22:11:44.356608 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:11:44.356529 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:11:53.759720 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:11:53.759677 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:12:03.763048 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:12:03.763012 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:12:12.856593 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:12:12.856556 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:12:24.052014 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:12:24.051964 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:13:30.557222 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:13:30.557184 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:13:41.441966 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:13:41.441925 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:13:50.152714 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:13:50.152678 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:13:58.657996 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:13:58.657945 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:14:09.166099 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:14:09.166064 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:14:18.452287 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:14:18.452251 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:14:29.253536 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:14:29.253502 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:14:38.763055 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:14:38.762963 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:14:38.908259 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:14:38.908234 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x5qhq_7b1fa75b-f29d-45a1-af98-fa947c2058b2/console-operator/1.log" Jul 13 22:14:38.910252 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:14:38.910230 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x5qhq_7b1fa75b-f29d-45a1-af98-fa947c2058b2/console-operator/1.log" Jul 13 22:14:38.915281 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:14:38.915263 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw4h7_582b8d35-ee74-490e-aad6-e89ac36cb61c/ovn-acl-logging/0.log" Jul 13 22:14:38.917223 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:14:38.917207 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw4h7_582b8d35-ee74-490e-aad6-e89ac36cb61c/ovn-acl-logging/0.log" Jul 13 22:15:00.746732 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:15:00.746698 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:15:11.364048 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:15:11.364011 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:15:35.960335 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:15:35.960295 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:15:49.453437 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:15:49.453401 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:16:01.350324 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:16:01.350233 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:16:12.450050 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:16:12.450014 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:16:22.459936 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:16:22.459904 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:16:34.878570 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:16:34.878534 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:18:26.665047 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:18:26.665007 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:18:39.759928 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:18:39.759893 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:18:48.358030 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:18:48.357991 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:18:59.160046 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:18:59.160010 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:19:10.858779 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:19:10.858695 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:19:20.346353 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:19:20.346318 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:19:30.449421 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:19:30.449387 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:19:38.940389 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:19:38.940359 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x5qhq_7b1fa75b-f29d-45a1-af98-fa947c2058b2/console-operator/1.log" Jul 13 22:19:38.944951 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:19:38.944931 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x5qhq_7b1fa75b-f29d-45a1-af98-fa947c2058b2/console-operator/1.log" Jul 13 22:19:38.948045 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:19:38.948025 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw4h7_582b8d35-ee74-490e-aad6-e89ac36cb61c/ovn-acl-logging/0.log" Jul 13 22:19:38.952209 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:19:38.952191 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw4h7_582b8d35-ee74-490e-aad6-e89ac36cb61c/ovn-acl-logging/0.log" Jul 13 22:19:40.044572 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:19:40.044535 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:19:52.448350 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:19:52.448318 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:20:01.050168 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:20:01.050131 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:20:13.445722 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:20:13.445685 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:20:35.547499 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:20:35.547414 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:20:44.452902 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:20:44.452863 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:20:54.046692 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:20:54.046648 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:21:06.554535 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:21:06.554498 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:21:22.152372 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:21:22.152335 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:21:31.846057 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:21:31.846022 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:21:41.351065 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:21:41.351026 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:21:49.154927 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:21:49.154894 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:21:58.801070 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:21:58.801036 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:22:08.659084 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:22:08.654195 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:22:22.554163 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:22:22.554127 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:22:32.448150 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:22:32.448115 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:22:42.261111 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:22:42.261076 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:22:53.365126 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:22:53.365082 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:23:04.707821 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:23:04.707789 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:23:15.371349 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:23:15.371316 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:23:26.851455 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:23:26.851422 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:23:39.483999 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:23:39.483885 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:23:49.260102 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:23:49.260072 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:24:03.586925 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:24:03.586892 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:24:12.352603 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:24:12.352563 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:24:22.504279 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:24:22.504245 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:24:35.458821 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:24:35.458754 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:24:38.980540 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:24:38.980513 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x5qhq_7b1fa75b-f29d-45a1-af98-fa947c2058b2/console-operator/1.log" Jul 13 22:24:38.985522 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:24:38.985498 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x5qhq_7b1fa75b-f29d-45a1-af98-fa947c2058b2/console-operator/1.log" Jul 13 22:24:38.987840 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:24:38.987809 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw4h7_582b8d35-ee74-490e-aad6-e89ac36cb61c/ovn-acl-logging/0.log" Jul 13 22:24:38.992937 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:24:38.992921 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw4h7_582b8d35-ee74-490e-aad6-e89ac36cb61c/ovn-acl-logging/0.log" Jul 13 22:25:33.058945 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:25:33.058908 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:25:51.957380 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:25:51.957341 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:26:53.856515 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:26:53.856481 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:27:06.146010 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:27:06.145953 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:28:07.253843 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:28:07.253768 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:28:30.153533 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:28:30.153493 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:29:39.009299 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:29:39.009268 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x5qhq_7b1fa75b-f29d-45a1-af98-fa947c2058b2/console-operator/1.log" Jul 13 22:29:39.019493 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:29:39.019461 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x5qhq_7b1fa75b-f29d-45a1-af98-fa947c2058b2/console-operator/1.log" Jul 13 22:29:39.019678 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:29:39.019652 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw4h7_582b8d35-ee74-490e-aad6-e89ac36cb61c/ovn-acl-logging/0.log" Jul 13 22:29:39.026428 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:29:39.026405 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw4h7_582b8d35-ee74-490e-aad6-e89ac36cb61c/ovn-acl-logging/0.log" Jul 13 22:29:39.356438 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:29:39.356354 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:31:15.468556 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:31:15.468471 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:31:24.748347 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:31:24.748310 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:31:35.861593 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:31:35.861553 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:31:56.142650 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:31:56.142617 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:32:34.744040 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:34.743945 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:32:58.254805 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:58.254767 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w"] Jul 13 22:32:58.258386 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:58.258369 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" Jul 13 22:32:58.260582 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:58.260560 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-86e22e9e\"/\"kube-root-ca.crt\"" Jul 13 22:32:58.260683 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:58.260642 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-86e22e9e\"/\"default-dockercfg-b7l4k\"" Jul 13 22:32:58.261240 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:58.261223 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-86e22e9e\"/\"e2e-lifecycle-model-86e22e9e-kserve-self-signed-certs\"" Jul 13 22:32:58.261299 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:58.261239 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-86e22e9e\"/\"openshift-service-ca.crt\"" Jul 13 22:32:58.268759 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:58.268737 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w"] Jul 13 22:32:58.324047 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:58.324015 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e439440-278c-4f4e-b7b1-f431c56f968b-dshm\") pod \"e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w\" (UID: \"9e439440-278c-4f4e-b7b1-f431c56f968b\") " pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" Jul 13 22:32:58.324184 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:58.324050 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e439440-278c-4f4e-b7b1-f431c56f968b-home\") pod \"e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w\" (UID: \"9e439440-278c-4f4e-b7b1-f431c56f968b\") " pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" Jul 13 22:32:58.324184 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:58.324069 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e439440-278c-4f4e-b7b1-f431c56f968b-tls-certs\") pod \"e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w\" (UID: \"9e439440-278c-4f4e-b7b1-f431c56f968b\") " pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" Jul 13 22:32:58.324184 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:58.324085 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbtqd\" (UniqueName: \"kubernetes.io/projected/9e439440-278c-4f4e-b7b1-f431c56f968b-kube-api-access-wbtqd\") pod \"e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w\" (UID: \"9e439440-278c-4f4e-b7b1-f431c56f968b\") " pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" Jul 13 22:32:58.324184 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:58.324126 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e439440-278c-4f4e-b7b1-f431c56f968b-tmp-dir\") pod \"e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w\" (UID: \"9e439440-278c-4f4e-b7b1-f431c56f968b\") " pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" Jul 13 22:32:58.324184 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:58.324165 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e439440-278c-4f4e-b7b1-f431c56f968b-model-cache\") pod \"e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w\" (UID: \"9e439440-278c-4f4e-b7b1-f431c56f968b\") " pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" Jul 13 22:32:58.425684 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:58.425650 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e439440-278c-4f4e-b7b1-f431c56f968b-dshm\") pod \"e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w\" (UID: \"9e439440-278c-4f4e-b7b1-f431c56f968b\") " pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" Jul 13 22:32:58.425882 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:58.425691 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e439440-278c-4f4e-b7b1-f431c56f968b-home\") pod \"e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w\" (UID: \"9e439440-278c-4f4e-b7b1-f431c56f968b\") " pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" Jul 13 22:32:58.425882 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:58.425713 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e439440-278c-4f4e-b7b1-f431c56f968b-tls-certs\") pod \"e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w\" (UID: \"9e439440-278c-4f4e-b7b1-f431c56f968b\") " pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" Jul 13 22:32:58.425882 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:58.425735 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbtqd\" (UniqueName: \"kubernetes.io/projected/9e439440-278c-4f4e-b7b1-f431c56f968b-kube-api-access-wbtqd\") pod \"e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w\" (UID: \"9e439440-278c-4f4e-b7b1-f431c56f968b\") " pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" Jul 13 22:32:58.425882 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:58.425841 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e439440-278c-4f4e-b7b1-f431c56f968b-tmp-dir\") pod \"e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w\" (UID: \"9e439440-278c-4f4e-b7b1-f431c56f968b\") " pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" Jul 13 22:32:58.425882 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:58.425874 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e439440-278c-4f4e-b7b1-f431c56f968b-model-cache\") pod \"e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w\" (UID: \"9e439440-278c-4f4e-b7b1-f431c56f968b\") " pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" Jul 13 22:32:58.426169 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:58.426143 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e439440-278c-4f4e-b7b1-f431c56f968b-home\") pod \"e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w\" (UID: \"9e439440-278c-4f4e-b7b1-f431c56f968b\") " pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" Jul 13 22:32:58.426316 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:58.426204 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e439440-278c-4f4e-b7b1-f431c56f968b-tmp-dir\") pod \"e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w\" (UID: \"9e439440-278c-4f4e-b7b1-f431c56f968b\") " pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" Jul 13 22:32:58.426316 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:58.426247 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e439440-278c-4f4e-b7b1-f431c56f968b-model-cache\") pod \"e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w\" (UID: \"9e439440-278c-4f4e-b7b1-f431c56f968b\") " pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" Jul 13 22:32:58.427955 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:58.427937 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e439440-278c-4f4e-b7b1-f431c56f968b-dshm\") pod \"e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w\" (UID: \"9e439440-278c-4f4e-b7b1-f431c56f968b\") " pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" Jul 13 22:32:58.428235 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:58.428219 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e439440-278c-4f4e-b7b1-f431c56f968b-tls-certs\") pod \"e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w\" (UID: \"9e439440-278c-4f4e-b7b1-f431c56f968b\") " pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" Jul 13 22:32:58.432747 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:58.432726 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbtqd\" (UniqueName: \"kubernetes.io/projected/9e439440-278c-4f4e-b7b1-f431c56f968b-kube-api-access-wbtqd\") pod \"e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w\" (UID: \"9e439440-278c-4f4e-b7b1-f431c56f968b\") " pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" Jul 13 22:32:58.568833 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:58.568748 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" Jul 13 22:32:58.704546 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:58.704518 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w"] Jul 13 22:32:58.707207 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:32:58.707175 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e439440_278c_4f4e_b7b1_f431c56f968b.slice/crio-75bdc8fd8b7e6a1f490d84b8ca531a9bb55294f9c940fe647a7f637246e3f8a4 WatchSource:0}: Error finding container 75bdc8fd8b7e6a1f490d84b8ca531a9bb55294f9c940fe647a7f637246e3f8a4: Status 404 returned error can't find the container with id 75bdc8fd8b7e6a1f490d84b8ca531a9bb55294f9c940fe647a7f637246e3f8a4 Jul 13 22:32:59.344384 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:59.344346 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" event={"ID":"9e439440-278c-4f4e-b7b1-f431c56f968b","Type":"ContainerStarted","Data":"949605b3adde02c686b2b2fa436c66508ce3dfebfc99135fb327b97df84d7e0c"} Jul 13 22:32:59.344384 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:59.344391 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" event={"ID":"9e439440-278c-4f4e-b7b1-f431c56f968b","Type":"ContainerStarted","Data":"75bdc8fd8b7e6a1f490d84b8ca531a9bb55294f9c940fe647a7f637246e3f8a4"} Jul 13 22:32:59.360822 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:32:59.360770 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" podStartSLOduration=1.360755069 podStartE2EDuration="1.360755069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:32:59.359475031 +0000 UTC m=+2000.810030561" watchObservedRunningTime="2026-07-13 22:32:59.360755069 +0000 UTC m=+2000.811310606" Jul 13 22:33:01.747695 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:01.747654 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:33:08.569636 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:08.569600 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" Jul 13 22:33:08.570146 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:08.569650 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" Jul 13 22:33:08.582235 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:08.582202 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" Jul 13 22:33:09.402538 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:09.402510 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" Jul 13 22:33:14.442783 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:14.442749 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:33:31.752786 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:31.752750 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:33:37.688455 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:37.688372 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w"] Jul 13 22:33:52.709242 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:52.709179 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" podUID="9e439440-278c-4f4e-b7b1-f431c56f968b" containerName="main" containerID="cri-o://949605b3adde02c686b2b2fa436c66508ce3dfebfc99135fb327b97df84d7e0c" gracePeriod=45 Jul 13 22:33:52.956817 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:52.956794 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" Jul 13 22:33:53.033427 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:53.033357 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e439440-278c-4f4e-b7b1-f431c56f968b-dshm\") pod \"9e439440-278c-4f4e-b7b1-f431c56f968b\" (UID: \"9e439440-278c-4f4e-b7b1-f431c56f968b\") " Jul 13 22:33:53.033556 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:53.033436 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e439440-278c-4f4e-b7b1-f431c56f968b-tmp-dir\") pod \"9e439440-278c-4f4e-b7b1-f431c56f968b\" (UID: \"9e439440-278c-4f4e-b7b1-f431c56f968b\") " Jul 13 22:33:53.033556 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:53.033498 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbtqd\" (UniqueName: \"kubernetes.io/projected/9e439440-278c-4f4e-b7b1-f431c56f968b-kube-api-access-wbtqd\") pod \"9e439440-278c-4f4e-b7b1-f431c56f968b\" (UID: \"9e439440-278c-4f4e-b7b1-f431c56f968b\") " Jul 13 22:33:53.033556 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:53.033546 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e439440-278c-4f4e-b7b1-f431c56f968b-model-cache\") pod \"9e439440-278c-4f4e-b7b1-f431c56f968b\" (UID: \"9e439440-278c-4f4e-b7b1-f431c56f968b\") " Jul 13 22:33:53.033667 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:53.033574 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e439440-278c-4f4e-b7b1-f431c56f968b-home\") pod \"9e439440-278c-4f4e-b7b1-f431c56f968b\" (UID: \"9e439440-278c-4f4e-b7b1-f431c56f968b\") " Jul 13 22:33:53.033667 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:53.033632 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e439440-278c-4f4e-b7b1-f431c56f968b-tls-certs\") pod \"9e439440-278c-4f4e-b7b1-f431c56f968b\" (UID: \"9e439440-278c-4f4e-b7b1-f431c56f968b\") " Jul 13 22:33:53.033765 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:53.033723 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e439440-278c-4f4e-b7b1-f431c56f968b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9e439440-278c-4f4e-b7b1-f431c56f968b" (UID: "9e439440-278c-4f4e-b7b1-f431c56f968b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:33:53.033822 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:53.033794 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e439440-278c-4f4e-b7b1-f431c56f968b-model-cache" (OuterVolumeSpecName: "model-cache") pod "9e439440-278c-4f4e-b7b1-f431c56f968b" (UID: "9e439440-278c-4f4e-b7b1-f431c56f968b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:33:53.033907 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:53.033883 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e439440-278c-4f4e-b7b1-f431c56f968b-home" (OuterVolumeSpecName: "home") pod "9e439440-278c-4f4e-b7b1-f431c56f968b" (UID: "9e439440-278c-4f4e-b7b1-f431c56f968b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:33:53.034021 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:53.034000 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e439440-278c-4f4e-b7b1-f431c56f968b-model-cache\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:33:53.034021 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:53.034021 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e439440-278c-4f4e-b7b1-f431c56f968b-tmp-dir\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:33:53.035694 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:53.035663 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e439440-278c-4f4e-b7b1-f431c56f968b-dshm" (OuterVolumeSpecName: "dshm") pod "9e439440-278c-4f4e-b7b1-f431c56f968b" (UID: "9e439440-278c-4f4e-b7b1-f431c56f968b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:33:53.035790 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:53.035714 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e439440-278c-4f4e-b7b1-f431c56f968b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9e439440-278c-4f4e-b7b1-f431c56f968b" (UID: "9e439440-278c-4f4e-b7b1-f431c56f968b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:33:53.035790 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:53.035720 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e439440-278c-4f4e-b7b1-f431c56f968b-kube-api-access-wbtqd" (OuterVolumeSpecName: "kube-api-access-wbtqd") pod "9e439440-278c-4f4e-b7b1-f431c56f968b" (UID: "9e439440-278c-4f4e-b7b1-f431c56f968b"). InnerVolumeSpecName "kube-api-access-wbtqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:33:53.134988 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:53.134954 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbtqd\" (UniqueName: \"kubernetes.io/projected/9e439440-278c-4f4e-b7b1-f431c56f968b-kube-api-access-wbtqd\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:33:53.135138 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:53.135008 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e439440-278c-4f4e-b7b1-f431c56f968b-home\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:33:53.135138 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:53.135023 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e439440-278c-4f4e-b7b1-f431c56f968b-tls-certs\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:33:53.135138 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:53.135036 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e439440-278c-4f4e-b7b1-f431c56f968b-dshm\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:33:53.607193 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:53.607150 2562 generic.go:356] "Generic (PLEG): container finished" podID="9e439440-278c-4f4e-b7b1-f431c56f968b" containerID="949605b3adde02c686b2b2fa436c66508ce3dfebfc99135fb327b97df84d7e0c" exitCode=0 Jul 13 22:33:53.607363 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:53.607207 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" Jul 13 22:33:53.607363 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:53.607231 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" event={"ID":"9e439440-278c-4f4e-b7b1-f431c56f968b","Type":"ContainerDied","Data":"949605b3adde02c686b2b2fa436c66508ce3dfebfc99135fb327b97df84d7e0c"} Jul 13 22:33:53.607363 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:53.607280 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w" event={"ID":"9e439440-278c-4f4e-b7b1-f431c56f968b","Type":"ContainerDied","Data":"75bdc8fd8b7e6a1f490d84b8ca531a9bb55294f9c940fe647a7f637246e3f8a4"} Jul 13 22:33:53.607363 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:53.607298 2562 scope.go:117] "RemoveContainer" containerID="949605b3adde02c686b2b2fa436c66508ce3dfebfc99135fb327b97df84d7e0c" Jul 13 22:33:53.616284 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:53.616252 2562 scope.go:117] "RemoveContainer" containerID="949605b3adde02c686b2b2fa436c66508ce3dfebfc99135fb327b97df84d7e0c" Jul 13 22:33:53.616597 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:33:53.616579 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"949605b3adde02c686b2b2fa436c66508ce3dfebfc99135fb327b97df84d7e0c\": container with ID starting with 949605b3adde02c686b2b2fa436c66508ce3dfebfc99135fb327b97df84d7e0c not found: ID does not exist" containerID="949605b3adde02c686b2b2fa436c66508ce3dfebfc99135fb327b97df84d7e0c" Jul 13 22:33:53.616645 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:53.616605 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"949605b3adde02c686b2b2fa436c66508ce3dfebfc99135fb327b97df84d7e0c"} err="failed to get container status \"949605b3adde02c686b2b2fa436c66508ce3dfebfc99135fb327b97df84d7e0c\": rpc error: code = NotFound desc = could not find container \"949605b3adde02c686b2b2fa436c66508ce3dfebfc99135fb327b97df84d7e0c\": container with ID starting with 949605b3adde02c686b2b2fa436c66508ce3dfebfc99135fb327b97df84d7e0c not found: ID does not exist" Jul 13 22:33:53.627395 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:53.627365 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w"] Jul 13 22:33:53.631161 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:53.631137 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-86e22e9e/e2e-lifecycle-model-86e22e9e-kserve-549d7d9b59-t9b8w"] Jul 13 22:33:54.966726 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:33:54.966681 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e439440-278c-4f4e-b7b1-f431c56f968b" path="/var/lib/kubelet/pods/9e439440-278c-4f4e-b7b1-f431c56f968b/volumes" Jul 13 22:34:39.041695 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:34:39.041668 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x5qhq_7b1fa75b-f29d-45a1-af98-fa947c2058b2/console-operator/1.log" Jul 13 22:34:39.049039 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:34:39.049016 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw4h7_582b8d35-ee74-490e-aad6-e89ac36cb61c/ovn-acl-logging/0.log" Jul 13 22:34:39.049903 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:34:39.049884 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x5qhq_7b1fa75b-f29d-45a1-af98-fa947c2058b2/console-operator/1.log" Jul 13 22:34:39.056884 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:34:39.056865 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw4h7_582b8d35-ee74-490e-aad6-e89ac36cb61c/ovn-acl-logging/0.log" Jul 13 22:34:49.050999 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:34:49.050948 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:35:01.478534 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:01.478495 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:35:51.183185 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:51.183152 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:35:59.252902 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.252868 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4"] Jul 13 22:35:59.253437 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.253384 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e439440-278c-4f4e-b7b1-f431c56f968b" containerName="main" Jul 13 22:35:59.253437 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.253405 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e439440-278c-4f4e-b7b1-f431c56f968b" containerName="main" Jul 13 22:35:59.253556 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.253507 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e439440-278c-4f4e-b7b1-f431c56f968b" containerName="main" Jul 13 22:35:59.257103 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.257073 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" Jul 13 22:35:59.259445 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.259418 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7da020\"/\"kube-root-ca.crt\"" Jul 13 22:35:59.259709 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.259687 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7da020\"/\"test-model-7da020-kserve-self-signed-certs\"" Jul 13 22:35:59.259960 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.259939 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7da020\"/\"openshift-service-ca.crt\"" Jul 13 22:35:59.259960 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.259948 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7da020\"/\"default-dockercfg-76rxw\"" Jul 13 22:35:59.263641 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.263619 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4"] Jul 13 22:35:59.351469 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.351437 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-dshm\") pod \"test-model-7da020-kserve-77cdb64fd9-xsvz4\" (UID: \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\") " pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" Jul 13 22:35:59.351610 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.351474 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-home\") pod \"test-model-7da020-kserve-77cdb64fd9-xsvz4\" (UID: \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\") " pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" Jul 13 22:35:59.351610 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.351503 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-model-cache\") pod \"test-model-7da020-kserve-77cdb64fd9-xsvz4\" (UID: \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\") " pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" Jul 13 22:35:59.351610 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.351560 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-tls-certs\") pod \"test-model-7da020-kserve-77cdb64fd9-xsvz4\" (UID: \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\") " pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" Jul 13 22:35:59.351715 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.351630 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-tmp-dir\") pod \"test-model-7da020-kserve-77cdb64fd9-xsvz4\" (UID: \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\") " pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" Jul 13 22:35:59.351715 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.351666 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx4t6\" (UniqueName: \"kubernetes.io/projected/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-kube-api-access-hx4t6\") pod \"test-model-7da020-kserve-77cdb64fd9-xsvz4\" (UID: \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\") " pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" Jul 13 22:35:59.453537 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.453501 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-dshm\") pod \"test-model-7da020-kserve-77cdb64fd9-xsvz4\" (UID: \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\") " pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" Jul 13 22:35:59.453727 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.453546 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-home\") pod \"test-model-7da020-kserve-77cdb64fd9-xsvz4\" (UID: \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\") " pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" Jul 13 22:35:59.453727 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.453577 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-model-cache\") pod \"test-model-7da020-kserve-77cdb64fd9-xsvz4\" (UID: \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\") " pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" Jul 13 22:35:59.453727 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.453603 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-tls-certs\") pod \"test-model-7da020-kserve-77cdb64fd9-xsvz4\" (UID: \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\") " pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" Jul 13 22:35:59.453727 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.453660 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-tmp-dir\") pod \"test-model-7da020-kserve-77cdb64fd9-xsvz4\" (UID: \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\") " pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" Jul 13 22:35:59.453727 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.453708 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx4t6\" (UniqueName: \"kubernetes.io/projected/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-kube-api-access-hx4t6\") pod \"test-model-7da020-kserve-77cdb64fd9-xsvz4\" (UID: \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\") " pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" Jul 13 22:35:59.454038 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.454002 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-home\") pod \"test-model-7da020-kserve-77cdb64fd9-xsvz4\" (UID: \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\") " pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" Jul 13 22:35:59.454126 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.454102 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-model-cache\") pod \"test-model-7da020-kserve-77cdb64fd9-xsvz4\" (UID: \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\") " pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" Jul 13 22:35:59.454261 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.454231 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-tmp-dir\") pod \"test-model-7da020-kserve-77cdb64fd9-xsvz4\" (UID: \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\") " pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" Jul 13 22:35:59.456254 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.456236 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-tls-certs\") pod \"test-model-7da020-kserve-77cdb64fd9-xsvz4\" (UID: \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\") " pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" Jul 13 22:35:59.456357 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.456284 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-dshm\") pod \"test-model-7da020-kserve-77cdb64fd9-xsvz4\" (UID: \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\") " pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" Jul 13 22:35:59.462162 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.462139 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx4t6\" (UniqueName: \"kubernetes.io/projected/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-kube-api-access-hx4t6\") pod \"test-model-7da020-kserve-77cdb64fd9-xsvz4\" (UID: \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\") " pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" Jul 13 22:35:59.569140 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.569050 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" Jul 13 22:35:59.700933 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:35:59.700830 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4"] Jul 13 22:35:59.703264 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:35:59.703237 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6988d6f_aff0_4b7a_914b_4e0d7be91deb.slice/crio-7cc3c2a3e36bc3258f09932513f4fc85a12dcfdb6a333f6dc5ef7defc0195eae WatchSource:0}: Error finding container 7cc3c2a3e36bc3258f09932513f4fc85a12dcfdb6a333f6dc5ef7defc0195eae: Status 404 returned error can't find the container with id 7cc3c2a3e36bc3258f09932513f4fc85a12dcfdb6a333f6dc5ef7defc0195eae Jul 13 22:36:00.193386 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:00.193341 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" event={"ID":"e6988d6f-aff0-4b7a-914b-4e0d7be91deb","Type":"ContainerStarted","Data":"a460422cd90dfea2b4f3c3ce3784bf8b564f951f1a1df04a1612c16b76e9bc79"} Jul 13 22:36:00.193386 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:00.193388 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" event={"ID":"e6988d6f-aff0-4b7a-914b-4e0d7be91deb","Type":"ContainerStarted","Data":"7cc3c2a3e36bc3258f09932513f4fc85a12dcfdb6a333f6dc5ef7defc0195eae"} Jul 13 22:36:00.209335 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:00.209289 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" podStartSLOduration=1.209274649 podStartE2EDuration="1.209274649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:36:00.207613908 +0000 UTC m=+2181.658169447" watchObservedRunningTime="2026-07-13 22:36:00.209274649 +0000 UTC m=+2181.659830186" Jul 13 22:36:06.349923 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:06.349887 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:36:09.570152 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:09.570121 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" Jul 13 22:36:09.570152 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:09.570161 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" Jul 13 22:36:09.582694 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:09.582671 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" Jul 13 22:36:10.253590 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:10.253557 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" Jul 13 22:36:28.453514 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:28.453478 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4"] Jul 13 22:36:28.457099 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:28.457071 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" Jul 13 22:36:28.459183 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:28.459164 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a502db\"/\"openshift-service-ca.crt\"" Jul 13 22:36:28.459747 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:28.459705 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a502db\"/\"default-dockercfg-qgwmf\"" Jul 13 22:36:28.459747 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:28.459738 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a502db\"/\"kube-root-ca.crt\"" Jul 13 22:36:28.459909 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:28.459757 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a502db\"/\"test-model-a502db-kserve-self-signed-certs\"" Jul 13 22:36:28.464800 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:28.464778 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4"] Jul 13 22:36:28.612691 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:28.612655 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88e76216-5b9b-4e79-a17c-d23cf10fbebe-home\") pod \"test-model-a502db-kserve-6578d84765-52pb4\" (UID: \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\") " pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" Jul 13 22:36:28.612857 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:28.612711 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v95z\" (UniqueName: \"kubernetes.io/projected/88e76216-5b9b-4e79-a17c-d23cf10fbebe-kube-api-access-6v95z\") pod \"test-model-a502db-kserve-6578d84765-52pb4\" (UID: \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\") " pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" Jul 13 22:36:28.612857 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:28.612741 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88e76216-5b9b-4e79-a17c-d23cf10fbebe-tmp-dir\") pod \"test-model-a502db-kserve-6578d84765-52pb4\" (UID: \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\") " pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" Jul 13 22:36:28.612857 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:28.612824 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e76216-5b9b-4e79-a17c-d23cf10fbebe-tls-certs\") pod \"test-model-a502db-kserve-6578d84765-52pb4\" (UID: \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\") " pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" Jul 13 22:36:28.613009 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:28.612864 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88e76216-5b9b-4e79-a17c-d23cf10fbebe-model-cache\") pod \"test-model-a502db-kserve-6578d84765-52pb4\" (UID: \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\") " pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" Jul 13 22:36:28.613009 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:28.612885 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88e76216-5b9b-4e79-a17c-d23cf10fbebe-dshm\") pod \"test-model-a502db-kserve-6578d84765-52pb4\" (UID: \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\") " pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" Jul 13 22:36:28.714097 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:28.714020 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v95z\" (UniqueName: \"kubernetes.io/projected/88e76216-5b9b-4e79-a17c-d23cf10fbebe-kube-api-access-6v95z\") pod \"test-model-a502db-kserve-6578d84765-52pb4\" (UID: \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\") " pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" Jul 13 22:36:28.714097 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:28.714058 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88e76216-5b9b-4e79-a17c-d23cf10fbebe-tmp-dir\") pod \"test-model-a502db-kserve-6578d84765-52pb4\" (UID: \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\") " pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" Jul 13 22:36:28.714097 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:28.714091 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e76216-5b9b-4e79-a17c-d23cf10fbebe-tls-certs\") pod \"test-model-a502db-kserve-6578d84765-52pb4\" (UID: \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\") " pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" Jul 13 22:36:28.714335 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:28.714117 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88e76216-5b9b-4e79-a17c-d23cf10fbebe-model-cache\") pod \"test-model-a502db-kserve-6578d84765-52pb4\" (UID: \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\") " pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" Jul 13 22:36:28.714335 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:28.714135 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88e76216-5b9b-4e79-a17c-d23cf10fbebe-dshm\") pod \"test-model-a502db-kserve-6578d84765-52pb4\" (UID: \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\") " pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" Jul 13 22:36:28.714335 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:28.714168 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88e76216-5b9b-4e79-a17c-d23cf10fbebe-home\") pod \"test-model-a502db-kserve-6578d84765-52pb4\" (UID: \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\") " pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" Jul 13 22:36:28.714492 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:28.714426 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88e76216-5b9b-4e79-a17c-d23cf10fbebe-tmp-dir\") pod \"test-model-a502db-kserve-6578d84765-52pb4\" (UID: \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\") " pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" Jul 13 22:36:28.714551 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:28.714487 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88e76216-5b9b-4e79-a17c-d23cf10fbebe-home\") pod \"test-model-a502db-kserve-6578d84765-52pb4\" (UID: \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\") " pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" Jul 13 22:36:28.714603 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:28.714555 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88e76216-5b9b-4e79-a17c-d23cf10fbebe-model-cache\") pod \"test-model-a502db-kserve-6578d84765-52pb4\" (UID: \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\") " pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" Jul 13 22:36:28.716445 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:28.716422 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88e76216-5b9b-4e79-a17c-d23cf10fbebe-dshm\") pod \"test-model-a502db-kserve-6578d84765-52pb4\" (UID: \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\") " pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" Jul 13 22:36:28.716594 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:28.716577 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e76216-5b9b-4e79-a17c-d23cf10fbebe-tls-certs\") pod \"test-model-a502db-kserve-6578d84765-52pb4\" (UID: \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\") " pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" Jul 13 22:36:28.721502 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:28.721480 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v95z\" (UniqueName: \"kubernetes.io/projected/88e76216-5b9b-4e79-a17c-d23cf10fbebe-kube-api-access-6v95z\") pod \"test-model-a502db-kserve-6578d84765-52pb4\" (UID: \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\") " pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" Jul 13 22:36:28.768421 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:28.768390 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" Jul 13 22:36:28.899050 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:28.899008 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4"] Jul 13 22:36:28.901347 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:36:28.901320 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88e76216_5b9b_4e79_a17c_d23cf10fbebe.slice/crio-0b11f0972137305d345facf5aa6aba75f607c7b2f6d9adb6de197de97816ac31 WatchSource:0}: Error finding container 0b11f0972137305d345facf5aa6aba75f607c7b2f6d9adb6de197de97816ac31: Status 404 returned error can't find the container with id 0b11f0972137305d345facf5aa6aba75f607c7b2f6d9adb6de197de97816ac31 Jul 13 22:36:29.334486 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:29.334450 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" event={"ID":"88e76216-5b9b-4e79-a17c-d23cf10fbebe","Type":"ContainerStarted","Data":"3cfea72852dd8e24f1784fd6c2516119a5a560a17f6d7978600d1d78ea55e730"} Jul 13 22:36:29.334486 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:29.334491 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" event={"ID":"88e76216-5b9b-4e79-a17c-d23cf10fbebe","Type":"ContainerStarted","Data":"0b11f0972137305d345facf5aa6aba75f607c7b2f6d9adb6de197de97816ac31"} Jul 13 22:36:29.352916 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:29.352870 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" podStartSLOduration=1.3528549380000001 podStartE2EDuration="1.352854938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:36:29.351049852 +0000 UTC m=+2210.801605390" watchObservedRunningTime="2026-07-13 22:36:29.352854938 +0000 UTC m=+2210.803410475" Jul 13 22:36:32.849762 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:32.849725 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:36:38.769053 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:38.769016 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" Jul 13 22:36:38.769554 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:38.769087 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" Jul 13 22:36:38.781495 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:38.781472 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" Jul 13 22:36:39.390739 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:39.390707 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" Jul 13 22:36:47.153726 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:47.153689 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:36:49.011225 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:49.011186 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4"] Jul 13 22:36:49.867818 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:49.867780 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4"] Jul 13 22:36:58.744947 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:36:58.744907 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:37:04.031161 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.031061 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" podUID="e6988d6f-aff0-4b7a-914b-4e0d7be91deb" containerName="main" containerID="cri-o://a460422cd90dfea2b4f3c3ce3784bf8b564f951f1a1df04a1612c16b76e9bc79" gracePeriod=45 Jul 13 22:37:04.291370 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.291314 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" Jul 13 22:37:04.364029 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.363996 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-model-cache\") pod \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\" (UID: \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\") " Jul 13 22:37:04.364183 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.364060 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-tls-certs\") pod \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\" (UID: \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\") " Jul 13 22:37:04.364183 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.364087 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hx4t6\" (UniqueName: \"kubernetes.io/projected/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-kube-api-access-hx4t6\") pod \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\" (UID: \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\") " Jul 13 22:37:04.364183 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.364161 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-tmp-dir\") pod \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\" (UID: \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\") " Jul 13 22:37:04.364308 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.364193 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-dshm\") pod \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\" (UID: \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\") " Jul 13 22:37:04.364308 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.364215 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-model-cache" (OuterVolumeSpecName: "model-cache") pod "e6988d6f-aff0-4b7a-914b-4e0d7be91deb" (UID: "e6988d6f-aff0-4b7a-914b-4e0d7be91deb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:37:04.364308 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.364269 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-home\") pod \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\" (UID: \"e6988d6f-aff0-4b7a-914b-4e0d7be91deb\") " Jul 13 22:37:04.364461 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.364425 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e6988d6f-aff0-4b7a-914b-4e0d7be91deb" (UID: "e6988d6f-aff0-4b7a-914b-4e0d7be91deb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:37:04.364588 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.364563 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-home" (OuterVolumeSpecName: "home") pod "e6988d6f-aff0-4b7a-914b-4e0d7be91deb" (UID: "e6988d6f-aff0-4b7a-914b-4e0d7be91deb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:37:04.366410 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.364668 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-home\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:37:04.366410 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.365074 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-model-cache\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:37:04.366410 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.365092 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-tmp-dir\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:37:04.372320 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.372292 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6988d6f-aff0-4b7a-914b-4e0d7be91deb" (UID: "e6988d6f-aff0-4b7a-914b-4e0d7be91deb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:37:04.372320 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.372304 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-dshm" (OuterVolumeSpecName: "dshm") pod "e6988d6f-aff0-4b7a-914b-4e0d7be91deb" (UID: "e6988d6f-aff0-4b7a-914b-4e0d7be91deb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:37:04.372503 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.372319 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-kube-api-access-hx4t6" (OuterVolumeSpecName: "kube-api-access-hx4t6") pod "e6988d6f-aff0-4b7a-914b-4e0d7be91deb" (UID: "e6988d6f-aff0-4b7a-914b-4e0d7be91deb"). InnerVolumeSpecName "kube-api-access-hx4t6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:37:04.465950 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.465915 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-tls-certs\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:37:04.465950 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.465945 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx4t6\" (UniqueName: \"kubernetes.io/projected/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-kube-api-access-hx4t6\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:37:04.465950 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.465955 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6988d6f-aff0-4b7a-914b-4e0d7be91deb-dshm\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:37:04.502083 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.502048 2562 generic.go:356] "Generic (PLEG): container finished" podID="e6988d6f-aff0-4b7a-914b-4e0d7be91deb" containerID="a460422cd90dfea2b4f3c3ce3784bf8b564f951f1a1df04a1612c16b76e9bc79" exitCode=0 Jul 13 22:37:04.502249 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.502105 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" Jul 13 22:37:04.502249 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.502132 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" event={"ID":"e6988d6f-aff0-4b7a-914b-4e0d7be91deb","Type":"ContainerDied","Data":"a460422cd90dfea2b4f3c3ce3784bf8b564f951f1a1df04a1612c16b76e9bc79"} Jul 13 22:37:04.502249 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.502180 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4" event={"ID":"e6988d6f-aff0-4b7a-914b-4e0d7be91deb","Type":"ContainerDied","Data":"7cc3c2a3e36bc3258f09932513f4fc85a12dcfdb6a333f6dc5ef7defc0195eae"} Jul 13 22:37:04.502249 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.502202 2562 scope.go:117] "RemoveContainer" containerID="a460422cd90dfea2b4f3c3ce3784bf8b564f951f1a1df04a1612c16b76e9bc79" Jul 13 22:37:04.511541 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.511521 2562 scope.go:117] "RemoveContainer" containerID="a460422cd90dfea2b4f3c3ce3784bf8b564f951f1a1df04a1612c16b76e9bc79" Jul 13 22:37:04.511785 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:37:04.511764 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a460422cd90dfea2b4f3c3ce3784bf8b564f951f1a1df04a1612c16b76e9bc79\": container with ID starting with a460422cd90dfea2b4f3c3ce3784bf8b564f951f1a1df04a1612c16b76e9bc79 not found: ID does not exist" containerID="a460422cd90dfea2b4f3c3ce3784bf8b564f951f1a1df04a1612c16b76e9bc79" Jul 13 22:37:04.511829 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.511793 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a460422cd90dfea2b4f3c3ce3784bf8b564f951f1a1df04a1612c16b76e9bc79"} err="failed to get container status \"a460422cd90dfea2b4f3c3ce3784bf8b564f951f1a1df04a1612c16b76e9bc79\": rpc error: code = NotFound desc = could not find container \"a460422cd90dfea2b4f3c3ce3784bf8b564f951f1a1df04a1612c16b76e9bc79\": container with ID starting with a460422cd90dfea2b4f3c3ce3784bf8b564f951f1a1df04a1612c16b76e9bc79 not found: ID does not exist" Jul 13 22:37:04.521961 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.521941 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4"] Jul 13 22:37:04.528898 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.528878 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7da020/test-model-7da020-kserve-77cdb64fd9-xsvz4"] Jul 13 22:37:04.888427 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.888375 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" podUID="88e76216-5b9b-4e79-a17c-d23cf10fbebe" containerName="main" containerID="cri-o://3cfea72852dd8e24f1784fd6c2516119a5a560a17f6d7978600d1d78ea55e730" gracePeriod=45 Jul 13 22:37:04.965674 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:04.965643 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6988d6f-aff0-4b7a-914b-4e0d7be91deb" path="/var/lib/kubelet/pods/e6988d6f-aff0-4b7a-914b-4e0d7be91deb/volumes" Jul 13 22:37:05.163401 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.163372 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" Jul 13 22:37:05.172342 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.172313 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88e76216-5b9b-4e79-a17c-d23cf10fbebe-model-cache\") pod \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\" (UID: \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\") " Jul 13 22:37:05.172484 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.172397 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88e76216-5b9b-4e79-a17c-d23cf10fbebe-dshm\") pod \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\" (UID: \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\") " Jul 13 22:37:05.172484 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.172420 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88e76216-5b9b-4e79-a17c-d23cf10fbebe-home\") pod \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\" (UID: \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\") " Jul 13 22:37:05.172484 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.172469 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88e76216-5b9b-4e79-a17c-d23cf10fbebe-tmp-dir\") pod \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\" (UID: \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\") " Jul 13 22:37:05.172649 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.172527 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e76216-5b9b-4e79-a17c-d23cf10fbebe-tls-certs\") pod \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\" (UID: \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\") " Jul 13 22:37:05.172649 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.172565 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e76216-5b9b-4e79-a17c-d23cf10fbebe-model-cache" (OuterVolumeSpecName: "model-cache") pod "88e76216-5b9b-4e79-a17c-d23cf10fbebe" (UID: "88e76216-5b9b-4e79-a17c-d23cf10fbebe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:37:05.172649 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.172620 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6v95z\" (UniqueName: \"kubernetes.io/projected/88e76216-5b9b-4e79-a17c-d23cf10fbebe-kube-api-access-6v95z\") pod \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\" (UID: \"88e76216-5b9b-4e79-a17c-d23cf10fbebe\") " Jul 13 22:37:05.172773 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.172670 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e76216-5b9b-4e79-a17c-d23cf10fbebe-home" (OuterVolumeSpecName: "home") pod "88e76216-5b9b-4e79-a17c-d23cf10fbebe" (UID: "88e76216-5b9b-4e79-a17c-d23cf10fbebe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:37:05.172831 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.172801 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e76216-5b9b-4e79-a17c-d23cf10fbebe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88e76216-5b9b-4e79-a17c-d23cf10fbebe" (UID: "88e76216-5b9b-4e79-a17c-d23cf10fbebe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:37:05.172969 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.172947 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88e76216-5b9b-4e79-a17c-d23cf10fbebe-model-cache\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:37:05.173058 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.172993 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88e76216-5b9b-4e79-a17c-d23cf10fbebe-home\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:37:05.173058 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.173006 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88e76216-5b9b-4e79-a17c-d23cf10fbebe-tmp-dir\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:37:05.174754 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.174725 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e76216-5b9b-4e79-a17c-d23cf10fbebe-dshm" (OuterVolumeSpecName: "dshm") pod "88e76216-5b9b-4e79-a17c-d23cf10fbebe" (UID: "88e76216-5b9b-4e79-a17c-d23cf10fbebe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:37:05.174847 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.174792 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e76216-5b9b-4e79-a17c-d23cf10fbebe-kube-api-access-6v95z" (OuterVolumeSpecName: "kube-api-access-6v95z") pod "88e76216-5b9b-4e79-a17c-d23cf10fbebe" (UID: "88e76216-5b9b-4e79-a17c-d23cf10fbebe"). InnerVolumeSpecName "kube-api-access-6v95z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:37:05.174847 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.174798 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e76216-5b9b-4e79-a17c-d23cf10fbebe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88e76216-5b9b-4e79-a17c-d23cf10fbebe" (UID: "88e76216-5b9b-4e79-a17c-d23cf10fbebe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:37:05.274018 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.273955 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v95z\" (UniqueName: \"kubernetes.io/projected/88e76216-5b9b-4e79-a17c-d23cf10fbebe-kube-api-access-6v95z\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:37:05.274018 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.274007 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88e76216-5b9b-4e79-a17c-d23cf10fbebe-dshm\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:37:05.274018 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.274022 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88e76216-5b9b-4e79-a17c-d23cf10fbebe-tls-certs\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:37:05.508354 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.508323 2562 generic.go:356] "Generic (PLEG): container finished" podID="88e76216-5b9b-4e79-a17c-d23cf10fbebe" containerID="3cfea72852dd8e24f1784fd6c2516119a5a560a17f6d7978600d1d78ea55e730" exitCode=0 Jul 13 22:37:05.508525 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.508360 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" Jul 13 22:37:05.508525 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.508412 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" event={"ID":"88e76216-5b9b-4e79-a17c-d23cf10fbebe","Type":"ContainerDied","Data":"3cfea72852dd8e24f1784fd6c2516119a5a560a17f6d7978600d1d78ea55e730"} Jul 13 22:37:05.508525 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.508451 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4" event={"ID":"88e76216-5b9b-4e79-a17c-d23cf10fbebe","Type":"ContainerDied","Data":"0b11f0972137305d345facf5aa6aba75f607c7b2f6d9adb6de197de97816ac31"} Jul 13 22:37:05.508525 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.508470 2562 scope.go:117] "RemoveContainer" containerID="3cfea72852dd8e24f1784fd6c2516119a5a560a17f6d7978600d1d78ea55e730" Jul 13 22:37:05.517763 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.517746 2562 scope.go:117] "RemoveContainer" containerID="3cfea72852dd8e24f1784fd6c2516119a5a560a17f6d7978600d1d78ea55e730" Jul 13 22:37:05.518067 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:37:05.518043 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cfea72852dd8e24f1784fd6c2516119a5a560a17f6d7978600d1d78ea55e730\": container with ID starting with 3cfea72852dd8e24f1784fd6c2516119a5a560a17f6d7978600d1d78ea55e730 not found: ID does not exist" containerID="3cfea72852dd8e24f1784fd6c2516119a5a560a17f6d7978600d1d78ea55e730" Jul 13 22:37:05.518126 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.518077 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cfea72852dd8e24f1784fd6c2516119a5a560a17f6d7978600d1d78ea55e730"} err="failed to get container status \"3cfea72852dd8e24f1784fd6c2516119a5a560a17f6d7978600d1d78ea55e730\": rpc error: code = NotFound desc = could not find container \"3cfea72852dd8e24f1784fd6c2516119a5a560a17f6d7978600d1d78ea55e730\": container with ID starting with 3cfea72852dd8e24f1784fd6c2516119a5a560a17f6d7978600d1d78ea55e730 not found: ID does not exist" Jul 13 22:37:05.528951 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.528928 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4"] Jul 13 22:37:05.532124 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:05.532102 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a502db/test-model-a502db-kserve-6578d84765-52pb4"] Jul 13 22:37:06.967053 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:06.967022 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88e76216-5b9b-4e79-a17c-d23cf10fbebe" path="/var/lib/kubelet/pods/88e76216-5b9b-4e79-a17c-d23cf10fbebe/volumes" Jul 13 22:37:09.350783 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:09.350750 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:37:29.750437 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:29.750401 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:37:51.349997 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:37:51.349952 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:38:08.448356 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:38:08.448318 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:38:21.550902 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:38:21.550860 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:38:46.441591 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:38:46.441559 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:39:07.561438 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:07.561400 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:39:24.057190 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:24.057157 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:39:29.458717 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.458684 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw"] Jul 13 22:39:29.459111 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.459073 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6988d6f-aff0-4b7a-914b-4e0d7be91deb" containerName="main" Jul 13 22:39:29.459111 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.459085 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6988d6f-aff0-4b7a-914b-4e0d7be91deb" containerName="main" Jul 13 22:39:29.459111 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.459094 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88e76216-5b9b-4e79-a17c-d23cf10fbebe" containerName="main" Jul 13 22:39:29.459111 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.459099 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="88e76216-5b9b-4e79-a17c-d23cf10fbebe" containerName="main" Jul 13 22:39:29.459254 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.459160 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="88e76216-5b9b-4e79-a17c-d23cf10fbebe" containerName="main" Jul 13 22:39:29.459254 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.459169 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6988d6f-aff0-4b7a-914b-4e0d7be91deb" containerName="main" Jul 13 22:39:29.462228 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.462210 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" Jul 13 22:39:29.464558 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.464533 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-13bc0c\"/\"default-dockercfg-58ljc\"" Jul 13 22:39:29.465173 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.465150 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-13bc0c\"/\"openshift-service-ca.crt\"" Jul 13 22:39:29.465287 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.465213 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-13bc0c\"/\"auth-test-model-13bc0c-kserve-self-signed-certs\"" Jul 13 22:39:29.465287 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.465241 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-13bc0c\"/\"kube-root-ca.crt\"" Jul 13 22:39:29.473568 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.473538 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw"] Jul 13 22:39:29.569914 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.569883 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56f4b26f-6586-4955-bea1-3759ab54b340-home\") pod \"auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw\" (UID: \"56f4b26f-6586-4955-bea1-3759ab54b340\") " pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" Jul 13 22:39:29.569914 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.569916 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56f4b26f-6586-4955-bea1-3759ab54b340-dshm\") pod \"auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw\" (UID: \"56f4b26f-6586-4955-bea1-3759ab54b340\") " pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" Jul 13 22:39:29.570166 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.569934 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56f4b26f-6586-4955-bea1-3759ab54b340-tls-certs\") pod \"auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw\" (UID: \"56f4b26f-6586-4955-bea1-3759ab54b340\") " pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" Jul 13 22:39:29.570166 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.570088 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56f4b26f-6586-4955-bea1-3759ab54b340-tmp-dir\") pod \"auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw\" (UID: \"56f4b26f-6586-4955-bea1-3759ab54b340\") " pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" Jul 13 22:39:29.570166 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.570119 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6t97\" (UniqueName: \"kubernetes.io/projected/56f4b26f-6586-4955-bea1-3759ab54b340-kube-api-access-k6t97\") pod \"auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw\" (UID: \"56f4b26f-6586-4955-bea1-3759ab54b340\") " pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" Jul 13 22:39:29.570166 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.570139 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56f4b26f-6586-4955-bea1-3759ab54b340-model-cache\") pod \"auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw\" (UID: \"56f4b26f-6586-4955-bea1-3759ab54b340\") " pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" Jul 13 22:39:29.671362 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.671327 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56f4b26f-6586-4955-bea1-3759ab54b340-home\") pod \"auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw\" (UID: \"56f4b26f-6586-4955-bea1-3759ab54b340\") " pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" Jul 13 22:39:29.671362 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.671364 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56f4b26f-6586-4955-bea1-3759ab54b340-dshm\") pod \"auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw\" (UID: \"56f4b26f-6586-4955-bea1-3759ab54b340\") " pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" Jul 13 22:39:29.671538 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.671388 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56f4b26f-6586-4955-bea1-3759ab54b340-tls-certs\") pod \"auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw\" (UID: \"56f4b26f-6586-4955-bea1-3759ab54b340\") " pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" Jul 13 22:39:29.671538 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.671476 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56f4b26f-6586-4955-bea1-3759ab54b340-tmp-dir\") pod \"auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw\" (UID: \"56f4b26f-6586-4955-bea1-3759ab54b340\") " pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" Jul 13 22:39:29.671623 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.671527 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6t97\" (UniqueName: \"kubernetes.io/projected/56f4b26f-6586-4955-bea1-3759ab54b340-kube-api-access-k6t97\") pod \"auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw\" (UID: \"56f4b26f-6586-4955-bea1-3759ab54b340\") " pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" Jul 13 22:39:29.671623 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.671572 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56f4b26f-6586-4955-bea1-3759ab54b340-model-cache\") pod \"auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw\" (UID: \"56f4b26f-6586-4955-bea1-3759ab54b340\") " pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" Jul 13 22:39:29.671887 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.671863 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56f4b26f-6586-4955-bea1-3759ab54b340-home\") pod \"auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw\" (UID: \"56f4b26f-6586-4955-bea1-3759ab54b340\") " pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" Jul 13 22:39:29.671887 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.671877 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56f4b26f-6586-4955-bea1-3759ab54b340-tmp-dir\") pod \"auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw\" (UID: \"56f4b26f-6586-4955-bea1-3759ab54b340\") " pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" Jul 13 22:39:29.672100 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.671944 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56f4b26f-6586-4955-bea1-3759ab54b340-model-cache\") pod \"auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw\" (UID: \"56f4b26f-6586-4955-bea1-3759ab54b340\") " pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" Jul 13 22:39:29.673567 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.673542 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56f4b26f-6586-4955-bea1-3759ab54b340-dshm\") pod \"auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw\" (UID: \"56f4b26f-6586-4955-bea1-3759ab54b340\") " pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" Jul 13 22:39:29.674016 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.673999 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56f4b26f-6586-4955-bea1-3759ab54b340-tls-certs\") pod \"auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw\" (UID: \"56f4b26f-6586-4955-bea1-3759ab54b340\") " pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" Jul 13 22:39:29.678572 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.678549 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6t97\" (UniqueName: \"kubernetes.io/projected/56f4b26f-6586-4955-bea1-3759ab54b340-kube-api-access-k6t97\") pod \"auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw\" (UID: \"56f4b26f-6586-4955-bea1-3759ab54b340\") " pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" Jul 13 22:39:29.773135 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.773053 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" Jul 13 22:39:29.914187 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:29.906431 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw"] Jul 13 22:39:30.181422 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:30.181341 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" event={"ID":"56f4b26f-6586-4955-bea1-3759ab54b340","Type":"ContainerStarted","Data":"e14cb823394fff8fd9c0dbb9ce2c14f9f75e11f60850c0a9fdea8c5204c52880"} Jul 13 22:39:30.181422 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:30.181418 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" event={"ID":"56f4b26f-6586-4955-bea1-3759ab54b340","Type":"ContainerStarted","Data":"9d022d864c7d22e8e4c06baafb3739f42fc1ae730206a40768c236c9baed9852"} Jul 13 22:39:30.197281 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:30.197227 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" podStartSLOduration=1.197209809 podStartE2EDuration="1.197209809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:39:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:39:30.196299555 +0000 UTC m=+2391.646855093" watchObservedRunningTime="2026-07-13 22:39:30.197209809 +0000 UTC m=+2391.647765346" Jul 13 22:39:33.346026 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:33.345990 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:39:39.070917 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:39.070880 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x5qhq_7b1fa75b-f29d-45a1-af98-fa947c2058b2/console-operator/1.log" Jul 13 22:39:39.078178 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:39.078150 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw4h7_582b8d35-ee74-490e-aad6-e89ac36cb61c/ovn-acl-logging/0.log" Jul 13 22:39:39.079730 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:39.079709 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x5qhq_7b1fa75b-f29d-45a1-af98-fa947c2058b2/console-operator/1.log" Jul 13 22:39:39.086589 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:39.086568 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw4h7_582b8d35-ee74-490e-aad6-e89ac36cb61c/ovn-acl-logging/0.log" Jul 13 22:39:39.773254 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:39.773212 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" Jul 13 22:39:39.773254 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:39.773263 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" Jul 13 22:39:39.787179 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:39.787148 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" Jul 13 22:39:40.239465 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:40.239433 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" Jul 13 22:39:59.953892 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:59.953858 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj"] Jul 13 22:39:59.957531 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:59.957503 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" Jul 13 22:39:59.960231 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:59.960207 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bdc0cb\"/\"default-dockercfg-rh4zh\"" Jul 13 22:39:59.960459 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:59.960267 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bdc0cb\"/\"openshift-service-ca.crt\"" Jul 13 22:39:59.960536 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:59.960279 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bdc0cb\"/\"auth-test-model-bdc0cb-kserve-self-signed-certs\"" Jul 13 22:39:59.960536 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:59.960293 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bdc0cb\"/\"kube-root-ca.crt\"" Jul 13 22:39:59.966674 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:39:59.966647 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj"] Jul 13 22:40:00.039549 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:00.039517 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/855770ce-ba3d-4fb5-add0-88872c966ac3-model-cache\") pod \"auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj\" (UID: \"855770ce-ba3d-4fb5-add0-88872c966ac3\") " pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" Jul 13 22:40:00.039549 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:00.039551 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/855770ce-ba3d-4fb5-add0-88872c966ac3-home\") pod \"auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj\" (UID: \"855770ce-ba3d-4fb5-add0-88872c966ac3\") " pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" Jul 13 22:40:00.039802 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:00.039578 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/855770ce-ba3d-4fb5-add0-88872c966ac3-dshm\") pod \"auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj\" (UID: \"855770ce-ba3d-4fb5-add0-88872c966ac3\") " pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" Jul 13 22:40:00.039802 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:00.039743 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpp6p\" (UniqueName: \"kubernetes.io/projected/855770ce-ba3d-4fb5-add0-88872c966ac3-kube-api-access-bpp6p\") pod \"auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj\" (UID: \"855770ce-ba3d-4fb5-add0-88872c966ac3\") " pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" Jul 13 22:40:00.039802 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:00.039778 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/855770ce-ba3d-4fb5-add0-88872c966ac3-tmp-dir\") pod \"auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj\" (UID: \"855770ce-ba3d-4fb5-add0-88872c966ac3\") " pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" Jul 13 22:40:00.039923 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:00.039889 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/855770ce-ba3d-4fb5-add0-88872c966ac3-tls-certs\") pod \"auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj\" (UID: \"855770ce-ba3d-4fb5-add0-88872c966ac3\") " pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" Jul 13 22:40:00.140731 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:00.140693 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/855770ce-ba3d-4fb5-add0-88872c966ac3-dshm\") pod \"auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj\" (UID: \"855770ce-ba3d-4fb5-add0-88872c966ac3\") " pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" Jul 13 22:40:00.140926 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:00.140744 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpp6p\" (UniqueName: \"kubernetes.io/projected/855770ce-ba3d-4fb5-add0-88872c966ac3-kube-api-access-bpp6p\") pod \"auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj\" (UID: \"855770ce-ba3d-4fb5-add0-88872c966ac3\") " pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" Jul 13 22:40:00.140926 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:00.140768 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/855770ce-ba3d-4fb5-add0-88872c966ac3-tmp-dir\") pod \"auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj\" (UID: \"855770ce-ba3d-4fb5-add0-88872c966ac3\") " pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" Jul 13 22:40:00.140926 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:00.140834 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/855770ce-ba3d-4fb5-add0-88872c966ac3-tls-certs\") pod \"auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj\" (UID: \"855770ce-ba3d-4fb5-add0-88872c966ac3\") " pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" Jul 13 22:40:00.141162 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:00.140937 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/855770ce-ba3d-4fb5-add0-88872c966ac3-model-cache\") pod \"auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj\" (UID: \"855770ce-ba3d-4fb5-add0-88872c966ac3\") " pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" Jul 13 22:40:00.141162 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:00.140963 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/855770ce-ba3d-4fb5-add0-88872c966ac3-home\") pod \"auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj\" (UID: \"855770ce-ba3d-4fb5-add0-88872c966ac3\") " pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" Jul 13 22:40:00.141286 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:00.141223 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/855770ce-ba3d-4fb5-add0-88872c966ac3-tmp-dir\") pod \"auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj\" (UID: \"855770ce-ba3d-4fb5-add0-88872c966ac3\") " pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" Jul 13 22:40:00.141286 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:00.141267 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/855770ce-ba3d-4fb5-add0-88872c966ac3-model-cache\") pod \"auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj\" (UID: \"855770ce-ba3d-4fb5-add0-88872c966ac3\") " pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" Jul 13 22:40:00.141388 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:00.141331 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/855770ce-ba3d-4fb5-add0-88872c966ac3-home\") pod \"auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj\" (UID: \"855770ce-ba3d-4fb5-add0-88872c966ac3\") " pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" Jul 13 22:40:00.142995 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:00.142964 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/855770ce-ba3d-4fb5-add0-88872c966ac3-dshm\") pod \"auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj\" (UID: \"855770ce-ba3d-4fb5-add0-88872c966ac3\") " pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" Jul 13 22:40:00.143335 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:00.143315 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/855770ce-ba3d-4fb5-add0-88872c966ac3-tls-certs\") pod \"auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj\" (UID: \"855770ce-ba3d-4fb5-add0-88872c966ac3\") " pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" Jul 13 22:40:00.148651 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:00.148625 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpp6p\" (UniqueName: \"kubernetes.io/projected/855770ce-ba3d-4fb5-add0-88872c966ac3-kube-api-access-bpp6p\") pod \"auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj\" (UID: \"855770ce-ba3d-4fb5-add0-88872c966ac3\") " pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" Jul 13 22:40:00.267632 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:00.267559 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" Jul 13 22:40:00.400843 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:00.400809 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj"] Jul 13 22:40:00.404022 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:40:00.403991 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod855770ce_ba3d_4fb5_add0_88872c966ac3.slice/crio-ac9b98907b745115c2bf170cb02679a287f36bca7000a16af9f2d8f70a683295 WatchSource:0}: Error finding container ac9b98907b745115c2bf170cb02679a287f36bca7000a16af9f2d8f70a683295: Status 404 returned error can't find the container with id ac9b98907b745115c2bf170cb02679a287f36bca7000a16af9f2d8f70a683295 Jul 13 22:40:01.337086 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:01.337006 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" event={"ID":"855770ce-ba3d-4fb5-add0-88872c966ac3","Type":"ContainerStarted","Data":"0792353b61d14f150840f9d6d83ce23dbb2adcf97815869686faa23f529bb05b"} Jul 13 22:40:01.337086 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:01.337053 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" event={"ID":"855770ce-ba3d-4fb5-add0-88872c966ac3","Type":"ContainerStarted","Data":"ac9b98907b745115c2bf170cb02679a287f36bca7000a16af9f2d8f70a683295"} Jul 13 22:40:01.355693 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:01.355634 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" podStartSLOduration=2.355616517 podStartE2EDuration="2.355616517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:40:01.353560385 +0000 UTC m=+2422.804115922" watchObservedRunningTime="2026-07-13 22:40:01.355616517 +0000 UTC m=+2422.806172054" Jul 13 22:40:03.641583 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:03.641545 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:40:10.268091 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:10.268058 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" Jul 13 22:40:10.268509 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:10.268113 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" Jul 13 22:40:10.284460 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:10.284434 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" Jul 13 22:40:10.392873 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:10.392839 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" Jul 13 22:40:36.553529 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:36.553497 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:40:55.351143 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:40:55.351112 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:41:10.208231 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:41:10.208196 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:41:22.448242 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:41:22.448201 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:41:36.947177 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:41:36.947100 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:41:54.650266 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:41:54.650225 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:42:05.840992 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:05.840951 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw"] Jul 13 22:42:06.672077 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:06.672044 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj"] Jul 13 22:42:09.945047 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:09.945011 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:42:13.457775 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:13.457741 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld"] Jul 13 22:42:13.461354 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:13.461333 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" Jul 13 22:42:13.463441 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:13.463421 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-13bc0c\"/\"sub-test-model-13bc0c-kserve-self-signed-certs\"" Jul 13 22:42:13.469443 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:13.469420 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld"] Jul 13 22:42:13.586072 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:13.586030 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ec30ef7-a748-44f5-a6fd-c0197913d441-model-cache\") pod \"sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld\" (UID: \"3ec30ef7-a748-44f5-a6fd-c0197913d441\") " pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" Jul 13 22:42:13.586263 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:13.586091 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gk98\" (UniqueName: \"kubernetes.io/projected/3ec30ef7-a748-44f5-a6fd-c0197913d441-kube-api-access-8gk98\") pod \"sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld\" (UID: \"3ec30ef7-a748-44f5-a6fd-c0197913d441\") " pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" Jul 13 22:42:13.586263 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:13.586144 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ec30ef7-a748-44f5-a6fd-c0197913d441-dshm\") pod \"sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld\" (UID: \"3ec30ef7-a748-44f5-a6fd-c0197913d441\") " pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" Jul 13 22:42:13.586263 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:13.586161 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ec30ef7-a748-44f5-a6fd-c0197913d441-tmp-dir\") pod \"sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld\" (UID: \"3ec30ef7-a748-44f5-a6fd-c0197913d441\") " pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" Jul 13 22:42:13.586402 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:13.586294 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ec30ef7-a748-44f5-a6fd-c0197913d441-tls-certs\") pod \"sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld\" (UID: \"3ec30ef7-a748-44f5-a6fd-c0197913d441\") " pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" Jul 13 22:42:13.586402 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:13.586326 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ec30ef7-a748-44f5-a6fd-c0197913d441-home\") pod \"sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld\" (UID: \"3ec30ef7-a748-44f5-a6fd-c0197913d441\") " pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" Jul 13 22:42:13.687428 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:13.687382 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ec30ef7-a748-44f5-a6fd-c0197913d441-dshm\") pod \"sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld\" (UID: \"3ec30ef7-a748-44f5-a6fd-c0197913d441\") " pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" Jul 13 22:42:13.687428 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:13.687429 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ec30ef7-a748-44f5-a6fd-c0197913d441-tmp-dir\") pod \"sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld\" (UID: \"3ec30ef7-a748-44f5-a6fd-c0197913d441\") " pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" Jul 13 22:42:13.687681 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:13.687470 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ec30ef7-a748-44f5-a6fd-c0197913d441-tls-certs\") pod \"sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld\" (UID: \"3ec30ef7-a748-44f5-a6fd-c0197913d441\") " pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" Jul 13 22:42:13.687681 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:13.687499 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ec30ef7-a748-44f5-a6fd-c0197913d441-home\") pod \"sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld\" (UID: \"3ec30ef7-a748-44f5-a6fd-c0197913d441\") " pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" Jul 13 22:42:13.687681 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:13.687535 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ec30ef7-a748-44f5-a6fd-c0197913d441-model-cache\") pod \"sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld\" (UID: \"3ec30ef7-a748-44f5-a6fd-c0197913d441\") " pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" Jul 13 22:42:13.687681 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:13.687597 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gk98\" (UniqueName: \"kubernetes.io/projected/3ec30ef7-a748-44f5-a6fd-c0197913d441-kube-api-access-8gk98\") pod \"sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld\" (UID: \"3ec30ef7-a748-44f5-a6fd-c0197913d441\") " pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" Jul 13 22:42:13.687925 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:13.687893 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ec30ef7-a748-44f5-a6fd-c0197913d441-tmp-dir\") pod \"sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld\" (UID: \"3ec30ef7-a748-44f5-a6fd-c0197913d441\") " pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" Jul 13 22:42:13.688080 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:13.687927 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ec30ef7-a748-44f5-a6fd-c0197913d441-home\") pod \"sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld\" (UID: \"3ec30ef7-a748-44f5-a6fd-c0197913d441\") " pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" Jul 13 22:42:13.688080 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:13.687954 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ec30ef7-a748-44f5-a6fd-c0197913d441-model-cache\") pod \"sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld\" (UID: \"3ec30ef7-a748-44f5-a6fd-c0197913d441\") " pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" Jul 13 22:42:13.689740 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:13.689723 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ec30ef7-a748-44f5-a6fd-c0197913d441-dshm\") pod \"sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld\" (UID: \"3ec30ef7-a748-44f5-a6fd-c0197913d441\") " pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" Jul 13 22:42:13.690000 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:13.689965 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ec30ef7-a748-44f5-a6fd-c0197913d441-tls-certs\") pod \"sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld\" (UID: \"3ec30ef7-a748-44f5-a6fd-c0197913d441\") " pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" Jul 13 22:42:13.695641 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:13.695619 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gk98\" (UniqueName: \"kubernetes.io/projected/3ec30ef7-a748-44f5-a6fd-c0197913d441-kube-api-access-8gk98\") pod \"sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld\" (UID: \"3ec30ef7-a748-44f5-a6fd-c0197913d441\") " pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" Jul 13 22:42:13.770605 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:13.770516 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" Jul 13 22:42:13.903002 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:13.902961 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld"] Jul 13 22:42:13.904801 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:42:13.904772 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ec30ef7_a748_44f5_a6fd_c0197913d441.slice/crio-e5f87e92d19cbc9807f405cf322e13d92c66c98ff44dceb960b17183c9be38a7 WatchSource:0}: Error finding container e5f87e92d19cbc9807f405cf322e13d92c66c98ff44dceb960b17183c9be38a7: Status 404 returned error can't find the container with id e5f87e92d19cbc9807f405cf322e13d92c66c98ff44dceb960b17183c9be38a7 Jul 13 22:42:13.967942 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:13.967915 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" event={"ID":"3ec30ef7-a748-44f5-a6fd-c0197913d441","Type":"ContainerStarted","Data":"e5f87e92d19cbc9807f405cf322e13d92c66c98ff44dceb960b17183c9be38a7"} Jul 13 22:42:14.980031 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:14.979970 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" event={"ID":"3ec30ef7-a748-44f5-a6fd-c0197913d441","Type":"ContainerStarted","Data":"6ae3208b971afc85cb1030ee2ca0aba34fbca014c4593d25195b03c582f92a14"} Jul 13 22:42:14.996513 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:14.996462 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" podStartSLOduration=1.996448732 podStartE2EDuration="1.996448732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:42:14.995400067 +0000 UTC m=+2556.445955604" watchObservedRunningTime="2026-07-13 22:42:14.996448732 +0000 UTC m=+2556.447004269" Jul 13 22:42:20.861138 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:20.861103 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" podUID="56f4b26f-6586-4955-bea1-3759ab54b340" containerName="main" containerID="cri-o://e14cb823394fff8fd9c0dbb9ce2c14f9f75e11f60850c0a9fdea8c5204c52880" gracePeriod=45 Jul 13 22:42:21.008482 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:21.008451 2562 generic.go:356] "Generic (PLEG): container finished" podID="56f4b26f-6586-4955-bea1-3759ab54b340" containerID="e14cb823394fff8fd9c0dbb9ce2c14f9f75e11f60850c0a9fdea8c5204c52880" exitCode=0 Jul 13 22:42:21.008636 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:21.008529 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" event={"ID":"56f4b26f-6586-4955-bea1-3759ab54b340","Type":"ContainerDied","Data":"e14cb823394fff8fd9c0dbb9ce2c14f9f75e11f60850c0a9fdea8c5204c52880"} Jul 13 22:42:21.117599 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:21.117538 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" Jul 13 22:42:21.262627 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:21.262593 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56f4b26f-6586-4955-bea1-3759ab54b340-tmp-dir\") pod \"56f4b26f-6586-4955-bea1-3759ab54b340\" (UID: \"56f4b26f-6586-4955-bea1-3759ab54b340\") " Jul 13 22:42:21.262627 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:21.262633 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56f4b26f-6586-4955-bea1-3759ab54b340-dshm\") pod \"56f4b26f-6586-4955-bea1-3759ab54b340\" (UID: \"56f4b26f-6586-4955-bea1-3759ab54b340\") " Jul 13 22:42:21.262843 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:21.262718 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56f4b26f-6586-4955-bea1-3759ab54b340-model-cache\") pod \"56f4b26f-6586-4955-bea1-3759ab54b340\" (UID: \"56f4b26f-6586-4955-bea1-3759ab54b340\") " Jul 13 22:42:21.262843 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:21.262779 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56f4b26f-6586-4955-bea1-3759ab54b340-tls-certs\") pod \"56f4b26f-6586-4955-bea1-3759ab54b340\" (UID: \"56f4b26f-6586-4955-bea1-3759ab54b340\") " Jul 13 22:42:21.262843 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:21.262804 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6t97\" (UniqueName: \"kubernetes.io/projected/56f4b26f-6586-4955-bea1-3759ab54b340-kube-api-access-k6t97\") pod \"56f4b26f-6586-4955-bea1-3759ab54b340\" (UID: \"56f4b26f-6586-4955-bea1-3759ab54b340\") " Jul 13 22:42:21.262843 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:21.262829 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56f4b26f-6586-4955-bea1-3759ab54b340-home\") pod \"56f4b26f-6586-4955-bea1-3759ab54b340\" (UID: \"56f4b26f-6586-4955-bea1-3759ab54b340\") " Jul 13 22:42:21.263084 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:21.262852 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f4b26f-6586-4955-bea1-3759ab54b340-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "56f4b26f-6586-4955-bea1-3759ab54b340" (UID: "56f4b26f-6586-4955-bea1-3759ab54b340"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:42:21.263084 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:21.262992 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f4b26f-6586-4955-bea1-3759ab54b340-model-cache" (OuterVolumeSpecName: "model-cache") pod "56f4b26f-6586-4955-bea1-3759ab54b340" (UID: "56f4b26f-6586-4955-bea1-3759ab54b340"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:42:21.263200 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:21.263176 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56f4b26f-6586-4955-bea1-3759ab54b340-tmp-dir\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:42:21.263200 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:21.263200 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56f4b26f-6586-4955-bea1-3759ab54b340-model-cache\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:42:21.263280 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:21.263176 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f4b26f-6586-4955-bea1-3759ab54b340-home" (OuterVolumeSpecName: "home") pod "56f4b26f-6586-4955-bea1-3759ab54b340" (UID: "56f4b26f-6586-4955-bea1-3759ab54b340"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:42:21.264960 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:21.264936 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f4b26f-6586-4955-bea1-3759ab54b340-dshm" (OuterVolumeSpecName: "dshm") pod "56f4b26f-6586-4955-bea1-3759ab54b340" (UID: "56f4b26f-6586-4955-bea1-3759ab54b340"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:42:21.265058 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:21.265041 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f4b26f-6586-4955-bea1-3759ab54b340-kube-api-access-k6t97" (OuterVolumeSpecName: "kube-api-access-k6t97") pod "56f4b26f-6586-4955-bea1-3759ab54b340" (UID: "56f4b26f-6586-4955-bea1-3759ab54b340"). InnerVolumeSpecName "kube-api-access-k6t97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:42:21.265123 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:21.265105 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56f4b26f-6586-4955-bea1-3759ab54b340-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "56f4b26f-6586-4955-bea1-3759ab54b340" (UID: "56f4b26f-6586-4955-bea1-3759ab54b340"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:42:21.364051 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:21.364028 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56f4b26f-6586-4955-bea1-3759ab54b340-tls-certs\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:42:21.364051 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:21.364049 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6t97\" (UniqueName: \"kubernetes.io/projected/56f4b26f-6586-4955-bea1-3759ab54b340-kube-api-access-k6t97\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:42:21.364205 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:21.364059 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56f4b26f-6586-4955-bea1-3759ab54b340-home\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:42:21.364205 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:21.364067 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56f4b26f-6586-4955-bea1-3759ab54b340-dshm\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:42:21.692304 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:21.692265 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" podUID="855770ce-ba3d-4fb5-add0-88872c966ac3" containerName="main" containerID="cri-o://0792353b61d14f150840f9d6d83ce23dbb2adcf97815869686faa23f529bb05b" gracePeriod=45 Jul 13 22:42:21.940369 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:21.940333 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" Jul 13 22:42:22.014687 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.014652 2562 generic.go:356] "Generic (PLEG): container finished" podID="855770ce-ba3d-4fb5-add0-88872c966ac3" containerID="0792353b61d14f150840f9d6d83ce23dbb2adcf97815869686faa23f529bb05b" exitCode=0 Jul 13 22:42:22.014687 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.014690 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" Jul 13 22:42:22.014930 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.014731 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" event={"ID":"855770ce-ba3d-4fb5-add0-88872c966ac3","Type":"ContainerDied","Data":"0792353b61d14f150840f9d6d83ce23dbb2adcf97815869686faa23f529bb05b"} Jul 13 22:42:22.014930 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.014772 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj" event={"ID":"855770ce-ba3d-4fb5-add0-88872c966ac3","Type":"ContainerDied","Data":"ac9b98907b745115c2bf170cb02679a287f36bca7000a16af9f2d8f70a683295"} Jul 13 22:42:22.014930 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.014789 2562 scope.go:117] "RemoveContainer" containerID="0792353b61d14f150840f9d6d83ce23dbb2adcf97815869686faa23f529bb05b" Jul 13 22:42:22.016126 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.016109 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" event={"ID":"56f4b26f-6586-4955-bea1-3759ab54b340","Type":"ContainerDied","Data":"9d022d864c7d22e8e4c06baafb3739f42fc1ae730206a40768c236c9baed9852"} Jul 13 22:42:22.016231 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.016149 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw" Jul 13 22:42:22.024955 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.024827 2562 scope.go:117] "RemoveContainer" containerID="0792353b61d14f150840f9d6d83ce23dbb2adcf97815869686faa23f529bb05b" Jul 13 22:42:22.025224 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:42:22.025201 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0792353b61d14f150840f9d6d83ce23dbb2adcf97815869686faa23f529bb05b\": container with ID starting with 0792353b61d14f150840f9d6d83ce23dbb2adcf97815869686faa23f529bb05b not found: ID does not exist" containerID="0792353b61d14f150840f9d6d83ce23dbb2adcf97815869686faa23f529bb05b" Jul 13 22:42:22.025308 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.025238 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0792353b61d14f150840f9d6d83ce23dbb2adcf97815869686faa23f529bb05b"} err="failed to get container status \"0792353b61d14f150840f9d6d83ce23dbb2adcf97815869686faa23f529bb05b\": rpc error: code = NotFound desc = could not find container \"0792353b61d14f150840f9d6d83ce23dbb2adcf97815869686faa23f529bb05b\": container with ID starting with 0792353b61d14f150840f9d6d83ce23dbb2adcf97815869686faa23f529bb05b not found: ID does not exist" Jul 13 22:42:22.025308 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.025263 2562 scope.go:117] "RemoveContainer" containerID="e14cb823394fff8fd9c0dbb9ce2c14f9f75e11f60850c0a9fdea8c5204c52880" Jul 13 22:42:22.038678 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.038655 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw"] Jul 13 22:42:22.042396 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.042376 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-13bc0c/auth-test-model-13bc0c-kserve-5dc5dfcd67-drsjw"] Jul 13 22:42:22.071319 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.071298 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/855770ce-ba3d-4fb5-add0-88872c966ac3-tls-certs\") pod \"855770ce-ba3d-4fb5-add0-88872c966ac3\" (UID: \"855770ce-ba3d-4fb5-add0-88872c966ac3\") " Jul 13 22:42:22.071407 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.071343 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/855770ce-ba3d-4fb5-add0-88872c966ac3-home\") pod \"855770ce-ba3d-4fb5-add0-88872c966ac3\" (UID: \"855770ce-ba3d-4fb5-add0-88872c966ac3\") " Jul 13 22:42:22.071407 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.071360 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/855770ce-ba3d-4fb5-add0-88872c966ac3-tmp-dir\") pod \"855770ce-ba3d-4fb5-add0-88872c966ac3\" (UID: \"855770ce-ba3d-4fb5-add0-88872c966ac3\") " Jul 13 22:42:22.071407 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.071379 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/855770ce-ba3d-4fb5-add0-88872c966ac3-model-cache\") pod \"855770ce-ba3d-4fb5-add0-88872c966ac3\" (UID: \"855770ce-ba3d-4fb5-add0-88872c966ac3\") " Jul 13 22:42:22.071549 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.071422 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpp6p\" (UniqueName: \"kubernetes.io/projected/855770ce-ba3d-4fb5-add0-88872c966ac3-kube-api-access-bpp6p\") pod \"855770ce-ba3d-4fb5-add0-88872c966ac3\" (UID: \"855770ce-ba3d-4fb5-add0-88872c966ac3\") " Jul 13 22:42:22.071549 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.071460 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/855770ce-ba3d-4fb5-add0-88872c966ac3-dshm\") pod \"855770ce-ba3d-4fb5-add0-88872c966ac3\" (UID: \"855770ce-ba3d-4fb5-add0-88872c966ac3\") " Jul 13 22:42:22.071662 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.071587 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/855770ce-ba3d-4fb5-add0-88872c966ac3-home" (OuterVolumeSpecName: "home") pod "855770ce-ba3d-4fb5-add0-88872c966ac3" (UID: "855770ce-ba3d-4fb5-add0-88872c966ac3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:42:22.071662 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.071624 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/855770ce-ba3d-4fb5-add0-88872c966ac3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "855770ce-ba3d-4fb5-add0-88872c966ac3" (UID: "855770ce-ba3d-4fb5-add0-88872c966ac3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:42:22.071764 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.071719 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/855770ce-ba3d-4fb5-add0-88872c966ac3-model-cache" (OuterVolumeSpecName: "model-cache") pod "855770ce-ba3d-4fb5-add0-88872c966ac3" (UID: "855770ce-ba3d-4fb5-add0-88872c966ac3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:42:22.071849 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.071819 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/855770ce-ba3d-4fb5-add0-88872c966ac3-home\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:42:22.071849 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.071842 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/855770ce-ba3d-4fb5-add0-88872c966ac3-tmp-dir\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:42:22.071940 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.071857 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/855770ce-ba3d-4fb5-add0-88872c966ac3-model-cache\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:42:22.073563 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.073528 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/855770ce-ba3d-4fb5-add0-88872c966ac3-dshm" (OuterVolumeSpecName: "dshm") pod "855770ce-ba3d-4fb5-add0-88872c966ac3" (UID: "855770ce-ba3d-4fb5-add0-88872c966ac3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:42:22.073678 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.073657 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/855770ce-ba3d-4fb5-add0-88872c966ac3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "855770ce-ba3d-4fb5-add0-88872c966ac3" (UID: "855770ce-ba3d-4fb5-add0-88872c966ac3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:42:22.073720 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.073657 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/855770ce-ba3d-4fb5-add0-88872c966ac3-kube-api-access-bpp6p" (OuterVolumeSpecName: "kube-api-access-bpp6p") pod "855770ce-ba3d-4fb5-add0-88872c966ac3" (UID: "855770ce-ba3d-4fb5-add0-88872c966ac3"). InnerVolumeSpecName "kube-api-access-bpp6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:42:22.172891 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.172855 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/855770ce-ba3d-4fb5-add0-88872c966ac3-tls-certs\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:42:22.172891 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.172884 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpp6p\" (UniqueName: \"kubernetes.io/projected/855770ce-ba3d-4fb5-add0-88872c966ac3-kube-api-access-bpp6p\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:42:22.172891 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.172893 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/855770ce-ba3d-4fb5-add0-88872c966ac3-dshm\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:42:22.335468 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.335432 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj"] Jul 13 22:42:22.341369 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.341339 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-bdc0cb/auth-test-model-bdc0cb-kserve-7f9dfc6475-km4rj"] Jul 13 22:42:22.966481 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.966444 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56f4b26f-6586-4955-bea1-3759ab54b340" path="/var/lib/kubelet/pods/56f4b26f-6586-4955-bea1-3759ab54b340/volumes" Jul 13 22:42:22.966899 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:22.966884 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="855770ce-ba3d-4fb5-add0-88872c966ac3" path="/var/lib/kubelet/pods/855770ce-ba3d-4fb5-add0-88872c966ac3/volumes" Jul 13 22:42:23.771484 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:23.771450 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" Jul 13 22:42:23.771660 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:23.771498 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" Jul 13 22:42:23.783949 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:23.783926 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" Jul 13 22:42:24.038957 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:24.038867 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" Jul 13 22:42:26.745484 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:26.745446 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:42:54.861649 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:54.857296 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz"] Jul 13 22:42:54.861649 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:54.858154 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="855770ce-ba3d-4fb5-add0-88872c966ac3" containerName="main" Jul 13 22:42:54.861649 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:54.858174 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="855770ce-ba3d-4fb5-add0-88872c966ac3" containerName="main" Jul 13 22:42:54.861649 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:54.858235 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56f4b26f-6586-4955-bea1-3759ab54b340" containerName="main" Jul 13 22:42:54.861649 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:54.858244 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="56f4b26f-6586-4955-bea1-3759ab54b340" containerName="main" Jul 13 22:42:54.861649 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:54.858440 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="56f4b26f-6586-4955-bea1-3759ab54b340" containerName="main" Jul 13 22:42:54.861649 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:54.858458 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="855770ce-ba3d-4fb5-add0-88872c966ac3" containerName="main" Jul 13 22:42:54.862751 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:54.862727 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" Jul 13 22:42:54.865309 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:54.865291 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bdc0cb\"/\"openshift-service-ca.crt\"" Jul 13 22:42:54.865802 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:54.865785 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bdc0cb\"/\"kube-root-ca.crt\"" Jul 13 22:42:54.865908 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:54.865857 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bdc0cb\"/\"default-dockercfg-rh4zh\"" Jul 13 22:42:54.865908 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:54.865895 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bdc0cb\"/\"sub-test-model-bdc0cb-kserve-self-signed-certs\"" Jul 13 22:42:54.869789 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:54.869767 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz"] Jul 13 22:42:54.984392 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:54.984360 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60efeae2-2ba8-40e2-bc35-742b9ceab006-tls-certs\") pod \"sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz\" (UID: \"60efeae2-2ba8-40e2-bc35-742b9ceab006\") " pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" Jul 13 22:42:54.984392 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:54.984398 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60efeae2-2ba8-40e2-bc35-742b9ceab006-tmp-dir\") pod \"sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz\" (UID: \"60efeae2-2ba8-40e2-bc35-742b9ceab006\") " pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" Jul 13 22:42:54.984599 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:54.984422 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60efeae2-2ba8-40e2-bc35-742b9ceab006-home\") pod \"sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz\" (UID: \"60efeae2-2ba8-40e2-bc35-742b9ceab006\") " pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" Jul 13 22:42:54.984599 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:54.984490 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60efeae2-2ba8-40e2-bc35-742b9ceab006-dshm\") pod \"sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz\" (UID: \"60efeae2-2ba8-40e2-bc35-742b9ceab006\") " pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" Jul 13 22:42:54.984599 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:54.984521 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60efeae2-2ba8-40e2-bc35-742b9ceab006-model-cache\") pod \"sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz\" (UID: \"60efeae2-2ba8-40e2-bc35-742b9ceab006\") " pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" Jul 13 22:42:54.984599 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:54.984550 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k85m6\" (UniqueName: \"kubernetes.io/projected/60efeae2-2ba8-40e2-bc35-742b9ceab006-kube-api-access-k85m6\") pod \"sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz\" (UID: \"60efeae2-2ba8-40e2-bc35-742b9ceab006\") " pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" Jul 13 22:42:55.085767 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:55.085734 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60efeae2-2ba8-40e2-bc35-742b9ceab006-dshm\") pod \"sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz\" (UID: \"60efeae2-2ba8-40e2-bc35-742b9ceab006\") " pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" Jul 13 22:42:55.085767 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:55.085769 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60efeae2-2ba8-40e2-bc35-742b9ceab006-model-cache\") pod \"sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz\" (UID: \"60efeae2-2ba8-40e2-bc35-742b9ceab006\") " pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" Jul 13 22:42:55.086007 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:55.085801 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k85m6\" (UniqueName: \"kubernetes.io/projected/60efeae2-2ba8-40e2-bc35-742b9ceab006-kube-api-access-k85m6\") pod \"sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz\" (UID: \"60efeae2-2ba8-40e2-bc35-742b9ceab006\") " pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" Jul 13 22:42:55.086007 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:55.085859 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60efeae2-2ba8-40e2-bc35-742b9ceab006-tls-certs\") pod \"sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz\" (UID: \"60efeae2-2ba8-40e2-bc35-742b9ceab006\") " pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" Jul 13 22:42:55.086007 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:55.085880 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60efeae2-2ba8-40e2-bc35-742b9ceab006-tmp-dir\") pod \"sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz\" (UID: \"60efeae2-2ba8-40e2-bc35-742b9ceab006\") " pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" Jul 13 22:42:55.086007 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:55.085918 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60efeae2-2ba8-40e2-bc35-742b9ceab006-home\") pod \"sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz\" (UID: \"60efeae2-2ba8-40e2-bc35-742b9ceab006\") " pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" Jul 13 22:42:55.086269 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:55.086247 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60efeae2-2ba8-40e2-bc35-742b9ceab006-model-cache\") pod \"sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz\" (UID: \"60efeae2-2ba8-40e2-bc35-742b9ceab006\") " pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" Jul 13 22:42:55.086346 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:55.086304 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60efeae2-2ba8-40e2-bc35-742b9ceab006-tmp-dir\") pod \"sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz\" (UID: \"60efeae2-2ba8-40e2-bc35-742b9ceab006\") " pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" Jul 13 22:42:55.086403 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:55.086349 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60efeae2-2ba8-40e2-bc35-742b9ceab006-home\") pod \"sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz\" (UID: \"60efeae2-2ba8-40e2-bc35-742b9ceab006\") " pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" Jul 13 22:42:55.088142 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:55.088122 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60efeae2-2ba8-40e2-bc35-742b9ceab006-dshm\") pod \"sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz\" (UID: \"60efeae2-2ba8-40e2-bc35-742b9ceab006\") " pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" Jul 13 22:42:55.088318 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:55.088302 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60efeae2-2ba8-40e2-bc35-742b9ceab006-tls-certs\") pod \"sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz\" (UID: \"60efeae2-2ba8-40e2-bc35-742b9ceab006\") " pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" Jul 13 22:42:55.093268 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:55.093247 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k85m6\" (UniqueName: \"kubernetes.io/projected/60efeae2-2ba8-40e2-bc35-742b9ceab006-kube-api-access-k85m6\") pod \"sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz\" (UID: \"60efeae2-2ba8-40e2-bc35-742b9ceab006\") " pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" Jul 13 22:42:55.173363 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:55.173274 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" Jul 13 22:42:55.307792 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:55.307767 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz"] Jul 13 22:42:55.310122 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:42:55.310091 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60efeae2_2ba8_40e2_bc35_742b9ceab006.slice/crio-0f3ca65d6be1f4bb13de55e79814efd32ead10ddcb27fe33576f30b0f1180fc7 WatchSource:0}: Error finding container 0f3ca65d6be1f4bb13de55e79814efd32ead10ddcb27fe33576f30b0f1180fc7: Status 404 returned error can't find the container with id 0f3ca65d6be1f4bb13de55e79814efd32ead10ddcb27fe33576f30b0f1180fc7 Jul 13 22:42:56.179418 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:56.179380 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" event={"ID":"60efeae2-2ba8-40e2-bc35-742b9ceab006","Type":"ContainerStarted","Data":"2be58fc0c0857e8b4ce7dee601359d1a448f457806ac06b337606ec985e96fea"} Jul 13 22:42:56.179418 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:56.179423 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" event={"ID":"60efeae2-2ba8-40e2-bc35-742b9ceab006","Type":"ContainerStarted","Data":"0f3ca65d6be1f4bb13de55e79814efd32ead10ddcb27fe33576f30b0f1180fc7"} Jul 13 22:42:56.194659 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:56.194606 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" podStartSLOduration=2.194589661 podStartE2EDuration="2.194589661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:42:56.194176119 +0000 UTC m=+2597.644731646" watchObservedRunningTime="2026-07-13 22:42:56.194589661 +0000 UTC m=+2597.645145199" Jul 13 22:42:59.147739 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:42:59.147706 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:43:05.174463 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:05.174376 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" Jul 13 22:43:05.174463 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:05.174421 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" Jul 13 22:43:05.186823 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:05.186801 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" Jul 13 22:43:05.231879 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:05.231849 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" Jul 13 22:43:19.449614 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:19.449578 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:43:35.347049 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:35.347016 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:43:41.851613 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:41.851579 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld"] Jul 13 22:43:42.833615 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:42.833585 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz"] Jul 13 22:43:55.345494 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:55.345458 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:43:56.871618 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:56.871574 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" podUID="3ec30ef7-a748-44f5-a6fd-c0197913d441" containerName="main" containerID="cri-o://6ae3208b971afc85cb1030ee2ca0aba34fbca014c4593d25195b03c582f92a14" gracePeriod=45 Jul 13 22:43:57.125420 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.125353 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" Jul 13 22:43:57.152880 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.152851 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ec30ef7-a748-44f5-a6fd-c0197913d441-dshm\") pod \"3ec30ef7-a748-44f5-a6fd-c0197913d441\" (UID: \"3ec30ef7-a748-44f5-a6fd-c0197913d441\") " Jul 13 22:43:57.153087 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.152899 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ec30ef7-a748-44f5-a6fd-c0197913d441-tmp-dir\") pod \"3ec30ef7-a748-44f5-a6fd-c0197913d441\" (UID: \"3ec30ef7-a748-44f5-a6fd-c0197913d441\") " Jul 13 22:43:57.153087 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.152941 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8gk98\" (UniqueName: \"kubernetes.io/projected/3ec30ef7-a748-44f5-a6fd-c0197913d441-kube-api-access-8gk98\") pod \"3ec30ef7-a748-44f5-a6fd-c0197913d441\" (UID: \"3ec30ef7-a748-44f5-a6fd-c0197913d441\") " Jul 13 22:43:57.153087 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.153026 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ec30ef7-a748-44f5-a6fd-c0197913d441-tls-certs\") pod \"3ec30ef7-a748-44f5-a6fd-c0197913d441\" (UID: \"3ec30ef7-a748-44f5-a6fd-c0197913d441\") " Jul 13 22:43:57.153087 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.153063 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ec30ef7-a748-44f5-a6fd-c0197913d441-home\") pod \"3ec30ef7-a748-44f5-a6fd-c0197913d441\" (UID: \"3ec30ef7-a748-44f5-a6fd-c0197913d441\") " Jul 13 22:43:57.153324 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.153120 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ec30ef7-a748-44f5-a6fd-c0197913d441-model-cache\") pod \"3ec30ef7-a748-44f5-a6fd-c0197913d441\" (UID: \"3ec30ef7-a748-44f5-a6fd-c0197913d441\") " Jul 13 22:43:57.153324 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.153230 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec30ef7-a748-44f5-a6fd-c0197913d441-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3ec30ef7-a748-44f5-a6fd-c0197913d441" (UID: "3ec30ef7-a748-44f5-a6fd-c0197913d441"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:43:57.153439 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.153359 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec30ef7-a748-44f5-a6fd-c0197913d441-home" (OuterVolumeSpecName: "home") pod "3ec30ef7-a748-44f5-a6fd-c0197913d441" (UID: "3ec30ef7-a748-44f5-a6fd-c0197913d441"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:43:57.153496 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.153440 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec30ef7-a748-44f5-a6fd-c0197913d441-model-cache" (OuterVolumeSpecName: "model-cache") pod "3ec30ef7-a748-44f5-a6fd-c0197913d441" (UID: "3ec30ef7-a748-44f5-a6fd-c0197913d441"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:43:57.153556 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.153513 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ec30ef7-a748-44f5-a6fd-c0197913d441-home\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:43:57.153556 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.153532 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ec30ef7-a748-44f5-a6fd-c0197913d441-tmp-dir\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:43:57.155080 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.155042 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec30ef7-a748-44f5-a6fd-c0197913d441-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ec30ef7-a748-44f5-a6fd-c0197913d441" (UID: "3ec30ef7-a748-44f5-a6fd-c0197913d441"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:43:57.155193 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.155088 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec30ef7-a748-44f5-a6fd-c0197913d441-kube-api-access-8gk98" (OuterVolumeSpecName: "kube-api-access-8gk98") pod "3ec30ef7-a748-44f5-a6fd-c0197913d441" (UID: "3ec30ef7-a748-44f5-a6fd-c0197913d441"). InnerVolumeSpecName "kube-api-access-8gk98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:43:57.155193 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.155129 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec30ef7-a748-44f5-a6fd-c0197913d441-dshm" (OuterVolumeSpecName: "dshm") pod "3ec30ef7-a748-44f5-a6fd-c0197913d441" (UID: "3ec30ef7-a748-44f5-a6fd-c0197913d441"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:43:57.254255 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.254222 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ec30ef7-a748-44f5-a6fd-c0197913d441-dshm\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:43:57.254255 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.254254 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gk98\" (UniqueName: \"kubernetes.io/projected/3ec30ef7-a748-44f5-a6fd-c0197913d441-kube-api-access-8gk98\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:43:57.254525 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.254270 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ec30ef7-a748-44f5-a6fd-c0197913d441-tls-certs\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:43:57.254525 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.254291 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ec30ef7-a748-44f5-a6fd-c0197913d441-model-cache\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:43:57.480341 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.480311 2562 generic.go:356] "Generic (PLEG): container finished" podID="3ec30ef7-a748-44f5-a6fd-c0197913d441" containerID="6ae3208b971afc85cb1030ee2ca0aba34fbca014c4593d25195b03c582f92a14" exitCode=0 Jul 13 22:43:57.480521 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.480345 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" event={"ID":"3ec30ef7-a748-44f5-a6fd-c0197913d441","Type":"ContainerDied","Data":"6ae3208b971afc85cb1030ee2ca0aba34fbca014c4593d25195b03c582f92a14"} Jul 13 22:43:57.480521 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.480390 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" event={"ID":"3ec30ef7-a748-44f5-a6fd-c0197913d441","Type":"ContainerDied","Data":"e5f87e92d19cbc9807f405cf322e13d92c66c98ff44dceb960b17183c9be38a7"} Jul 13 22:43:57.480521 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.480406 2562 scope.go:117] "RemoveContainer" containerID="6ae3208b971afc85cb1030ee2ca0aba34fbca014c4593d25195b03c582f92a14" Jul 13 22:43:57.480521 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.480353 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld" Jul 13 22:43:57.489180 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.489165 2562 scope.go:117] "RemoveContainer" containerID="6ae3208b971afc85cb1030ee2ca0aba34fbca014c4593d25195b03c582f92a14" Jul 13 22:43:57.489453 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:43:57.489437 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ae3208b971afc85cb1030ee2ca0aba34fbca014c4593d25195b03c582f92a14\": container with ID starting with 6ae3208b971afc85cb1030ee2ca0aba34fbca014c4593d25195b03c582f92a14 not found: ID does not exist" containerID="6ae3208b971afc85cb1030ee2ca0aba34fbca014c4593d25195b03c582f92a14" Jul 13 22:43:57.489505 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.489460 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ae3208b971afc85cb1030ee2ca0aba34fbca014c4593d25195b03c582f92a14"} err="failed to get container status \"6ae3208b971afc85cb1030ee2ca0aba34fbca014c4593d25195b03c582f92a14\": rpc error: code = NotFound desc = could not find container \"6ae3208b971afc85cb1030ee2ca0aba34fbca014c4593d25195b03c582f92a14\": container with ID starting with 6ae3208b971afc85cb1030ee2ca0aba34fbca014c4593d25195b03c582f92a14 not found: ID does not exist" Jul 13 22:43:57.506049 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.502834 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld"] Jul 13 22:43:57.506889 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.506867 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-13bc0c/sub-test-model-13bc0c-kserve-6d687b7c47-tt5ld"] Jul 13 22:43:57.853716 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:57.853599 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" podUID="60efeae2-2ba8-40e2-bc35-742b9ceab006" containerName="main" containerID="cri-o://2be58fc0c0857e8b4ce7dee601359d1a448f457806ac06b337606ec985e96fea" gracePeriod=45 Jul 13 22:43:58.109438 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.109378 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" Jul 13 22:43:58.160516 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.160488 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60efeae2-2ba8-40e2-bc35-742b9ceab006-tmp-dir\") pod \"60efeae2-2ba8-40e2-bc35-742b9ceab006\" (UID: \"60efeae2-2ba8-40e2-bc35-742b9ceab006\") " Jul 13 22:43:58.160696 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.160526 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60efeae2-2ba8-40e2-bc35-742b9ceab006-tls-certs\") pod \"60efeae2-2ba8-40e2-bc35-742b9ceab006\" (UID: \"60efeae2-2ba8-40e2-bc35-742b9ceab006\") " Jul 13 22:43:58.160696 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.160583 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k85m6\" (UniqueName: \"kubernetes.io/projected/60efeae2-2ba8-40e2-bc35-742b9ceab006-kube-api-access-k85m6\") pod \"60efeae2-2ba8-40e2-bc35-742b9ceab006\" (UID: \"60efeae2-2ba8-40e2-bc35-742b9ceab006\") " Jul 13 22:43:58.160696 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.160606 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60efeae2-2ba8-40e2-bc35-742b9ceab006-home\") pod \"60efeae2-2ba8-40e2-bc35-742b9ceab006\" (UID: \"60efeae2-2ba8-40e2-bc35-742b9ceab006\") " Jul 13 22:43:58.160696 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.160631 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60efeae2-2ba8-40e2-bc35-742b9ceab006-model-cache\") pod \"60efeae2-2ba8-40e2-bc35-742b9ceab006\" (UID: \"60efeae2-2ba8-40e2-bc35-742b9ceab006\") " Jul 13 22:43:58.160915 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.160698 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60efeae2-2ba8-40e2-bc35-742b9ceab006-dshm\") pod \"60efeae2-2ba8-40e2-bc35-742b9ceab006\" (UID: \"60efeae2-2ba8-40e2-bc35-742b9ceab006\") " Jul 13 22:43:58.160915 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.160754 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60efeae2-2ba8-40e2-bc35-742b9ceab006-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "60efeae2-2ba8-40e2-bc35-742b9ceab006" (UID: "60efeae2-2ba8-40e2-bc35-742b9ceab006"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:43:58.160915 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.160894 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60efeae2-2ba8-40e2-bc35-742b9ceab006-home" (OuterVolumeSpecName: "home") pod "60efeae2-2ba8-40e2-bc35-742b9ceab006" (UID: "60efeae2-2ba8-40e2-bc35-742b9ceab006"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:43:58.161092 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.160939 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60efeae2-2ba8-40e2-bc35-742b9ceab006-model-cache" (OuterVolumeSpecName: "model-cache") pod "60efeae2-2ba8-40e2-bc35-742b9ceab006" (UID: "60efeae2-2ba8-40e2-bc35-742b9ceab006"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:43:58.161092 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.161054 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/60efeae2-2ba8-40e2-bc35-742b9ceab006-home\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:43:58.161203 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.161092 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/60efeae2-2ba8-40e2-bc35-742b9ceab006-model-cache\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:43:58.161203 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.161108 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/60efeae2-2ba8-40e2-bc35-742b9ceab006-tmp-dir\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:43:58.162799 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.162774 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60efeae2-2ba8-40e2-bc35-742b9ceab006-kube-api-access-k85m6" (OuterVolumeSpecName: "kube-api-access-k85m6") pod "60efeae2-2ba8-40e2-bc35-742b9ceab006" (UID: "60efeae2-2ba8-40e2-bc35-742b9ceab006"). InnerVolumeSpecName "kube-api-access-k85m6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:43:58.162922 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.162805 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60efeae2-2ba8-40e2-bc35-742b9ceab006-dshm" (OuterVolumeSpecName: "dshm") pod "60efeae2-2ba8-40e2-bc35-742b9ceab006" (UID: "60efeae2-2ba8-40e2-bc35-742b9ceab006"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:43:58.162922 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.162857 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60efeae2-2ba8-40e2-bc35-742b9ceab006-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "60efeae2-2ba8-40e2-bc35-742b9ceab006" (UID: "60efeae2-2ba8-40e2-bc35-742b9ceab006"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:43:58.261893 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.261859 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k85m6\" (UniqueName: \"kubernetes.io/projected/60efeae2-2ba8-40e2-bc35-742b9ceab006-kube-api-access-k85m6\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:43:58.261893 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.261887 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/60efeae2-2ba8-40e2-bc35-742b9ceab006-dshm\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:43:58.261893 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.261897 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/60efeae2-2ba8-40e2-bc35-742b9ceab006-tls-certs\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:43:58.486478 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.486449 2562 generic.go:356] "Generic (PLEG): container finished" podID="60efeae2-2ba8-40e2-bc35-742b9ceab006" containerID="2be58fc0c0857e8b4ce7dee601359d1a448f457806ac06b337606ec985e96fea" exitCode=0 Jul 13 22:43:58.486667 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.486485 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" Jul 13 22:43:58.486667 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.486544 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" event={"ID":"60efeae2-2ba8-40e2-bc35-742b9ceab006","Type":"ContainerDied","Data":"2be58fc0c0857e8b4ce7dee601359d1a448f457806ac06b337606ec985e96fea"} Jul 13 22:43:58.486667 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.486585 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz" event={"ID":"60efeae2-2ba8-40e2-bc35-742b9ceab006","Type":"ContainerDied","Data":"0f3ca65d6be1f4bb13de55e79814efd32ead10ddcb27fe33576f30b0f1180fc7"} Jul 13 22:43:58.486667 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.486605 2562 scope.go:117] "RemoveContainer" containerID="2be58fc0c0857e8b4ce7dee601359d1a448f457806ac06b337606ec985e96fea" Jul 13 22:43:58.495558 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.495542 2562 scope.go:117] "RemoveContainer" containerID="2be58fc0c0857e8b4ce7dee601359d1a448f457806ac06b337606ec985e96fea" Jul 13 22:43:58.495791 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:43:58.495771 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2be58fc0c0857e8b4ce7dee601359d1a448f457806ac06b337606ec985e96fea\": container with ID starting with 2be58fc0c0857e8b4ce7dee601359d1a448f457806ac06b337606ec985e96fea not found: ID does not exist" containerID="2be58fc0c0857e8b4ce7dee601359d1a448f457806ac06b337606ec985e96fea" Jul 13 22:43:58.495832 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.495799 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be58fc0c0857e8b4ce7dee601359d1a448f457806ac06b337606ec985e96fea"} err="failed to get container status \"2be58fc0c0857e8b4ce7dee601359d1a448f457806ac06b337606ec985e96fea\": rpc error: code = NotFound desc = could not find container \"2be58fc0c0857e8b4ce7dee601359d1a448f457806ac06b337606ec985e96fea\": container with ID starting with 2be58fc0c0857e8b4ce7dee601359d1a448f457806ac06b337606ec985e96fea not found: ID does not exist" Jul 13 22:43:58.508641 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.508622 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz"] Jul 13 22:43:58.511777 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.511759 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-bdc0cb/sub-test-model-bdc0cb-kserve-767bd9cbd7-8dsvz"] Jul 13 22:43:58.966561 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.966529 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ec30ef7-a748-44f5-a6fd-c0197913d441" path="/var/lib/kubelet/pods/3ec30ef7-a748-44f5-a6fd-c0197913d441/volumes" Jul 13 22:43:58.966932 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:43:58.966919 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60efeae2-2ba8-40e2-bc35-742b9ceab006" path="/var/lib/kubelet/pods/60efeae2-2ba8-40e2-bc35-742b9ceab006/volumes" Jul 13 22:44:07.736948 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.736915 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s"] Jul 13 22:44:07.737585 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.737298 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60efeae2-2ba8-40e2-bc35-742b9ceab006" containerName="main" Jul 13 22:44:07.737585 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.737311 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="60efeae2-2ba8-40e2-bc35-742b9ceab006" containerName="main" Jul 13 22:44:07.737585 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.737332 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ec30ef7-a748-44f5-a6fd-c0197913d441" containerName="main" Jul 13 22:44:07.737585 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.737338 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ec30ef7-a748-44f5-a6fd-c0197913d441" containerName="main" Jul 13 22:44:07.737585 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.737398 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="60efeae2-2ba8-40e2-bc35-742b9ceab006" containerName="main" Jul 13 22:44:07.737585 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.737408 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ec30ef7-a748-44f5-a6fd-c0197913d441" containerName="main" Jul 13 22:44:07.741785 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.741768 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" Jul 13 22:44:07.744680 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.744646 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bdc0cb\"/\"rate-test-model-bdc0cb-kserve-self-signed-certs\"" Jul 13 22:44:07.744680 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.744661 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bdc0cb\"/\"openshift-service-ca.crt\"" Jul 13 22:44:07.744865 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.744709 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bdc0cb\"/\"kube-root-ca.crt\"" Jul 13 22:44:07.744865 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.744719 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bdc0cb\"/\"default-dockercfg-rh4zh\"" Jul 13 22:44:07.748783 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.748759 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s"] Jul 13 22:44:07.848699 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.848666 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xftwx\" (UniqueName: \"kubernetes.io/projected/f62ffbaf-e725-411a-a90d-5b48d70150fa-kube-api-access-xftwx\") pod \"rate-test-model-bdc0cb-kserve-76df79cbbc-st29s\" (UID: \"f62ffbaf-e725-411a-a90d-5b48d70150fa\") " pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" Jul 13 22:44:07.848856 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.848709 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62ffbaf-e725-411a-a90d-5b48d70150fa-tmp-dir\") pod \"rate-test-model-bdc0cb-kserve-76df79cbbc-st29s\" (UID: \"f62ffbaf-e725-411a-a90d-5b48d70150fa\") " pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" Jul 13 22:44:07.848856 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.848776 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62ffbaf-e725-411a-a90d-5b48d70150fa-dshm\") pod \"rate-test-model-bdc0cb-kserve-76df79cbbc-st29s\" (UID: \"f62ffbaf-e725-411a-a90d-5b48d70150fa\") " pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" Jul 13 22:44:07.848856 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.848805 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62ffbaf-e725-411a-a90d-5b48d70150fa-model-cache\") pod \"rate-test-model-bdc0cb-kserve-76df79cbbc-st29s\" (UID: \"f62ffbaf-e725-411a-a90d-5b48d70150fa\") " pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" Jul 13 22:44:07.848856 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.848824 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62ffbaf-e725-411a-a90d-5b48d70150fa-tls-certs\") pod \"rate-test-model-bdc0cb-kserve-76df79cbbc-st29s\" (UID: \"f62ffbaf-e725-411a-a90d-5b48d70150fa\") " pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" Jul 13 22:44:07.849028 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.848865 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62ffbaf-e725-411a-a90d-5b48d70150fa-home\") pod \"rate-test-model-bdc0cb-kserve-76df79cbbc-st29s\" (UID: \"f62ffbaf-e725-411a-a90d-5b48d70150fa\") " pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" Jul 13 22:44:07.949999 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.949940 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62ffbaf-e725-411a-a90d-5b48d70150fa-dshm\") pod \"rate-test-model-bdc0cb-kserve-76df79cbbc-st29s\" (UID: \"f62ffbaf-e725-411a-a90d-5b48d70150fa\") " pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" Jul 13 22:44:07.950167 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.950007 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62ffbaf-e725-411a-a90d-5b48d70150fa-model-cache\") pod \"rate-test-model-bdc0cb-kserve-76df79cbbc-st29s\" (UID: \"f62ffbaf-e725-411a-a90d-5b48d70150fa\") " pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" Jul 13 22:44:07.950167 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.950027 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62ffbaf-e725-411a-a90d-5b48d70150fa-tls-certs\") pod \"rate-test-model-bdc0cb-kserve-76df79cbbc-st29s\" (UID: \"f62ffbaf-e725-411a-a90d-5b48d70150fa\") " pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" Jul 13 22:44:07.950167 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.950059 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62ffbaf-e725-411a-a90d-5b48d70150fa-home\") pod \"rate-test-model-bdc0cb-kserve-76df79cbbc-st29s\" (UID: \"f62ffbaf-e725-411a-a90d-5b48d70150fa\") " pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" Jul 13 22:44:07.950167 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.950135 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xftwx\" (UniqueName: \"kubernetes.io/projected/f62ffbaf-e725-411a-a90d-5b48d70150fa-kube-api-access-xftwx\") pod \"rate-test-model-bdc0cb-kserve-76df79cbbc-st29s\" (UID: \"f62ffbaf-e725-411a-a90d-5b48d70150fa\") " pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" Jul 13 22:44:07.950167 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.950168 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62ffbaf-e725-411a-a90d-5b48d70150fa-tmp-dir\") pod \"rate-test-model-bdc0cb-kserve-76df79cbbc-st29s\" (UID: \"f62ffbaf-e725-411a-a90d-5b48d70150fa\") " pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" Jul 13 22:44:07.950497 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.950475 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62ffbaf-e725-411a-a90d-5b48d70150fa-model-cache\") pod \"rate-test-model-bdc0cb-kserve-76df79cbbc-st29s\" (UID: \"f62ffbaf-e725-411a-a90d-5b48d70150fa\") " pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" Jul 13 22:44:07.950581 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.950528 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62ffbaf-e725-411a-a90d-5b48d70150fa-tmp-dir\") pod \"rate-test-model-bdc0cb-kserve-76df79cbbc-st29s\" (UID: \"f62ffbaf-e725-411a-a90d-5b48d70150fa\") " pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" Jul 13 22:44:07.950581 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.950555 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62ffbaf-e725-411a-a90d-5b48d70150fa-home\") pod \"rate-test-model-bdc0cb-kserve-76df79cbbc-st29s\" (UID: \"f62ffbaf-e725-411a-a90d-5b48d70150fa\") " pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" Jul 13 22:44:07.952390 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.952365 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62ffbaf-e725-411a-a90d-5b48d70150fa-dshm\") pod \"rate-test-model-bdc0cb-kserve-76df79cbbc-st29s\" (UID: \"f62ffbaf-e725-411a-a90d-5b48d70150fa\") " pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" Jul 13 22:44:07.952499 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.952482 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62ffbaf-e725-411a-a90d-5b48d70150fa-tls-certs\") pod \"rate-test-model-bdc0cb-kserve-76df79cbbc-st29s\" (UID: \"f62ffbaf-e725-411a-a90d-5b48d70150fa\") " pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" Jul 13 22:44:07.957376 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:07.957359 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xftwx\" (UniqueName: \"kubernetes.io/projected/f62ffbaf-e725-411a-a90d-5b48d70150fa-kube-api-access-xftwx\") pod \"rate-test-model-bdc0cb-kserve-76df79cbbc-st29s\" (UID: \"f62ffbaf-e725-411a-a90d-5b48d70150fa\") " pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" Jul 13 22:44:08.052017 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:08.051898 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" Jul 13 22:44:08.185231 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:08.185199 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s"] Jul 13 22:44:08.188661 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:44:08.188634 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf62ffbaf_e725_411a_a90d_5b48d70150fa.slice/crio-f371b6f40a08b0810219cc6a5ccadd7912295c963a87d20065b4b65ddc3f0f09 WatchSource:0}: Error finding container f371b6f40a08b0810219cc6a5ccadd7912295c963a87d20065b4b65ddc3f0f09: Status 404 returned error can't find the container with id f371b6f40a08b0810219cc6a5ccadd7912295c963a87d20065b4b65ddc3f0f09 Jul 13 22:44:08.537406 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:08.537371 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" event={"ID":"f62ffbaf-e725-411a-a90d-5b48d70150fa","Type":"ContainerStarted","Data":"d3cd1e9a90f0c47371cbe6b3eec7bbeec090243fe196e03e05f205358d69cc5b"} Jul 13 22:44:08.537406 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:08.537412 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" event={"ID":"f62ffbaf-e725-411a-a90d-5b48d70150fa","Type":"ContainerStarted","Data":"f371b6f40a08b0810219cc6a5ccadd7912295c963a87d20065b4b65ddc3f0f09"} Jul 13 22:44:08.553605 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:08.553548 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" podStartSLOduration=1.553528646 podStartE2EDuration="1.553528646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-13 22:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:44:08.5527747 +0000 UTC m=+2670.003330238" watchObservedRunningTime="2026-07-13 22:44:08.553528646 +0000 UTC m=+2670.004084184" Jul 13 22:44:14.942405 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:14.942370 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:44:18.052577 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:18.052538 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" Jul 13 22:44:18.053065 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:18.052599 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" Jul 13 22:44:18.065003 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:18.064963 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" Jul 13 22:44:18.598678 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:18.598651 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" Jul 13 22:44:35.844767 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:35.844684 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:44:39.100528 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:39.100493 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x5qhq_7b1fa75b-f29d-45a1-af98-fa947c2058b2/console-operator/1.log" Jul 13 22:44:39.108007 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:39.107962 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw4h7_582b8d35-ee74-490e-aad6-e89ac36cb61c/ovn-acl-logging/0.log" Jul 13 22:44:39.110439 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:39.110417 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x5qhq_7b1fa75b-f29d-45a1-af98-fa947c2058b2/console-operator/1.log" Jul 13 22:44:39.117348 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:39.117327 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zw4h7_582b8d35-ee74-490e-aad6-e89ac36cb61c/ovn-acl-logging/0.log" Jul 13 22:44:51.141051 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:51.141016 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:44:57.426601 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:44:57.426562 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s"] Jul 13 22:45:03.853650 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:03.853611 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:45:12.447731 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:12.447653 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" podUID="f62ffbaf-e725-411a-a90d-5b48d70150fa" containerName="main" containerID="cri-o://d3cd1e9a90f0c47371cbe6b3eec7bbeec090243fe196e03e05f205358d69cc5b" gracePeriod=45 Jul 13 22:45:12.693794 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:12.693771 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" Jul 13 22:45:12.852043 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:12.851954 2562 generic.go:356] "Generic (PLEG): container finished" podID="f62ffbaf-e725-411a-a90d-5b48d70150fa" containerID="d3cd1e9a90f0c47371cbe6b3eec7bbeec090243fe196e03e05f205358d69cc5b" exitCode=0 Jul 13 22:45:12.852043 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:12.852020 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" Jul 13 22:45:12.852244 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:12.852044 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" event={"ID":"f62ffbaf-e725-411a-a90d-5b48d70150fa","Type":"ContainerDied","Data":"d3cd1e9a90f0c47371cbe6b3eec7bbeec090243fe196e03e05f205358d69cc5b"} Jul 13 22:45:12.852244 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:12.852088 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s" event={"ID":"f62ffbaf-e725-411a-a90d-5b48d70150fa","Type":"ContainerDied","Data":"f371b6f40a08b0810219cc6a5ccadd7912295c963a87d20065b4b65ddc3f0f09"} Jul 13 22:45:12.852244 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:12.852110 2562 scope.go:117] "RemoveContainer" containerID="d3cd1e9a90f0c47371cbe6b3eec7bbeec090243fe196e03e05f205358d69cc5b" Jul 13 22:45:12.858392 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:12.858371 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62ffbaf-e725-411a-a90d-5b48d70150fa-model-cache\") pod \"f62ffbaf-e725-411a-a90d-5b48d70150fa\" (UID: \"f62ffbaf-e725-411a-a90d-5b48d70150fa\") " Jul 13 22:45:12.858523 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:12.858415 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62ffbaf-e725-411a-a90d-5b48d70150fa-home\") pod \"f62ffbaf-e725-411a-a90d-5b48d70150fa\" (UID: \"f62ffbaf-e725-411a-a90d-5b48d70150fa\") " Jul 13 22:45:12.858523 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:12.858437 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62ffbaf-e725-411a-a90d-5b48d70150fa-dshm\") pod \"f62ffbaf-e725-411a-a90d-5b48d70150fa\" (UID: \"f62ffbaf-e725-411a-a90d-5b48d70150fa\") " Jul 13 22:45:12.858523 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:12.858483 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62ffbaf-e725-411a-a90d-5b48d70150fa-tls-certs\") pod \"f62ffbaf-e725-411a-a90d-5b48d70150fa\" (UID: \"f62ffbaf-e725-411a-a90d-5b48d70150fa\") " Jul 13 22:45:12.858683 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:12.858532 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xftwx\" (UniqueName: \"kubernetes.io/projected/f62ffbaf-e725-411a-a90d-5b48d70150fa-kube-api-access-xftwx\") pod \"f62ffbaf-e725-411a-a90d-5b48d70150fa\" (UID: \"f62ffbaf-e725-411a-a90d-5b48d70150fa\") " Jul 13 22:45:12.858683 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:12.858579 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62ffbaf-e725-411a-a90d-5b48d70150fa-tmp-dir\") pod \"f62ffbaf-e725-411a-a90d-5b48d70150fa\" (UID: \"f62ffbaf-e725-411a-a90d-5b48d70150fa\") " Jul 13 22:45:12.858781 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:12.858682 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62ffbaf-e725-411a-a90d-5b48d70150fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "f62ffbaf-e725-411a-a90d-5b48d70150fa" (UID: "f62ffbaf-e725-411a-a90d-5b48d70150fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:45:12.858867 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:12.858690 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62ffbaf-e725-411a-a90d-5b48d70150fa-home" (OuterVolumeSpecName: "home") pod "f62ffbaf-e725-411a-a90d-5b48d70150fa" (UID: "f62ffbaf-e725-411a-a90d-5b48d70150fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:45:12.858925 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:12.858904 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62ffbaf-e725-411a-a90d-5b48d70150fa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f62ffbaf-e725-411a-a90d-5b48d70150fa" (UID: "f62ffbaf-e725-411a-a90d-5b48d70150fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:45:12.859028 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:12.859010 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62ffbaf-e725-411a-a90d-5b48d70150fa-tmp-dir\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:45:12.859091 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:12.859034 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62ffbaf-e725-411a-a90d-5b48d70150fa-model-cache\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:45:12.859091 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:12.859046 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62ffbaf-e725-411a-a90d-5b48d70150fa-home\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:45:12.860679 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:12.860659 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62ffbaf-e725-411a-a90d-5b48d70150fa-dshm" (OuterVolumeSpecName: "dshm") pod "f62ffbaf-e725-411a-a90d-5b48d70150fa" (UID: "f62ffbaf-e725-411a-a90d-5b48d70150fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 13 22:45:12.860754 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:12.860695 2562 scope.go:117] "RemoveContainer" containerID="d3cd1e9a90f0c47371cbe6b3eec7bbeec090243fe196e03e05f205358d69cc5b" Jul 13 22:45:12.861020 ip-10-0-132-128 kubenswrapper[2562]: E0713 22:45:12.860995 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3cd1e9a90f0c47371cbe6b3eec7bbeec090243fe196e03e05f205358d69cc5b\": container with ID starting with d3cd1e9a90f0c47371cbe6b3eec7bbeec090243fe196e03e05f205358d69cc5b not found: ID does not exist" containerID="d3cd1e9a90f0c47371cbe6b3eec7bbeec090243fe196e03e05f205358d69cc5b" Jul 13 22:45:12.861095 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:12.861034 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3cd1e9a90f0c47371cbe6b3eec7bbeec090243fe196e03e05f205358d69cc5b"} err="failed to get container status \"d3cd1e9a90f0c47371cbe6b3eec7bbeec090243fe196e03e05f205358d69cc5b\": rpc error: code = NotFound desc = could not find container \"d3cd1e9a90f0c47371cbe6b3eec7bbeec090243fe196e03e05f205358d69cc5b\": container with ID starting with d3cd1e9a90f0c47371cbe6b3eec7bbeec090243fe196e03e05f205358d69cc5b not found: ID does not exist" Jul 13 22:45:12.861095 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:12.861039 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f62ffbaf-e725-411a-a90d-5b48d70150fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f62ffbaf-e725-411a-a90d-5b48d70150fa" (UID: "f62ffbaf-e725-411a-a90d-5b48d70150fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 13 22:45:12.861183 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:12.861129 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f62ffbaf-e725-411a-a90d-5b48d70150fa-kube-api-access-xftwx" (OuterVolumeSpecName: "kube-api-access-xftwx") pod "f62ffbaf-e725-411a-a90d-5b48d70150fa" (UID: "f62ffbaf-e725-411a-a90d-5b48d70150fa"). InnerVolumeSpecName "kube-api-access-xftwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 13 22:45:12.960429 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:12.960398 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62ffbaf-e725-411a-a90d-5b48d70150fa-dshm\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:45:12.960429 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:12.960432 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62ffbaf-e725-411a-a90d-5b48d70150fa-tls-certs\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:45:12.960633 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:12.960446 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xftwx\" (UniqueName: \"kubernetes.io/projected/f62ffbaf-e725-411a-a90d-5b48d70150fa-kube-api-access-xftwx\") on node \"ip-10-0-132-128.ec2.internal\" DevicePath \"\"" Jul 13 22:45:13.166946 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:13.166874 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s"] Jul 13 22:45:13.170591 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:13.170566 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-bdc0cb/rate-test-model-bdc0cb-kserve-76df79cbbc-st29s"] Jul 13 22:45:14.966634 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:14.966599 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f62ffbaf-e725-411a-a90d-5b48d70150fa" path="/var/lib/kubelet/pods/f62ffbaf-e725-411a-a90d-5b48d70150fa/volumes" Jul 13 22:45:30.748893 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:30.748851 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:45:56.747991 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:45:56.747938 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-gpfx4"] Jul 13 22:46:14.883822 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:14.883740 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67ddb65fc-tsrpc_61805a0f-31ae-4a71-954e-be2de5d6a38b/copy-manifests/0.log" Jul 13 22:46:14.889776 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:14.889754 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67ddb65fc-tsrpc_61805a0f-31ae-4a71-954e-be2de5d6a38b/manager/0.log" Jul 13 22:46:18.695672 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:18.695644 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-5nnfr_2c7f9792-825d-464c-ab40-16590978fd9c/kuadrant-console-plugin/0.log" Jul 13 22:46:19.047546 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:19.047460 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-gpfx4_41182abb-5070-47d1-9920-52287ac3f65c/limitador/0.log" Jul 13 22:46:19.734099 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:19.734071 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bf5d9c687-mx6hg_fd72b8d2-795c-4eb0-8086-6a685215395a/kube-auth-proxy/0.log" Jul 13 22:46:20.303140 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:20.303108 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5dbcf998b4-7qqv9_dbedfa07-690d-4c79-92d3-dc85b88124d7/router/0.log" Jul 13 22:46:20.746767 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:20.746736 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-2fq6l_35a2b15a-247b-4d8c-9114-49cd5a7f9174/main/0.log" Jul 13 22:46:21.075758 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:21.075673 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-zm8fh_ca9b0caf-6078-4867-9d51-46cfd78b72a1/main/0.log" Jul 13 22:46:21.185961 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:21.185936 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-k5x24_2e7ab012-158d-4cc4-a435-8a58c5769268/main/0.log" Jul 13 22:46:28.351159 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:28.351112 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zkrhd_09fdf756-6037-48d0-a284-719f7770ade3/global-pull-secret-syncer/0.log" Jul 13 22:46:28.429102 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:28.429072 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pz6k2_64c6ea85-17e8-4f97-b043-5da490a75040/konnectivity-agent/0.log" Jul 13 22:46:28.489765 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:28.489733 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-128.ec2.internal_fad131b74fcb56c348327d4dd10f9830/haproxy/0.log" Jul 13 22:46:34.446864 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:34.446835 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-5nnfr_2c7f9792-825d-464c-ab40-16590978fd9c/kuadrant-console-plugin/0.log" Jul 13 22:46:34.699649 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:34.699621 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-gpfx4_41182abb-5070-47d1-9920-52287ac3f65c/limitador/0.log" Jul 13 22:46:36.407630 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:36.407601 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fd8be815-5e2f-4987-ab66-f8a4d663ae49/alertmanager/0.log" Jul 13 22:46:36.429843 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:36.429820 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fd8be815-5e2f-4987-ab66-f8a4d663ae49/config-reloader/0.log" Jul 13 22:46:36.451706 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:36.451683 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fd8be815-5e2f-4987-ab66-f8a4d663ae49/kube-rbac-proxy-web/0.log" Jul 13 22:46:36.471223 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:36.471201 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fd8be815-5e2f-4987-ab66-f8a4d663ae49/kube-rbac-proxy/0.log" Jul 13 22:46:36.490242 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:36.490223 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fd8be815-5e2f-4987-ab66-f8a4d663ae49/kube-rbac-proxy-metric/0.log" Jul 13 22:46:36.509697 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:36.509682 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fd8be815-5e2f-4987-ab66-f8a4d663ae49/prom-label-proxy/0.log" Jul 13 22:46:36.530859 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:36.530839 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fd8be815-5e2f-4987-ab66-f8a4d663ae49/init-config-reloader/0.log" Jul 13 22:46:36.564388 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:36.564319 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-dlj2d_e9588e99-3cae-44f7-82b6-101ae1fbd604/cluster-monitoring-operator/0.log" Jul 13 22:46:36.667694 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:36.667660 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-nlnfs_90958798-39c1-49ab-9d43-e1ca66a4e00a/monitoring-plugin/0.log" Jul 13 22:46:36.869639 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:36.869558 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cxxqb_fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8/node-exporter/0.log" Jul 13 22:46:36.886714 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:36.886689 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cxxqb_fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8/kube-rbac-proxy/0.log" Jul 13 22:46:36.906932 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:36.906905 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cxxqb_fc07e6f6-e23b-4c1d-8ad5-56c86e3bcca8/init-textfile/0.log" Jul 13 22:46:37.009414 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:37.009389 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6ebde4e7-796f-4120-ba84-1a0b516a8d1e/prometheus/0.log" Jul 13 22:46:37.040459 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:37.040433 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6ebde4e7-796f-4120-ba84-1a0b516a8d1e/config-reloader/0.log" Jul 13 22:46:37.078689 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:37.078667 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6ebde4e7-796f-4120-ba84-1a0b516a8d1e/thanos-sidecar/0.log" Jul 13 22:46:37.106402 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:37.106382 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6ebde4e7-796f-4120-ba84-1a0b516a8d1e/kube-rbac-proxy-web/0.log" Jul 13 22:46:37.145279 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:37.145216 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6ebde4e7-796f-4120-ba84-1a0b516a8d1e/kube-rbac-proxy/0.log" Jul 13 22:46:37.164477 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:37.164457 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6ebde4e7-796f-4120-ba84-1a0b516a8d1e/kube-rbac-proxy-thanos/0.log" Jul 13 22:46:37.185142 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:37.185125 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6ebde4e7-796f-4120-ba84-1a0b516a8d1e/init-config-reloader/0.log" Jul 13 22:46:37.214124 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:37.214097 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-bbnk4_fa76e2d1-de61-45ff-824d-38d30db04a95/prometheus-operator/0.log" Jul 13 22:46:37.230747 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:37.230726 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-bbnk4_fa76e2d1-de61-45ff-824d-38d30db04a95/kube-rbac-proxy/0.log" Jul 13 22:46:37.253036 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:37.253009 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-gdkcd_623254a5-bbfa-4d83-bb75-85a15054f2cd/prometheus-operator-admission-webhook/0.log" Jul 13 22:46:38.239685 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:38.239655 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5tksb/perf-node-gather-daemonset-fhh55"] Jul 13 22:46:38.240092 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:38.240030 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f62ffbaf-e725-411a-a90d-5b48d70150fa" containerName="main" Jul 13 22:46:38.240092 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:38.240043 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f62ffbaf-e725-411a-a90d-5b48d70150fa" containerName="main" Jul 13 22:46:38.240166 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:38.240112 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f62ffbaf-e725-411a-a90d-5b48d70150fa" containerName="main" Jul 13 22:46:38.243496 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:38.243472 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-fhh55" Jul 13 22:46:38.245984 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:38.245952 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5tksb\"/\"default-dockercfg-b7v8x\"" Jul 13 22:46:38.246105 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:38.245957 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5tksb\"/\"kube-root-ca.crt\"" Jul 13 22:46:38.246521 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:38.246496 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5tksb\"/\"openshift-service-ca.crt\"" Jul 13 22:46:38.255710 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:38.255690 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5tksb/perf-node-gather-daemonset-fhh55"] Jul 13 22:46:38.315594 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:38.315565 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/19526d78-70f9-468d-b31c-b37b1fc6b86f-podres\") pod \"perf-node-gather-daemonset-fhh55\" (UID: \"19526d78-70f9-468d-b31c-b37b1fc6b86f\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-fhh55" Jul 13 22:46:38.315737 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:38.315647 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19526d78-70f9-468d-b31c-b37b1fc6b86f-lib-modules\") pod \"perf-node-gather-daemonset-fhh55\" (UID: \"19526d78-70f9-468d-b31c-b37b1fc6b86f\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-fhh55" Jul 13 22:46:38.315737 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:38.315716 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19526d78-70f9-468d-b31c-b37b1fc6b86f-sys\") pod \"perf-node-gather-daemonset-fhh55\" (UID: \"19526d78-70f9-468d-b31c-b37b1fc6b86f\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-fhh55" Jul 13 22:46:38.315812 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:38.315741 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdh8z\" (UniqueName: \"kubernetes.io/projected/19526d78-70f9-468d-b31c-b37b1fc6b86f-kube-api-access-tdh8z\") pod \"perf-node-gather-daemonset-fhh55\" (UID: \"19526d78-70f9-468d-b31c-b37b1fc6b86f\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-fhh55" Jul 13 22:46:38.315812 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:38.315799 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/19526d78-70f9-468d-b31c-b37b1fc6b86f-proc\") pod \"perf-node-gather-daemonset-fhh55\" (UID: \"19526d78-70f9-468d-b31c-b37b1fc6b86f\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-fhh55" Jul 13 22:46:38.417019 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:38.416961 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19526d78-70f9-468d-b31c-b37b1fc6b86f-lib-modules\") pod \"perf-node-gather-daemonset-fhh55\" (UID: \"19526d78-70f9-468d-b31c-b37b1fc6b86f\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-fhh55" Jul 13 22:46:38.417170 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:38.417145 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19526d78-70f9-468d-b31c-b37b1fc6b86f-sys\") pod \"perf-node-gather-daemonset-fhh55\" (UID: \"19526d78-70f9-468d-b31c-b37b1fc6b86f\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-fhh55" Jul 13 22:46:38.417213 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:38.417165 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19526d78-70f9-468d-b31c-b37b1fc6b86f-lib-modules\") pod \"perf-node-gather-daemonset-fhh55\" (UID: \"19526d78-70f9-468d-b31c-b37b1fc6b86f\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-fhh55" Jul 13 22:46:38.417213 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:38.417203 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdh8z\" (UniqueName: \"kubernetes.io/projected/19526d78-70f9-468d-b31c-b37b1fc6b86f-kube-api-access-tdh8z\") pod \"perf-node-gather-daemonset-fhh55\" (UID: \"19526d78-70f9-468d-b31c-b37b1fc6b86f\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-fhh55" Jul 13 22:46:38.417280 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:38.417228 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19526d78-70f9-468d-b31c-b37b1fc6b86f-sys\") pod \"perf-node-gather-daemonset-fhh55\" (UID: \"19526d78-70f9-468d-b31c-b37b1fc6b86f\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-fhh55" Jul 13 22:46:38.417280 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:38.417239 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/19526d78-70f9-468d-b31c-b37b1fc6b86f-proc\") pod \"perf-node-gather-daemonset-fhh55\" (UID: \"19526d78-70f9-468d-b31c-b37b1fc6b86f\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-fhh55" Jul 13 22:46:38.417354 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:38.417312 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/19526d78-70f9-468d-b31c-b37b1fc6b86f-podres\") pod \"perf-node-gather-daemonset-fhh55\" (UID: \"19526d78-70f9-468d-b31c-b37b1fc6b86f\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-fhh55" Jul 13 22:46:38.417393 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:38.417358 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/19526d78-70f9-468d-b31c-b37b1fc6b86f-proc\") pod \"perf-node-gather-daemonset-fhh55\" (UID: \"19526d78-70f9-468d-b31c-b37b1fc6b86f\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-fhh55" Jul 13 22:46:38.417485 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:38.417469 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/19526d78-70f9-468d-b31c-b37b1fc6b86f-podres\") pod \"perf-node-gather-daemonset-fhh55\" (UID: \"19526d78-70f9-468d-b31c-b37b1fc6b86f\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-fhh55" Jul 13 22:46:38.425782 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:38.425761 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdh8z\" (UniqueName: \"kubernetes.io/projected/19526d78-70f9-468d-b31c-b37b1fc6b86f-kube-api-access-tdh8z\") pod \"perf-node-gather-daemonset-fhh55\" (UID: \"19526d78-70f9-468d-b31c-b37b1fc6b86f\") " pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-fhh55" Jul 13 22:46:38.554058 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:38.553941 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-fhh55" Jul 13 22:46:38.685970 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:38.685946 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5tksb/perf-node-gather-daemonset-fhh55"] Jul 13 22:46:38.688475 ip-10-0-132-128 kubenswrapper[2562]: W0713 22:46:38.688446 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod19526d78_70f9_468d_b31c_b37b1fc6b86f.slice/crio-34db3bfc58ba788a2887824a2c31739e30b46d3223a220edac66f41a2f3f33e7 WatchSource:0}: Error finding container 34db3bfc58ba788a2887824a2c31739e30b46d3223a220edac66f41a2f3f33e7: Status 404 returned error can't find the container with id 34db3bfc58ba788a2887824a2c31739e30b46d3223a220edac66f41a2f3f33e7 Jul 13 22:46:39.215465 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:39.215434 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x5qhq_7b1fa75b-f29d-45a1-af98-fa947c2058b2/console-operator/1.log" Jul 13 22:46:39.220377 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:39.220349 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-x5qhq_7b1fa75b-f29d-45a1-af98-fa947c2058b2/console-operator/2.log" Jul 13 22:46:39.257526 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:39.257488 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-fhh55" event={"ID":"19526d78-70f9-468d-b31c-b37b1fc6b86f","Type":"ContainerStarted","Data":"32bc1c995dff6182702364a3cf098ec3ae578ca987b6660c7ab25729fb3cc35d"} Jul 13 22:46:39.257963 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:39.257571 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-fhh55" event={"ID":"19526d78-70f9-468d-b31c-b37b1fc6b86f","Type":"ContainerStarted","Data":"34db3bfc58ba788a2887824a2c31739e30b46d3223a220edac66f41a2f3f33e7"} Jul 13 22:46:39.257963 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:39.257654 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-fhh55" Jul 13 22:46:39.275076 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:39.275018 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-fhh55" podStartSLOduration=1.275000337 podStartE2EDuration="1.275000337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-13 22:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-13 22:46:39.27310567 +0000 UTC m=+2820.723661248" watchObservedRunningTime="2026-07-13 22:46:39.275000337 +0000 UTC m=+2820.725555898" Jul 13 22:46:40.244651 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:40.244618 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-g497d_b9117cd6-d8c5-43a8-9a5d-8b5a349a5943/volume-data-source-validator/0.log" Jul 13 22:46:41.335108 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:41.335081 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qzmt4_06450a5e-03fe-4a9e-a6e8-722e0fc36664/dns/0.log" Jul 13 22:46:41.352743 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:41.352717 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qzmt4_06450a5e-03fe-4a9e-a6e8-722e0fc36664/kube-rbac-proxy/0.log" Jul 13 22:46:41.414098 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:41.414068 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ccs2h_3f77cde0-c7bd-4803-8c7b-b5e4071ace68/dns-node-resolver/0.log" Jul 13 22:46:41.994324 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:41.994296 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7dpl6_348d04ea-a0a4-45da-ba4b-480288fd701b/node-ca/0.log" Jul 13 22:46:43.034992 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:43.034946 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bf5d9c687-mx6hg_fd72b8d2-795c-4eb0-8086-6a685215395a/kube-auth-proxy/0.log" Jul 13 22:46:43.221750 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:43.221718 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5dbcf998b4-7qqv9_dbedfa07-690d-4c79-92d3-dc85b88124d7/router/0.log" Jul 13 22:46:43.944323 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:43.944278 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hj2nl_b64f1d4a-95b8-4abb-8a05-8b246fe11842/serve-healthcheck-canary/0.log" Jul 13 22:46:44.397534 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:44.397501 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-5vww5_3815ceb2-a3b8-43bc-bf04-b1dd4adc4412/insights-operator/0.log" Jul 13 22:46:44.397995 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:44.397849 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-5vww5_3815ceb2-a3b8-43bc-bf04-b1dd4adc4412/insights-operator/1.log" Jul 13 22:46:44.644754 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:44.644728 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vhlb7_9d1a0995-48f0-46b7-a75d-6c10a0860e7b/kube-rbac-proxy/0.log" Jul 13 22:46:44.666479 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:44.666416 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vhlb7_9d1a0995-48f0-46b7-a75d-6c10a0860e7b/exporter/0.log" Jul 13 22:46:44.696200 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:44.696178 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vhlb7_9d1a0995-48f0-46b7-a75d-6c10a0860e7b/extractor/0.log" Jul 13 22:46:45.272067 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:45.272041 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5tksb/perf-node-gather-daemonset-fhh55" Jul 13 22:46:47.987479 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:47.987441 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67ddb65fc-tsrpc_61805a0f-31ae-4a71-954e-be2de5d6a38b/manager/0.log" Jul 13 22:46:48.006635 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:48.006614 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67ddb65fc-tsrpc_61805a0f-31ae-4a71-954e-be2de5d6a38b/copy-manifests/0.log" Jul 13 22:46:49.008695 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:49.008665 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bf5d9c687-mx6hg_fd72b8d2-795c-4eb0-8086-6a685215395a/kube-auth-proxy/0.log" Jul 13 22:46:49.160691 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:49.160667 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5dbcf998b4-7qqv9_dbedfa07-690d-4c79-92d3-dc85b88124d7/router/0.log" Jul 13 22:46:49.430207 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:49.430181 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-88cc59766-kkpsn_d7de19e7-aa27-4e7b-9035-1176e6aedff9/manager/0.log" Jul 13 22:46:50.510812 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:50.510782 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6bf5d9c687-mx6hg_fd72b8d2-795c-4eb0-8086-6a685215395a/kube-auth-proxy/0.log" Jul 13 22:46:50.680735 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:50.680700 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5dbcf998b4-7qqv9_dbedfa07-690d-4c79-92d3-dc85b88124d7/router/0.log" Jul 13 22:46:54.628231 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:54.628193 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-rcbqx_db233478-500b-472c-919f-510266682def/kube-storage-version-migrator-operator/1.log" Jul 13 22:46:54.628873 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:54.628853 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-rcbqx_db233478-500b-472c-919f-510266682def/kube-storage-version-migrator-operator/0.log" Jul 13 22:46:55.907913 ip-10-0-132-128 kubenswrapper[2562]: I0713 22:46:55.907885 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lvr8p_75ac4b1c-ecb1-4550-84ab-7220e7445349/kube-multus-additional-cni-plugins/0.log"