Jul 06 16:36:12.905727 ip-10-0-131-71 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 06 16:36:12.905738 ip-10-0-131-71 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 06 16:36:12.905747 ip-10-0-131-71 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 06 16:36:12.906057 ip-10-0-131-71 systemd[1]: Failed to start Kubernetes Kubelet. Jul 06 16:36:23.114780 ip-10-0-131-71 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 06 16:36:23.114800 ip-10-0-131-71 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot d8c6176292184fbcb85edcb69fcde89f -- Jul 06 16:38:25.161831 ip-10-0-131-71 systemd[1]: Starting Kubernetes Kubelet... Jul 06 16:38:25.644033 ip-10-0-131-71 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 16:38:25.644033 ip-10-0-131-71 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 16:38:25.644033 ip-10-0-131-71 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 16:38:25.644033 ip-10-0-131-71 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 16:38:25.644033 ip-10-0-131-71 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 16:38:25.644033 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.568097 2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 16:38:25.644033 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578596 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 16:38:25.644033 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578615 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 16:38:25.644033 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578620 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 16:38:25.644033 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578623 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 16:38:25.644033 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578626 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 16:38:25.644033 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578629 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 16:38:25.644033 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578632 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 16:38:25.644033 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578635 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 16:38:25.645128 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578638 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 16:38:25.645128 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578640 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 16:38:25.645128 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578643 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 16:38:25.645128 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578646 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 16:38:25.645128 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578648 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 16:38:25.645128 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578651 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 16:38:25.645128 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578653 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 16:38:25.645128 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578656 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 16:38:25.645128 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578658 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 16:38:25.645128 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578661 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 16:38:25.645128 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578665 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 16:38:25.645128 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578670 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 16:38:25.645128 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578673 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 16:38:25.645128 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578675 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 16:38:25.645128 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578678 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 16:38:25.645128 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578681 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 16:38:25.645128 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578683 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 16:38:25.645128 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578689 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 16:38:25.645128 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578692 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 16:38:25.645128 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578694 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 16:38:25.646572 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578697 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 16:38:25.646572 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578700 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 16:38:25.646572 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578702 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 16:38:25.646572 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578705 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 16:38:25.646572 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578708 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 16:38:25.646572 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578711 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 16:38:25.646572 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578714 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 16:38:25.646572 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578716 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 16:38:25.646572 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578719 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 16:38:25.646572 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578722 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 16:38:25.646572 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578724 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 16:38:25.646572 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578728 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 16:38:25.646572 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578730 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 16:38:25.646572 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578733 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 16:38:25.646572 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578736 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 16:38:25.646572 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578738 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 16:38:25.646572 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578741 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 16:38:25.646572 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578743 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 16:38:25.646572 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578746 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 16:38:25.646572 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578748 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 16:38:25.647368 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578770 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 16:38:25.647368 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578774 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 16:38:25.647368 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578778 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 16:38:25.647368 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578782 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 16:38:25.647368 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578785 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 16:38:25.647368 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578787 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 16:38:25.647368 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578790 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 16:38:25.647368 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578792 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 16:38:25.647368 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578795 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 16:38:25.647368 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578798 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 16:38:25.647368 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578802 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 16:38:25.647368 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578806 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 16:38:25.647368 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578808 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 16:38:25.647368 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578812 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 16:38:25.647368 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578815 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 16:38:25.647368 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578820 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 16:38:25.647368 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578823 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 16:38:25.647368 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578827 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 16:38:25.647368 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578831 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 16:38:25.648633 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578834 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 16:38:25.648633 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578837 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 16:38:25.648633 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578840 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 16:38:25.648633 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578842 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 16:38:25.648633 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578845 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 16:38:25.648633 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578849 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 16:38:25.648633 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578852 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 16:38:25.648633 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578855 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 16:38:25.648633 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578857 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 16:38:25.648633 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578860 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 16:38:25.648633 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578863 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 16:38:25.648633 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578865 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 16:38:25.648633 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578868 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 16:38:25.648633 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578870 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 06 16:38:25.648633 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578873 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 16:38:25.648633 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578875 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 16:38:25.648633 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578878 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 16:38:25.648633 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578881 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 16:38:25.648633 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578883 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 16:38:25.648633 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578885 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 16:38:25.649414 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578888 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 16:38:25.649414 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578890 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 16:38:25.649414 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578893 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 16:38:25.649414 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578896 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 16:38:25.649414 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578898 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 16:38:25.649414 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578901 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 16:38:25.649414 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578904 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 16:38:25.649414 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578906 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 16:38:25.649414 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578910 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 16:38:25.649414 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578912 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 16:38:25.649414 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578915 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 16:38:25.649414 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.578917 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 16:38:25.649414 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579650 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 16:38:25.649414 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579657 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 16:38:25.649414 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579660 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 16:38:25.649414 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579663 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 16:38:25.649414 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579668 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 16:38:25.649414 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579672 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 16:38:25.649414 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579675 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 16:38:25.649414 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579678 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 16:38:25.650650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579681 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 16:38:25.650650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579684 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 16:38:25.650650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579686 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 16:38:25.650650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579689 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 16:38:25.650650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579692 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 16:38:25.650650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579695 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 16:38:25.650650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579697 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 16:38:25.650650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579700 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 16:38:25.650650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579703 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 16:38:25.650650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579707 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 16:38:25.650650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579709 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 16:38:25.650650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579712 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 16:38:25.650650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579714 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 16:38:25.650650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579717 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 16:38:25.650650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579719 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 16:38:25.650650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579722 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 16:38:25.650650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579724 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 16:38:25.650650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579727 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 16:38:25.650650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579730 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 16:38:25.650650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579732 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 16:38:25.650413 ip-10-0-131-71 systemd[1]: Started Kubernetes Kubelet. Jul 06 16:38:25.652857 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579736 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 16:38:25.652857 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579739 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 16:38:25.652857 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579741 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 16:38:25.652857 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579744 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 16:38:25.652857 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579746 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 16:38:25.652857 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579749 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 16:38:25.652857 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579751 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 16:38:25.652857 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579754 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 16:38:25.652857 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579756 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 16:38:25.652857 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579759 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 16:38:25.652857 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579762 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 16:38:25.652857 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579764 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 16:38:25.652857 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579768 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 16:38:25.652857 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579772 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 16:38:25.652857 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579775 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 16:38:25.652857 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579777 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 16:38:25.652857 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579780 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 16:38:25.652857 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579782 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 16:38:25.652857 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579785 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 16:38:25.652857 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579788 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 16:38:25.653444 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579791 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 16:38:25.653444 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579793 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 16:38:25.653444 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579796 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 16:38:25.653444 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579799 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 16:38:25.653444 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579801 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 16:38:25.653444 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579804 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 16:38:25.653444 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579806 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 16:38:25.653444 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579809 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 16:38:25.653444 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579811 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 16:38:25.653444 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579814 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 16:38:25.653444 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579817 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 16:38:25.653444 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579819 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 06 16:38:25.653444 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579823 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 16:38:25.653444 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579825 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 16:38:25.653444 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579828 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 16:38:25.653444 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579831 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 16:38:25.653444 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579833 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 16:38:25.653444 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579836 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 16:38:25.653444 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579839 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 16:38:25.653986 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579841 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 16:38:25.653986 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579844 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 16:38:25.653986 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579846 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 16:38:25.653986 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579849 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 16:38:25.653986 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579851 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 16:38:25.653986 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579854 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 16:38:25.653986 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579856 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 16:38:25.653986 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579859 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 16:38:25.653986 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579861 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 16:38:25.653986 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579866 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 16:38:25.653986 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579869 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 16:38:25.653986 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579871 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 16:38:25.653986 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579874 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 16:38:25.653986 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579877 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 16:38:25.653986 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579879 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 16:38:25.653986 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579882 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 16:38:25.653986 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579885 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 16:38:25.653986 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579887 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 16:38:25.653986 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579890 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 16:38:25.653986 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579892 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 16:38:25.654558 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579895 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 16:38:25.654558 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579897 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 16:38:25.654558 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579901 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 16:38:25.654558 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579906 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 16:38:25.654558 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579910 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 16:38:25.654558 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579913 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 16:38:25.654558 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579916 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 16:38:25.654558 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579919 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 16:38:25.654558 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579922 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 16:38:25.654558 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579924 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 16:38:25.654558 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579927 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 16:38:25.654558 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.579930 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 16:38:25.654558 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580334 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 16:38:25.654558 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580339 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 16:38:25.654558 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580342 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 16:38:25.654558 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580344 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 16:38:25.654558 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580347 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 16:38:25.654558 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580349 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 16:38:25.654558 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580352 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 16:38:25.655057 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580355 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 16:38:25.655057 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580357 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 16:38:25.655057 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580365 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 16:38:25.655057 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580369 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 16:38:25.655057 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580373 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 16:38:25.655057 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580376 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 16:38:25.655057 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580379 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 16:38:25.655057 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580383 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 16:38:25.655057 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580387 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 16:38:25.655057 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580391 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 16:38:25.655057 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580394 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 16:38:25.655057 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580397 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 16:38:25.655057 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580400 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 16:38:25.655057 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580402 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 16:38:25.655057 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580404 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 16:38:25.655057 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580407 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 06 16:38:25.655057 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580410 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 16:38:25.655057 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580412 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 16:38:25.655057 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580415 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 16:38:25.655057 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580417 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 16:38:25.655555 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580421 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 16:38:25.655555 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580423 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 16:38:25.655555 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580426 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 16:38:25.655555 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580428 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 16:38:25.655555 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580431 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 16:38:25.655555 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580433 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 16:38:25.655555 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580436 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 16:38:25.655555 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580438 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 16:38:25.655555 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580441 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 16:38:25.655555 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580444 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 16:38:25.655555 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580447 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 16:38:25.655555 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580449 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 16:38:25.655555 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580452 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 16:38:25.655555 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580454 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 16:38:25.655555 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580458 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 16:38:25.655555 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580461 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 16:38:25.655555 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580484 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 16:38:25.655555 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580488 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 16:38:25.655555 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580492 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 16:38:25.656065 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580496 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 16:38:25.656065 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580499 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 16:38:25.656065 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580501 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 16:38:25.656065 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580505 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 16:38:25.656065 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580508 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 16:38:25.656065 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580510 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 16:38:25.656065 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580512 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 16:38:25.656065 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580515 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 16:38:25.656065 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580517 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 16:38:25.656065 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580520 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 16:38:25.656065 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580523 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 16:38:25.656065 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580526 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 16:38:25.656065 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580530 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 16:38:25.656065 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580533 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 16:38:25.656065 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580536 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 16:38:25.656065 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580538 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 16:38:25.656065 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580542 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 16:38:25.656065 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580545 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 16:38:25.656065 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580548 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 16:38:25.656538 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580550 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 16:38:25.656538 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580553 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 16:38:25.656538 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580555 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 16:38:25.656538 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580558 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 16:38:25.656538 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580560 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 16:38:25.656538 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580562 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 16:38:25.656538 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580565 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 16:38:25.656538 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580567 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 16:38:25.656538 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580571 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 16:38:25.656538 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580574 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 16:38:25.656538 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580576 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 16:38:25.656538 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580578 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 16:38:25.656538 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580581 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 16:38:25.656538 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580584 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 16:38:25.656538 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580586 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 16:38:25.656538 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580589 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 16:38:25.656538 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580591 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 16:38:25.656538 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580594 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 16:38:25.656538 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580596 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 16:38:25.656538 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580599 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 16:38:25.657009 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580601 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 16:38:25.657009 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580604 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 16:38:25.657009 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580606 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 16:38:25.657009 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580609 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 16:38:25.657009 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580612 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 16:38:25.657009 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580614 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 16:38:25.657009 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580617 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 16:38:25.657009 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580619 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 16:38:25.657009 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580622 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 16:38:25.657009 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580624 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 16:38:25.657009 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580627 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 16:38:25.657009 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580629 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 16:38:25.657009 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580632 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 16:38:25.657009 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.580634 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 16:38:25.657009 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580710 2574 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 16:38:25.657009 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580717 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 16:38:25.657009 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580724 2574 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 16:38:25.657009 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580728 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 16:38:25.657009 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580733 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 16:38:25.657009 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580736 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 16:38:25.657009 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580742 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 16:38:25.657554 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580746 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 16:38:25.657554 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580749 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 16:38:25.657554 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580752 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 16:38:25.657554 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580755 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 16:38:25.657554 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580759 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 16:38:25.657554 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580762 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 16:38:25.657554 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580765 2574 flags.go:64] FLAG: --cgroup-root="" Jul 06 16:38:25.657554 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580768 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 16:38:25.657554 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580771 2574 flags.go:64] FLAG: --client-ca-file="" Jul 06 16:38:25.657554 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580775 2574 flags.go:64] FLAG: --cloud-provider="external" Jul 06 16:38:25.657554 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580778 2574 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 16:38:25.657554 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580782 2574 flags.go:64] FLAG: --cluster-domain="" Jul 06 16:38:25.657554 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580784 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 16:38:25.657554 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580788 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 16:38:25.657554 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580791 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 16:38:25.657554 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580795 2574 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 16:38:25.657554 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580798 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 16:38:25.657554 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580801 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 16:38:25.657554 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580804 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 16:38:25.657554 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580808 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 16:38:25.657554 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580811 2574 flags.go:64] FLAG: --contention-profiling="false" Jul 06 16:38:25.657554 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580814 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 16:38:25.657554 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580816 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 16:38:25.657554 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580819 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 16:38:25.658146 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580822 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 16:38:25.658146 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580827 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 16:38:25.658146 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580830 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 16:38:25.658146 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580833 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 16:38:25.658146 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580835 2574 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 16:38:25.658146 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580838 2574 flags.go:64] FLAG: --enable-server="true" Jul 06 16:38:25.658146 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580841 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 16:38:25.658146 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580848 2574 flags.go:64] FLAG: --event-burst="100" Jul 06 16:38:25.658146 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580851 2574 flags.go:64] FLAG: --event-qps="50" Jul 06 16:38:25.658146 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580856 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 16:38:25.658146 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580859 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 16:38:25.658146 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580861 2574 flags.go:64] FLAG: --eviction-hard="" Jul 06 16:38:25.658146 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580865 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 16:38:25.658146 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580868 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 16:38:25.658146 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580871 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 16:38:25.658146 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580874 2574 flags.go:64] FLAG: --eviction-soft="" Jul 06 16:38:25.658146 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580877 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 16:38:25.658146 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580880 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 16:38:25.658146 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580884 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 16:38:25.658146 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580887 2574 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 16:38:25.658146 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580889 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 16:38:25.658146 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580892 2574 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 16:38:25.658146 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580895 2574 flags.go:64] FLAG: --feature-gates="" Jul 06 16:38:25.658146 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580899 2574 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 16:38:25.658146 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580902 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 16:38:25.658935 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580905 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 16:38:25.658935 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580908 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 16:38:25.658935 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580912 2574 flags.go:64] FLAG: --healthz-port="10248" Jul 06 16:38:25.658935 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580915 2574 flags.go:64] FLAG: --help="false" Jul 06 16:38:25.658935 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580917 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-131-71.ec2.internal" Jul 06 16:38:25.658935 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580921 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 16:38:25.658935 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580923 2574 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 16:38:25.658935 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580926 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 16:38:25.658935 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580929 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 16:38:25.658935 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580933 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 16:38:25.658935 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580936 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 16:38:25.658935 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580938 2574 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 16:38:25.658935 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580941 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 16:38:25.658935 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580944 2574 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 16:38:25.658935 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580948 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 16:38:25.658935 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580951 2574 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 16:38:25.658935 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580957 2574 flags.go:64] FLAG: --kube-reserved="" Jul 06 16:38:25.658935 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580959 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 16:38:25.658935 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580962 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 16:38:25.658935 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580965 2574 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 16:38:25.658935 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580968 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 16:38:25.658935 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580971 2574 flags.go:64] FLAG: --lock-file="" Jul 06 16:38:25.658935 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580974 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 16:38:25.658935 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580977 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 16:38:25.659732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580979 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 16:38:25.659732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580985 2574 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 16:38:25.659732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580989 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 16:38:25.659732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580992 2574 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 16:38:25.659732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580995 2574 flags.go:64] FLAG: --logging-format="text" Jul 06 16:38:25.659732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.580997 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 16:38:25.659732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581001 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 16:38:25.659732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581003 2574 flags.go:64] FLAG: --manifest-url="" Jul 06 16:38:25.659732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581006 2574 flags.go:64] FLAG: --manifest-url-header="" Jul 06 16:38:25.659732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581010 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 16:38:25.659732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581013 2574 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 16:38:25.659732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581018 2574 flags.go:64] FLAG: --max-pods="110" Jul 06 16:38:25.659732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581021 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 16:38:25.659732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581023 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 16:38:25.659732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581026 2574 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 16:38:25.659732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581029 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 16:38:25.659732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581032 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 16:38:25.659732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581035 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 16:38:25.659732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581038 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 16:38:25.659732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581045 2574 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 16:38:25.659732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581048 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 16:38:25.659732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581051 2574 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 16:38:25.659732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581055 2574 flags.go:64] FLAG: --pod-cidr="" Jul 06 16:38:25.660361 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581058 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 16:38:25.660361 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581063 2574 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 16:38:25.660361 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581067 2574 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 16:38:25.660361 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581070 2574 flags.go:64] FLAG: --pods-per-core="0" Jul 06 16:38:25.660361 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581073 2574 flags.go:64] FLAG: --port="10250" Jul 06 16:38:25.660361 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581076 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 16:38:25.660361 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581079 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ff3c3ee9930e2611" Jul 06 16:38:25.660361 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581082 2574 flags.go:64] FLAG: --qos-reserved="" Jul 06 16:38:25.660361 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581085 2574 flags.go:64] FLAG: --read-only-port="10255" Jul 06 16:38:25.660361 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581088 2574 flags.go:64] FLAG: --register-node="true" Jul 06 16:38:25.660361 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581091 2574 flags.go:64] FLAG: --register-with-taints="" Jul 06 16:38:25.660361 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581094 2574 flags.go:64] FLAG: --registry-burst="10" Jul 06 16:38:25.660361 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581098 2574 flags.go:64] FLAG: --registry-qps="5" Jul 06 16:38:25.660361 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581106 2574 flags.go:64] FLAG: --reserved-cpus="" Jul 06 16:38:25.660361 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581109 2574 flags.go:64] FLAG: --reserved-memory="" Jul 06 16:38:25.660361 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581112 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 16:38:25.660361 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581115 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 16:38:25.660361 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581119 2574 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 16:38:25.660361 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581122 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 16:38:25.660361 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581124 2574 flags.go:64] FLAG: --runonce="false" Jul 06 16:38:25.660361 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581127 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 16:38:25.660361 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581130 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 16:38:25.660361 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581134 2574 flags.go:64] FLAG: --seccomp-default="false" Jul 06 16:38:25.660361 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581137 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 16:38:25.660361 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581140 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 16:38:25.661389 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581142 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 16:38:25.661389 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581145 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 16:38:25.661389 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581148 2574 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 16:38:25.661389 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581151 2574 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 16:38:25.661389 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581154 2574 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 16:38:25.661389 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581157 2574 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 16:38:25.661389 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581161 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 16:38:25.661389 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581164 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 16:38:25.661389 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581166 2574 flags.go:64] FLAG: --system-cgroups="" Jul 06 16:38:25.661389 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581170 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 16:38:25.661389 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581176 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 16:38:25.661389 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581178 2574 flags.go:64] FLAG: --tls-cert-file="" Jul 06 16:38:25.661389 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581181 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 16:38:25.661389 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581185 2574 flags.go:64] FLAG: --tls-min-version="" Jul 06 16:38:25.661389 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581188 2574 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 16:38:25.661389 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581191 2574 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 16:38:25.661389 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581194 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 16:38:25.661389 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581196 2574 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 16:38:25.661389 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581200 2574 flags.go:64] FLAG: --v="2" Jul 06 16:38:25.661389 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581204 2574 flags.go:64] FLAG: --version="false" Jul 06 16:38:25.661389 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581208 2574 flags.go:64] FLAG: --vmodule="" Jul 06 16:38:25.661389 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581212 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 16:38:25.661389 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.581215 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 16:38:25.661389 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581328 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 16:38:25.662332 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581333 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 16:38:25.662332 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581336 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 16:38:25.662332 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581339 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 16:38:25.662332 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581342 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 16:38:25.662332 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581345 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 16:38:25.662332 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581347 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 16:38:25.662332 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581350 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 16:38:25.662332 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581353 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 16:38:25.662332 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581355 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 16:38:25.662332 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581358 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 16:38:25.662332 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581360 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 16:38:25.662332 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581363 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 16:38:25.662332 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581365 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 16:38:25.662332 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581367 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 16:38:25.662332 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581371 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 16:38:25.662332 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581374 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 16:38:25.662332 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581376 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 16:38:25.662332 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581380 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 16:38:25.662332 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581383 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 16:38:25.662332 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581385 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 16:38:25.662963 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581387 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 16:38:25.662963 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581390 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 16:38:25.662963 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581393 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 16:38:25.662963 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581395 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 16:38:25.662963 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581399 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 16:38:25.662963 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581402 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 16:38:25.662963 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581406 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 16:38:25.662963 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581408 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 16:38:25.662963 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581411 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 16:38:25.662963 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581414 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 06 16:38:25.662963 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581417 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 16:38:25.662963 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581419 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 16:38:25.662963 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581422 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 16:38:25.662963 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581424 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 16:38:25.662963 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581427 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 16:38:25.662963 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581429 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 16:38:25.662963 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581432 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 16:38:25.662963 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581434 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 16:38:25.662963 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581437 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 16:38:25.663420 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581440 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 16:38:25.663420 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581442 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 16:38:25.663420 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581444 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 16:38:25.663420 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581447 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 16:38:25.663420 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581450 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 16:38:25.663420 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581452 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 16:38:25.663420 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581455 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 16:38:25.663420 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581459 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 16:38:25.663420 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581478 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 16:38:25.663420 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581481 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 16:38:25.663420 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581485 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 16:38:25.663420 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581488 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 16:38:25.663420 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581490 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 16:38:25.663420 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581493 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 16:38:25.663420 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581496 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 16:38:25.663420 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581498 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 16:38:25.663420 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581501 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 16:38:25.663420 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581503 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 16:38:25.663420 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581506 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 16:38:25.663917 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581509 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 16:38:25.663917 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581511 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 16:38:25.663917 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581514 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 16:38:25.663917 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581516 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 16:38:25.663917 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581519 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 16:38:25.663917 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581522 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 16:38:25.663917 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581525 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 16:38:25.663917 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581528 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 16:38:25.663917 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581530 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 16:38:25.663917 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581533 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 16:38:25.663917 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581535 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 16:38:25.663917 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581538 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 16:38:25.663917 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581540 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 16:38:25.663917 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581543 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 16:38:25.663917 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581545 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 16:38:25.663917 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581548 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 16:38:25.663917 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581550 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 16:38:25.663917 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581553 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 16:38:25.663917 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581555 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 16:38:25.663917 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581558 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 16:38:25.664481 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581562 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 16:38:25.664481 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581564 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 16:38:25.664481 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581567 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 16:38:25.664481 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581571 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 16:38:25.664481 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581574 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 16:38:25.664481 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581577 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 16:38:25.664481 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581581 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 16:38:25.664481 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581583 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 16:38:25.664481 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581586 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 16:38:25.664481 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581589 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 16:38:25.664481 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581592 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 16:38:25.664481 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581595 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 16:38:25.664481 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581597 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 16:38:25.664481 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581600 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 16:38:25.664481 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581603 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 16:38:25.664481 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581606 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 16:38:25.664481 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581608 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 16:38:25.664481 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581611 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 16:38:25.664481 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581615 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 16:38:25.664481 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.581618 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 16:38:25.665413 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.582241 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 16:38:25.665413 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.589294 2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 16:38:25.665413 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.589314 2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 16:38:25.665413 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.589331 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 16:38:25.665413 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.589338 2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 16:38:25.665413 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589400 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 16:38:25.665413 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589406 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 16:38:25.665413 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589409 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 16:38:25.665413 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589412 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 16:38:25.665413 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589416 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 16:38:25.665413 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589419 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 16:38:25.665413 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589422 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 16:38:25.665413 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589425 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 16:38:25.665413 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589427 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 16:38:25.665413 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589430 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 16:38:25.665990 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589433 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 16:38:25.665990 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589435 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 16:38:25.665990 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589438 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 16:38:25.665990 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589440 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 16:38:25.665990 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589443 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 16:38:25.665990 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589446 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 16:38:25.665990 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589448 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 16:38:25.665990 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589479 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 16:38:25.665990 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589485 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 16:38:25.665990 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589488 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 16:38:25.665990 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589492 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 16:38:25.665990 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589497 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 16:38:25.665990 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589499 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 16:38:25.665990 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589502 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 16:38:25.665990 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589505 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 16:38:25.665990 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589507 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 16:38:25.665990 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589510 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 16:38:25.665990 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589512 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 16:38:25.665990 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589515 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 16:38:25.666650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589519 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 16:38:25.666650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589522 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 16:38:25.666650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589525 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 16:38:25.666650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589528 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 16:38:25.666650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589530 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 16:38:25.666650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589539 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 16:38:25.666650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589542 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 16:38:25.666650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589544 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 16:38:25.666650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589547 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 16:38:25.666650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589550 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 16:38:25.666650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589552 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 16:38:25.666650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589555 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 16:38:25.666650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589557 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 16:38:25.666650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589560 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 16:38:25.666650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589562 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 16:38:25.666650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589565 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 16:38:25.666650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589568 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 16:38:25.666650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589570 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 16:38:25.666650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589573 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 16:38:25.666650 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589576 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 16:38:25.667138 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589578 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 16:38:25.667138 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589581 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 16:38:25.667138 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589583 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 06 16:38:25.667138 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589585 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 16:38:25.667138 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589588 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 16:38:25.667138 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589591 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 16:38:25.667138 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589593 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 16:38:25.667138 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589596 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 16:38:25.667138 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589599 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 16:38:25.667138 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589601 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 16:38:25.667138 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589604 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 16:38:25.667138 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589606 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 16:38:25.667138 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589610 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 16:38:25.667138 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589616 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 16:38:25.667138 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589619 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 16:38:25.667138 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589622 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 16:38:25.667138 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589625 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 16:38:25.667138 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589627 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 16:38:25.667138 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589631 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 16:38:25.667138 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589633 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 16:38:25.667665 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589636 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 16:38:25.667665 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589639 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 16:38:25.667665 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589641 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 16:38:25.667665 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589644 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 16:38:25.667665 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589646 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 16:38:25.667665 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589649 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 16:38:25.667665 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589651 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 16:38:25.667665 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589654 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 16:38:25.667665 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589656 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 16:38:25.667665 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589659 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 16:38:25.667665 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589661 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 16:38:25.667665 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589664 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 16:38:25.667665 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589666 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 16:38:25.667665 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589669 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 16:38:25.667665 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589671 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 16:38:25.667665 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589673 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 16:38:25.667665 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589676 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 16:38:25.667665 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589678 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 16:38:25.667665 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589681 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 16:38:25.667665 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589683 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 16:38:25.668140 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589686 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 16:38:25.668140 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589688 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 16:38:25.668140 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589691 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 16:38:25.668140 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589693 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 16:38:25.668140 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589696 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 16:38:25.668140 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589700 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 16:38:25.668140 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589707 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 16:38:25.668140 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589710 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 16:38:25.668140 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589713 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 16:38:25.668140 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589716 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 16:38:25.668140 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.589721 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 16:38:25.668140 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589859 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 16:38:25.668140 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589863 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 16:38:25.668140 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589866 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 16:38:25.668140 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589869 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 16:38:25.668556 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589872 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 16:38:25.668556 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589875 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 16:38:25.668556 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589878 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 16:38:25.668556 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589880 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 16:38:25.668556 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589883 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 16:38:25.668556 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589885 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 16:38:25.668556 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589888 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 16:38:25.668556 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589891 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 16:38:25.668556 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589893 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 16:38:25.668556 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589895 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 16:38:25.668556 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589898 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 16:38:25.668556 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589901 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 16:38:25.668556 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589903 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 16:38:25.668556 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589906 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 16:38:25.668556 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589908 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 06 16:38:25.668556 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589911 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 16:38:25.668556 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589914 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 16:38:25.668556 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589916 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 16:38:25.668556 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589919 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 16:38:25.668556 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589921 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 16:38:25.669022 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589924 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 16:38:25.669022 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589926 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 16:38:25.669022 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589929 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 16:38:25.669022 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589932 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 16:38:25.669022 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589940 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 16:38:25.669022 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589943 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 16:38:25.669022 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589947 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 16:38:25.669022 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589951 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 16:38:25.669022 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589954 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 16:38:25.669022 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589957 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 16:38:25.669022 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589959 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 16:38:25.669022 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589962 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 16:38:25.669022 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589965 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 16:38:25.669022 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589967 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 16:38:25.669022 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589970 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 16:38:25.669022 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589972 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 16:38:25.669022 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589975 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 16:38:25.669022 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589977 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 16:38:25.669022 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589980 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 16:38:25.669022 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589982 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 16:38:25.669520 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589985 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 16:38:25.669520 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589987 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 16:38:25.669520 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589990 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 16:38:25.669520 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589992 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 16:38:25.669520 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589995 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 16:38:25.669520 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589997 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 16:38:25.669520 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.589999 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 16:38:25.669520 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590002 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 16:38:25.669520 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590005 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 16:38:25.669520 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590007 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 16:38:25.669520 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590010 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 16:38:25.669520 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590014 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 16:38:25.669520 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590017 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 16:38:25.669520 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590022 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 16:38:25.669520 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590026 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 16:38:25.669520 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590029 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 16:38:25.669520 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590031 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 16:38:25.669520 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590039 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 16:38:25.669520 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590042 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 16:38:25.669965 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590045 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 16:38:25.669965 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590047 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 16:38:25.669965 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590050 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 16:38:25.669965 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590052 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 16:38:25.669965 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590055 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 16:38:25.669965 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590058 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 16:38:25.669965 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590060 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 16:38:25.669965 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590063 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 16:38:25.669965 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590065 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 16:38:25.669965 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590068 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 16:38:25.669965 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590071 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 16:38:25.669965 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590073 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 16:38:25.669965 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590075 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 16:38:25.669965 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590078 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 16:38:25.669965 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590081 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 16:38:25.669965 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590083 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 16:38:25.669965 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590086 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 16:38:25.669965 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590088 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 16:38:25.669965 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590091 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 16:38:25.669965 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590093 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 16:38:25.670448 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590097 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 16:38:25.670448 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590100 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 16:38:25.670448 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590103 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 16:38:25.670448 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590106 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 16:38:25.670448 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590109 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 16:38:25.670448 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590112 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 16:38:25.670448 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590116 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 16:38:25.670448 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590121 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 16:38:25.670448 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590123 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 16:38:25.670448 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590126 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 16:38:25.670448 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590129 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 16:38:25.670448 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590137 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 16:38:25.670448 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590140 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 16:38:25.670448 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590142 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 16:38:25.670448 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590145 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 16:38:25.670448 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:25.590147 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 16:38:25.670851 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.590152 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 16:38:25.670851 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.590313 2574 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 16:38:25.670851 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.592808 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 16:38:25.670851 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.594210 2574 server.go:1019] "Starting client certificate rotation" Jul 06 16:38:25.670851 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.594311 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 16:38:25.670851 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.594365 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 16:38:25.670851 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.596666 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 16:38:25.670851 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.598176 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 16:38:25.670851 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.612630 2574 log.go:25] "Validated CRI v1 runtime API" Jul 06 16:38:25.670851 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.614886 2574 log.go:25] "Validated CRI v1 image API" Jul 06 16:38:25.670851 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.617336 2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 16:38:25.670851 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.619585 2574 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ca716879-2ffb-4b2a-921b-4423b7c8c681:/dev/nvme0n1p4 cd25ac8f-d7db-4351-a0c2-bac4ce87bcdf:/dev/nvme0n1p3] Jul 06 16:38:25.671146 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.619602 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 16:38:25.671178 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.624892 2574 manager.go:216] Machine: {Timestamp:2026-07-06 16:38:25.623566397 +0000 UTC m=+0.341107848 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105880 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dd54840b4aac68acf8e9b9f52bb9f SystemUUID:ec2dd548-40b4-aac6-8acf-8e9b9f52bb9f BootID:d8c61762-9218-4fbc-b85e-dcb69fcde89f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a8:2c:7e:d6:91 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a8:2c:7e:d6:91 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:9e:65:f9:72:9b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 16:38:25.671178 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.625011 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 16:38:25.671178 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.625096 2574 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 16:38:25.671178 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.626670 2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 16:38:25.671178 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.626698 2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-71.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 16:38:25.671178 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.627417 2574 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 16:38:25.671178 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.627429 2574 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 16:38:25.671178 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.627436 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.628104 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.629459 2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.632687 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.632809 2574 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.633557 2574 kubelet.go:486] "Attempting to sync node with API server" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.633569 2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.633579 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.633590 2574 kubelet.go:398] "Adding apiserver pod source" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.633599 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.633920 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.633935 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.636177 2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.638100 2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.638128 2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.645104 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.645613 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.645626 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.645632 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.645638 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.645646 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.645652 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.645661 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.645666 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.645675 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.645681 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.645697 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.645702 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.646687 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.646694 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.649773 2574 server.go:1268] "Started kubelet" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.650127 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.651862 2574 server.go:310] "Adding debug handlers to kubelet server" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.652592 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.652641 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.653939 2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:25.658213 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-71.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.658283 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-71.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:25.658397 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 16:38:25.671407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.659571 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 16:38:25.672395 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.660014 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 16:38:25.672395 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:25.662812 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 06 16:38:25.672395 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.663531 2574 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 16:38:25.672395 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.663541 2574 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 16:38:25.672395 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.663783 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 16:38:25.672395 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.663840 2574 reconstruct.go:97] "Volume reconstruction finished" Jul 06 16:38:25.672395 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.663847 2574 reconciler.go:29] "Reconciler: start to sync state" Jul 06 16:38:25.672395 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:25.664736 2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 16:38:25.672395 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.665857 2574 factory.go:153] Registering CRI-O factory Jul 06 16:38:25.672395 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.665875 2574 factory.go:223] Registration of the crio container factory successfully Jul 06 16:38:25.672395 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.665924 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 16:38:25.672395 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.665933 2574 factory.go:55] Registering systemd factory Jul 06 16:38:25.672395 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.665941 2574 factory.go:223] Registration of the systemd container factory successfully Jul 06 16:38:25.672395 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.665961 2574 factory.go:103] Registering Raw factory Jul 06 16:38:25.672395 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.665972 2574 manager.go:1196] Started watching for new ooms in manager Jul 06 16:38:25.672395 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.666270 2574 manager.go:349] Starting recovery of all containers Jul 06 16:38:25.672395 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:25.666660 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 16:38:26.913076 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:26.912968 2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-71.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 16:38:26.913076 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:26.913026 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jxd5v" Jul 06 16:38:26.913318 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:26.913079 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 16:38:26.914733 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:26.914678 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jxd5v" Jul 06 16:38:26.916957 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:26.913088 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-71.ec2.internal.18bfc0bf60592d16 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-71.ec2.internal,UID:ip-10-0-131-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-71.ec2.internal,},FirstTimestamp:2026-07-06 16:38:25.64974927 +0000 UTC m=+0.367290720,LastTimestamp:2026-07-06 16:38:25.64974927 +0000 UTC m=+0.367290720,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-71.ec2.internal,}" Jul 06 16:38:26.924722 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:26.924703 2574 manager.go:354] Recovery completed Jul 06 16:38:26.929802 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:26.929788 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 16:38:26.931839 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:26.931815 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientMemory" Jul 06 16:38:26.931903 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:26.931844 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 16:38:26.931903 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:26.931855 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientPID" Jul 06 16:38:26.932422 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:26.932407 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 16:38:26.932422 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:26.932416 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 16:38:26.932532 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:26.932419 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 16:38:26.932532 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:26.932445 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 06 16:38:26.933430 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:26.933415 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientMemory" Jul 06 16:38:26.933479 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:26.933442 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 16:38:26.933479 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:26.933452 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientPID" Jul 06 16:38:26.933901 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:26.933890 2574 policy_none.go:49] "None policy: Start" Jul 06 16:38:26.933932 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:26.933905 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 16:38:26.933932 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:26.933915 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 16:38:26.934841 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:26.934831 2574 policy_none.go:47] "Start" Jul 06 16:38:26.962135 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:26.961956 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 06 16:38:26.977298 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:26.977281 2574 manager.go:337] "Starting Device Plugin manager" Jul 06 16:38:26.977414 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:26.977324 2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 16:38:26.977414 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:26.977337 2574 server.go:87] "Starting device plugin registration server" Jul 06 16:38:26.977626 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:26.977612 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 16:38:26.977690 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:26.977625 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 16:38:26.977763 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:26.977749 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 16:38:26.977838 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:26.977826 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 16:38:26.977838 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:26.977838 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 16:38:26.978396 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:26.978377 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 16:38:26.978481 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:26.978424 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-71.ec2.internal\" not found" Jul 06 16:38:27.069677 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.069634 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 16:38:27.070750 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.069692 2574 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 16:38:27.070750 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.069715 2574 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 16:38:27.070750 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:27.069753 2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 06 16:38:27.077917 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.077897 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 16:38:27.077917 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.077907 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 16:38:27.080017 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.079999 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientMemory" Jul 06 16:38:27.080104 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.080030 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 16:38:27.080104 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.080040 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientPID" Jul 06 16:38:27.080104 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.080062 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-71.ec2.internal" Jul 06 16:38:27.092691 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.092669 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-71.ec2.internal" Jul 06 16:38:27.092781 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:27.092696 2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-71.ec2.internal\": node \"ip-10-0-131-71.ec2.internal\" not found" Jul 06 16:38:27.127887 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:27.127857 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 06 16:38:27.170276 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.170240 2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-71.ec2.internal"] Jul 06 16:38:27.170438 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.170333 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 16:38:27.171311 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.171290 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientMemory" Jul 06 16:38:27.171433 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.171325 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 16:38:27.171433 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.171338 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientPID" Jul 06 16:38:27.172982 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.172968 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 16:38:27.173133 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.173117 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal" Jul 06 16:38:27.173182 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.173150 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 16:38:27.173845 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.173830 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientMemory" Jul 06 16:38:27.173931 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.173832 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientMemory" Jul 06 16:38:27.173931 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.173856 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 16:38:27.173931 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.173869 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 16:38:27.173931 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.173885 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientPID" Jul 06 16:38:27.173931 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.173870 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientPID" Jul 06 16:38:27.175130 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.175114 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-71.ec2.internal" Jul 06 16:38:27.175260 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.175147 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 16:38:27.178869 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.178852 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientMemory" Jul 06 16:38:27.178947 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.178880 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 16:38:27.178947 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.178889 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeHasSufficientPID" Jul 06 16:38:27.192115 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:27.192096 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-71.ec2.internal\" not found" node="ip-10-0-131-71.ec2.internal" Jul 06 16:38:27.197413 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:27.197398 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-71.ec2.internal\" not found" node="ip-10-0-131-71.ec2.internal" Jul 06 16:38:27.228554 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:27.228483 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 06 16:38:27.263195 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.263159 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/17554f49b63e35a815e8bf1b98f97477-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal\" (UID: \"17554f49b63e35a815e8bf1b98f97477\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal" Jul 06 16:38:27.263195 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.263193 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17554f49b63e35a815e8bf1b98f97477-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal\" (UID: \"17554f49b63e35a815e8bf1b98f97477\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal" Jul 06 16:38:27.263437 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.263219 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/03b7a04aec66663a9ae8155170654101-config\") pod \"kube-apiserver-proxy-ip-10-0-131-71.ec2.internal\" (UID: \"03b7a04aec66663a9ae8155170654101\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-71.ec2.internal" Jul 06 16:38:27.328989 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:27.328958 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 06 16:38:27.364145 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.364117 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/17554f49b63e35a815e8bf1b98f97477-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal\" (UID: \"17554f49b63e35a815e8bf1b98f97477\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal" Jul 06 16:38:27.364205 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.364153 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17554f49b63e35a815e8bf1b98f97477-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal\" (UID: \"17554f49b63e35a815e8bf1b98f97477\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal" Jul 06 16:38:27.364205 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.364170 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/03b7a04aec66663a9ae8155170654101-config\") pod \"kube-apiserver-proxy-ip-10-0-131-71.ec2.internal\" (UID: \"03b7a04aec66663a9ae8155170654101\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-71.ec2.internal" Jul 06 16:38:27.364280 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.364217 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/03b7a04aec66663a9ae8155170654101-config\") pod \"kube-apiserver-proxy-ip-10-0-131-71.ec2.internal\" (UID: \"03b7a04aec66663a9ae8155170654101\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-71.ec2.internal" Jul 06 16:38:27.364280 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.364221 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/17554f49b63e35a815e8bf1b98f97477-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal\" (UID: \"17554f49b63e35a815e8bf1b98f97477\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal" Jul 06 16:38:27.364280 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.364226 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17554f49b63e35a815e8bf1b98f97477-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal\" (UID: \"17554f49b63e35a815e8bf1b98f97477\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal" Jul 06 16:38:27.429044 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:27.429016 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 06 16:38:27.496855 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.496792 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal" Jul 06 16:38:27.501676 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.501641 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-71.ec2.internal" Jul 06 16:38:27.529361 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:27.529333 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 06 16:38:27.629839 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:27.629789 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 06 16:38:27.730301 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:27.730263 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 06 16:38:27.791809 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.791745 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 16:38:27.791930 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.791912 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 16:38:27.831124 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:27.831100 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 06 16:38:27.918455 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.918399 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 16:33:26 +0000 UTC" deadline="2028-02-01 16:18:17.547294944 +0000 UTC" Jul 06 16:38:27.918455 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:27.918450 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13799h39m49.628849058s" Jul 06 16:38:27.931576 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:27.931554 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 06 16:38:28.028938 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:28.028909 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17554f49b63e35a815e8bf1b98f97477.slice/crio-cce5d9a85bfa4ebb0514d81abeaf1c7015816f727f70d48617782efe07f3cad3 WatchSource:0}: Error finding container cce5d9a85bfa4ebb0514d81abeaf1c7015816f727f70d48617782efe07f3cad3: Status 404 returned error can't find the container with id cce5d9a85bfa4ebb0514d81abeaf1c7015816f727f70d48617782efe07f3cad3 Jul 06 16:38:28.029149 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:28.029135 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b7a04aec66663a9ae8155170654101.slice/crio-a4e6288293c9902bae3bff6c56377ea78aa13b5717462c990c397956375daf9d WatchSource:0}: Error finding container a4e6288293c9902bae3bff6c56377ea78aa13b5717462c990c397956375daf9d: Status 404 returned error can't find the container with id a4e6288293c9902bae3bff6c56377ea78aa13b5717462c990c397956375daf9d Jul 06 16:38:28.031663 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:28.031640 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 06 16:38:28.033690 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.033674 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 16:38:28.073403 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.073299 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-71.ec2.internal" event={"ID":"03b7a04aec66663a9ae8155170654101","Type":"ContainerStarted","Data":"a4e6288293c9902bae3bff6c56377ea78aa13b5717462c990c397956375daf9d"} Jul 06 16:38:28.074312 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.074283 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal" event={"ID":"17554f49b63e35a815e8bf1b98f97477","Type":"ContainerStarted","Data":"cce5d9a85bfa4ebb0514d81abeaf1c7015816f727f70d48617782efe07f3cad3"} Jul 06 16:38:28.132564 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:28.132527 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 06 16:38:28.233010 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:28.232977 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 06 16:38:28.333868 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:28.333769 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-71.ec2.internal\" not found" Jul 06 16:38:28.371282 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.371256 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 16:38:28.433866 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.433832 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 16:38:28.446246 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.446219 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 16:38:28.462594 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.462568 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal" Jul 06 16:38:28.487622 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.487593 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 16:38:28.488588 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.488563 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-71.ec2.internal" Jul 06 16:38:28.507452 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.507422 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 16:38:28.832494 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.832419 2574 apiserver.go:52] "Watching apiserver" Jul 06 16:38:28.841311 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.841281 2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 16:38:28.842754 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.842724 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-nnd4r","kube-system/kube-apiserver-proxy-ip-10-0-131-71.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc","openshift-cluster-node-tuning-operator/tuned-s9xmf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal","openshift-multus/multus-additional-cni-plugins-p9rnx","openshift-multus/network-metrics-daemon-n9kmw","openshift-network-diagnostics/network-check-target-j5khq","kube-system/konnectivity-agent-xswlf","openshift-dns/node-resolver-nf6xc","openshift-image-registry/node-ca-lxbqr","openshift-multus/multus-bwztk","openshift-network-operator/iptables-alerter-xtmgz","openshift-ovn-kubernetes/ovnkube-node-zwzvg"] Jul 06 16:38:28.844301 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.844280 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:38:28.844420 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:28.844356 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9kmw" podUID="39a924d0-6e9a-48bb-9972-6d40dd4c50dc" Jul 06 16:38:28.845550 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.845532 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lxbqr" Jul 06 16:38:28.846818 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.846796 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" Jul 06 16:38:28.847981 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.847941 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.849588 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.849569 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.850913 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.850893 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p9rnx" Jul 06 16:38:28.852316 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.852296 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xswlf" Jul 06 16:38:28.854781 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.854647 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xtmgz" Jul 06 16:38:28.855794 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.855776 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 16:38:28.857149 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.856134 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 16:38:28.857149 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.856407 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 16:38:28.857149 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.856956 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 16:38:28.857357 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.857208 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:38:28.857357 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:28.857290 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5khq" podUID="51bffaee-68a0-486d-8a62-a04c17b4ae9d" Jul 06 16:38:28.858734 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.858288 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nf6xc" Jul 06 16:38:28.859999 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.859973 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lvln9\"" Jul 06 16:38:28.860286 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.860269 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:28.860447 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:28.860427 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnd4r" podUID="9ce13f97-b996-4f40-86fb-e0453bd3a476" Jul 06 16:38:28.861587 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.861564 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 16:38:28.861864 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.861845 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.864070 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.863368 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 16:38:28.864070 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.864062 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 16:38:28.864493 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.864105 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 16:38:28.864493 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.864145 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 16:38:28.864493 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.864062 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 16:38:28.864493 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.864211 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 16:38:28.864493 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.864109 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 16:38:28.864755 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.864601 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 16:38:28.864755 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.864637 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 16:38:28.864958 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.864939 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 16:38:28.864958 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.864955 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 16:38:28.865074 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.864993 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 16:38:28.865074 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.865018 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 16:38:28.865074 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.865073 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 16:38:28.865220 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.865167 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 16:38:28.865323 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.865278 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 16:38:28.865424 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.865390 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-w69mj\"" Jul 06 16:38:28.865424 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.865408 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-c6pvq\"" Jul 06 16:38:28.868385 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.868366 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mmvdt\"" Jul 06 16:38:28.868588 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.868570 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-k2xmx\"" Jul 06 16:38:28.869092 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.868921 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 16:38:28.869092 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.868924 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 16:38:28.869948 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.869129 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 16:38:28.869948 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.869187 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 16:38:28.869948 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.869288 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 16:38:28.869948 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.869359 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 16:38:28.869948 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.869367 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 16:38:28.869948 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.869458 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mvpjt\"" Jul 06 16:38:28.869948 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.869534 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-td2wd\"" Jul 06 16:38:28.873272 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.873251 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa71c0b-c28a-43a4-92d7-3d617a604f55-tmp-dir\") pod \"node-resolver-nf6xc\" (UID: \"5aa71c0b-c28a-43a4-92d7-3d617a604f55\") " pod="openshift-dns/node-resolver-nf6xc" Jul 06 16:38:28.873358 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.873288 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-etc-systemd\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.873358 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.873330 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-run\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.873358 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.873350 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-sys\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.873550 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.873374 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-multus-socket-dir-parent\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.873550 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.873397 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ssc2\" (UniqueName: \"kubernetes.io/projected/563411d8-43a0-49ab-b73b-c29dec275f46-kube-api-access-9ssc2\") pod \"multus-additional-cni-plugins-p9rnx\" (UID: \"563411d8-43a0-49ab-b73b-c29dec275f46\") " pod="openshift-multus/multus-additional-cni-plugins-p9rnx" Jul 06 16:38:28.873550 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.873439 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szlc4\" (UniqueName: \"kubernetes.io/projected/51bffaee-68a0-486d-8a62-a04c17b4ae9d-kube-api-access-szlc4\") pod \"network-check-target-j5khq\" (UID: \"51bffaee-68a0-486d-8a62-a04c17b4ae9d\") " pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:38:28.873550 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.873481 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-system-cni-dir\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.873550 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.873545 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-host-run-netns\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.873805 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.873579 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf-socket-dir\") pod \"aws-ebs-csi-driver-node-7gqsc\" (UID: \"36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" Jul 06 16:38:28.873805 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.873606 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e8ac8a0d-d06c-4b68-a3d3-c47080439729-iptables-alerter-script\") pod \"iptables-alerter-xtmgz\" (UID: \"e8ac8a0d-d06c-4b68-a3d3-c47080439729\") " pod="openshift-network-operator/iptables-alerter-xtmgz" Jul 06 16:38:28.873805 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.873647 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0f6a8b1b-224d-459c-aad2-1d86d1c38c79-serviceca\") pod \"node-ca-lxbqr\" (UID: \"0f6a8b1b-224d-459c-aad2-1d86d1c38c79\") " pod="openshift-image-registry/node-ca-lxbqr" Jul 06 16:38:28.873805 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.873737 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-host-kubelet\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.873805 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.873766 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-host-slash\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.874038 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.873813 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-run-systemd\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.874038 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.873839 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-host-var-lib-cni-multus\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.874038 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.873887 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-multus-conf-dir\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.874038 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.873910 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-host-run-multus-certs\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.874038 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.873934 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ls6b\" (UniqueName: \"kubernetes.io/projected/36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf-kube-api-access-6ls6b\") pod \"aws-ebs-csi-driver-node-7gqsc\" (UID: \"36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" Jul 06 16:38:28.874038 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.873973 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/079df086-1654-4d96-8776-36ff7f1c8f13-ovnkube-script-lib\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.874038 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.873994 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/563411d8-43a0-49ab-b73b-c29dec275f46-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p9rnx\" (UID: \"563411d8-43a0-49ab-b73b-c29dec275f46\") " pod="openshift-multus/multus-additional-cni-plugins-p9rnx" Jul 06 16:38:28.874038 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.874031 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/94a73cbd-9f23-4895-9127-fe752b38a3a5-agent-certs\") pod \"konnectivity-agent-xswlf\" (UID: \"94a73cbd-9f23-4895-9127-fe752b38a3a5\") " pod="kube-system/konnectivity-agent-xswlf" Jul 06 16:38:28.874419 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.874054 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/079df086-1654-4d96-8776-36ff7f1c8f13-ovnkube-config\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.874419 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.874079 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/079df086-1654-4d96-8776-36ff7f1c8f13-ovn-node-metrics-cert\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.874419 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.874123 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/94a73cbd-9f23-4895-9127-fe752b38a3a5-konnectivity-ca\") pod \"konnectivity-agent-xswlf\" (UID: \"94a73cbd-9f23-4895-9127-fe752b38a3a5\") " pod="kube-system/konnectivity-agent-xswlf" Jul 06 16:38:28.874419 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.874146 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39a924d0-6e9a-48bb-9972-6d40dd4c50dc-metrics-certs\") pod \"network-metrics-daemon-n9kmw\" (UID: \"39a924d0-6e9a-48bb-9972-6d40dd4c50dc\") " pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:38:28.874419 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.874168 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtln9\" (UniqueName: \"kubernetes.io/projected/39a924d0-6e9a-48bb-9972-6d40dd4c50dc-kube-api-access-wtln9\") pod \"network-metrics-daemon-n9kmw\" (UID: \"39a924d0-6e9a-48bb-9972-6d40dd4c50dc\") " pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:38:28.874419 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.874182 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-host-var-lib-kubelet\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.874419 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.874206 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf-registration-dir\") pod \"aws-ebs-csi-driver-node-7gqsc\" (UID: \"36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" Jul 06 16:38:28.874419 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.874235 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf-etc-selinux\") pod \"aws-ebs-csi-driver-node-7gqsc\" (UID: \"36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" Jul 06 16:38:28.874419 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.874274 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-etc-sysconfig\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.874419 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.874326 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-host\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.874419 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.874356 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/563411d8-43a0-49ab-b73b-c29dec275f46-cnibin\") pod \"multus-additional-cni-plugins-p9rnx\" (UID: \"563411d8-43a0-49ab-b73b-c29dec275f46\") " pod="openshift-multus/multus-additional-cni-plugins-p9rnx" Jul 06 16:38:28.874419 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.874391 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/563411d8-43a0-49ab-b73b-c29dec275f46-cni-binary-copy\") pod \"multus-additional-cni-plugins-p9rnx\" (UID: \"563411d8-43a0-49ab-b73b-c29dec275f46\") " pod="openshift-multus/multus-additional-cni-plugins-p9rnx" Jul 06 16:38:28.875025 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.874433 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-multus-cni-dir\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.875025 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.874462 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-hostroot\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.875025 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.874484 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrd5d\" (UniqueName: \"kubernetes.io/projected/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-kube-api-access-xrd5d\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.875025 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.874509 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-host-run-netns\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.875025 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.874546 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqb2m\" (UniqueName: \"kubernetes.io/projected/5aa71c0b-c28a-43a4-92d7-3d617a604f55-kube-api-access-kqb2m\") pod \"node-resolver-nf6xc\" (UID: \"5aa71c0b-c28a-43a4-92d7-3d617a604f55\") " pod="openshift-dns/node-resolver-nf6xc" Jul 06 16:38:28.875025 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.874574 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-log-socket\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.875025 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.874596 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.875025 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.874634 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9ce13f97-b996-4f40-86fb-e0453bd3a476-dbus\") pod \"global-pull-secret-syncer-nnd4r\" (UID: \"9ce13f97-b996-4f40-86fb-e0453bd3a476\") " pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:28.875025 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.874676 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ce13f97-b996-4f40-86fb-e0453bd3a476-original-pull-secret\") pod \"global-pull-secret-syncer-nnd4r\" (UID: \"9ce13f97-b996-4f40-86fb-e0453bd3a476\") " pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:28.875025 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.874796 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-etc-kubernetes\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.875025 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.874823 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/563411d8-43a0-49ab-b73b-c29dec275f46-system-cni-dir\") pod \"multus-additional-cni-plugins-p9rnx\" (UID: \"563411d8-43a0-49ab-b73b-c29dec275f46\") " pod="openshift-multus/multus-additional-cni-plugins-p9rnx" Jul 06 16:38:28.875025 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.874924 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsdtc\" (UniqueName: \"kubernetes.io/projected/e8ac8a0d-d06c-4b68-a3d3-c47080439729-kube-api-access-qsdtc\") pod \"iptables-alerter-xtmgz\" (UID: \"e8ac8a0d-d06c-4b68-a3d3-c47080439729\") " pod="openshift-network-operator/iptables-alerter-xtmgz" Jul 06 16:38:28.875025 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.874951 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/413730d5-8dc6-44e3-890f-fa03e3339ada-multus-daemon-config\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.875761 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875004 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7gqsc\" (UID: \"36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" Jul 06 16:38:28.875761 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875086 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-var-lib-kubelet\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.875761 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875131 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/563411d8-43a0-49ab-b73b-c29dec275f46-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p9rnx\" (UID: \"563411d8-43a0-49ab-b73b-c29dec275f46\") " pod="openshift-multus/multus-additional-cni-plugins-p9rnx" Jul 06 16:38:28.875761 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875157 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tlwb\" (UniqueName: \"kubernetes.io/projected/0f6a8b1b-224d-459c-aad2-1d86d1c38c79-kube-api-access-4tlwb\") pod \"node-ca-lxbqr\" (UID: \"0f6a8b1b-224d-459c-aad2-1d86d1c38c79\") " pod="openshift-image-registry/node-ca-lxbqr" Jul 06 16:38:28.875761 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875182 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-run-openvswitch\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.875761 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875239 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-run-ovn\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.875761 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875273 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-node-log\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.875761 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875300 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-host-cni-netd\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.875761 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875322 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/413730d5-8dc6-44e3-890f-fa03e3339ada-cni-binary-copy\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.875761 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875343 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf-device-dir\") pod \"aws-ebs-csi-driver-node-7gqsc\" (UID: \"36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" Jul 06 16:38:28.875761 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875377 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-etc-sysctl-d\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.875761 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875435 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-etc-sysctl-conf\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.875761 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875468 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8ac8a0d-d06c-4b68-a3d3-c47080439729-host-slash\") pod \"iptables-alerter-xtmgz\" (UID: \"e8ac8a0d-d06c-4b68-a3d3-c47080439729\") " pod="openshift-network-operator/iptables-alerter-xtmgz" Jul 06 16:38:28.875761 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875493 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-host-run-k8s-cni-cncf-io\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.876423 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875549 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-host-var-lib-cni-bin\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.876423 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875574 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-tmp\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.876423 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875592 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/563411d8-43a0-49ab-b73b-c29dec275f46-os-release\") pod \"multus-additional-cni-plugins-p9rnx\" (UID: \"563411d8-43a0-49ab-b73b-c29dec275f46\") " pod="openshift-multus/multus-additional-cni-plugins-p9rnx" Jul 06 16:38:28.876423 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875626 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5aa71c0b-c28a-43a4-92d7-3d617a604f55-hosts-file\") pod \"node-resolver-nf6xc\" (UID: \"5aa71c0b-c28a-43a4-92d7-3d617a604f55\") " pod="openshift-dns/node-resolver-nf6xc" Jul 06 16:38:28.876423 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875650 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-systemd-units\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.876423 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875676 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-var-lib-openvswitch\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.876423 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875700 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-etc-openvswitch\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.876423 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875723 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9ce13f97-b996-4f40-86fb-e0453bd3a476-kubelet-config\") pod \"global-pull-secret-syncer-nnd4r\" (UID: \"9ce13f97-b996-4f40-86fb-e0453bd3a476\") " pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:28.876423 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875745 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-os-release\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.876423 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875766 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-etc-modprobe-d\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.876423 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875790 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-etc-kubernetes\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.876423 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875832 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7rsk\" (UniqueName: \"kubernetes.io/projected/079df086-1654-4d96-8776-36ff7f1c8f13-kube-api-access-d7rsk\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.876423 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875883 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-cnibin\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.876423 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875915 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qw74\" (UniqueName: \"kubernetes.io/projected/413730d5-8dc6-44e3-890f-fa03e3339ada-kube-api-access-8qw74\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.877214 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875938 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/563411d8-43a0-49ab-b73b-c29dec275f46-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p9rnx\" (UID: \"563411d8-43a0-49ab-b73b-c29dec275f46\") " pod="openshift-multus/multus-additional-cni-plugins-p9rnx" Jul 06 16:38:28.877214 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875967 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f6a8b1b-224d-459c-aad2-1d86d1c38c79-host\") pod \"node-ca-lxbqr\" (UID: \"0f6a8b1b-224d-459c-aad2-1d86d1c38c79\") " pod="openshift-image-registry/node-ca-lxbqr" Jul 06 16:38:28.877214 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.875989 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-host-run-ovn-kubernetes\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.877214 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.876011 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-host-cni-bin\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.877214 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.876033 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/079df086-1654-4d96-8776-36ff7f1c8f13-env-overrides\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.877214 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.876079 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf-sys-fs\") pod \"aws-ebs-csi-driver-node-7gqsc\" (UID: \"36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" Jul 06 16:38:28.877214 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.876103 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-lib-modules\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.877214 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.876124 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-etc-tuned\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.881752 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.881727 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5rr97\"" Jul 06 16:38:28.881752 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.881743 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 16:38:28.882349 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.882310 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-plr8h\"" Jul 06 16:38:28.906611 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.906588 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 16:38:28.923316 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.923293 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fxppm" Jul 06 16:38:28.934196 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.934175 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fxppm" Jul 06 16:38:28.958226 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.958199 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 16:38:28.976502 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.976465 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-host-run-netns\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.976502 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.976504 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf-socket-dir\") pod \"aws-ebs-csi-driver-node-7gqsc\" (UID: \"36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" Jul 06 16:38:28.976708 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.976585 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-host-run-netns\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.976708 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.976625 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e8ac8a0d-d06c-4b68-a3d3-c47080439729-iptables-alerter-script\") pod \"iptables-alerter-xtmgz\" (UID: \"e8ac8a0d-d06c-4b68-a3d3-c47080439729\") " pod="openshift-network-operator/iptables-alerter-xtmgz" Jul 06 16:38:28.976708 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.976658 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0f6a8b1b-224d-459c-aad2-1d86d1c38c79-serviceca\") pod \"node-ca-lxbqr\" (UID: \"0f6a8b1b-224d-459c-aad2-1d86d1c38c79\") " pod="openshift-image-registry/node-ca-lxbqr" Jul 06 16:38:28.976708 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.976662 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf-socket-dir\") pod \"aws-ebs-csi-driver-node-7gqsc\" (UID: \"36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" Jul 06 16:38:28.976708 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.976685 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-host-kubelet\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.976951 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.976724 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-host-slash\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.976951 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.976745 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-run-systemd\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.976951 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.976780 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-host-var-lib-cni-multus\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.976951 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.976805 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-multus-conf-dir\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.976951 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.976806 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-host-kubelet\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.976951 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.976828 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-host-run-multus-certs\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.976951 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.976858 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-host-slash\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.976951 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.976872 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-host-run-multus-certs\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.976951 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.976870 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ls6b\" (UniqueName: \"kubernetes.io/projected/36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf-kube-api-access-6ls6b\") pod \"aws-ebs-csi-driver-node-7gqsc\" (UID: \"36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" Jul 06 16:38:28.976951 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.976902 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-run-systemd\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.976951 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.976909 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/079df086-1654-4d96-8776-36ff7f1c8f13-ovnkube-script-lib\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.976951 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.976938 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-multus-conf-dir\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.976951 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.976937 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/563411d8-43a0-49ab-b73b-c29dec275f46-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p9rnx\" (UID: \"563411d8-43a0-49ab-b73b-c29dec275f46\") " pod="openshift-multus/multus-additional-cni-plugins-p9rnx" Jul 06 16:38:28.977561 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.976972 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/94a73cbd-9f23-4895-9127-fe752b38a3a5-agent-certs\") pod \"konnectivity-agent-xswlf\" (UID: \"94a73cbd-9f23-4895-9127-fe752b38a3a5\") " pod="kube-system/konnectivity-agent-xswlf" Jul 06 16:38:28.977561 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.976994 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/079df086-1654-4d96-8776-36ff7f1c8f13-ovnkube-config\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.977561 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977019 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/079df086-1654-4d96-8776-36ff7f1c8f13-ovn-node-metrics-cert\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.977561 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977042 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/94a73cbd-9f23-4895-9127-fe752b38a3a5-konnectivity-ca\") pod \"konnectivity-agent-xswlf\" (UID: \"94a73cbd-9f23-4895-9127-fe752b38a3a5\") " pod="kube-system/konnectivity-agent-xswlf" Jul 06 16:38:28.977561 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977064 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39a924d0-6e9a-48bb-9972-6d40dd4c50dc-metrics-certs\") pod \"network-metrics-daemon-n9kmw\" (UID: \"39a924d0-6e9a-48bb-9972-6d40dd4c50dc\") " pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:38:28.977561 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977089 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtln9\" (UniqueName: \"kubernetes.io/projected/39a924d0-6e9a-48bb-9972-6d40dd4c50dc-kube-api-access-wtln9\") pod \"network-metrics-daemon-n9kmw\" (UID: \"39a924d0-6e9a-48bb-9972-6d40dd4c50dc\") " pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:38:28.977561 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977112 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-host-var-lib-kubelet\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.977561 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977134 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf-registration-dir\") pod \"aws-ebs-csi-driver-node-7gqsc\" (UID: \"36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" Jul 06 16:38:28.977561 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977159 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf-etc-selinux\") pod \"aws-ebs-csi-driver-node-7gqsc\" (UID: \"36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" Jul 06 16:38:28.977561 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977162 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0f6a8b1b-224d-459c-aad2-1d86d1c38c79-serviceca\") pod \"node-ca-lxbqr\" (UID: \"0f6a8b1b-224d-459c-aad2-1d86d1c38c79\") " pod="openshift-image-registry/node-ca-lxbqr" Jul 06 16:38:28.977561 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977181 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-etc-sysconfig\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.977561 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977207 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-host\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.977561 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.976910 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-host-var-lib-cni-multus\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.977561 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977238 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/563411d8-43a0-49ab-b73b-c29dec275f46-cnibin\") pod \"multus-additional-cni-plugins-p9rnx\" (UID: \"563411d8-43a0-49ab-b73b-c29dec275f46\") " pod="openshift-multus/multus-additional-cni-plugins-p9rnx" Jul 06 16:38:28.978223 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977258 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e8ac8a0d-d06c-4b68-a3d3-c47080439729-iptables-alerter-script\") pod \"iptables-alerter-xtmgz\" (UID: \"e8ac8a0d-d06c-4b68-a3d3-c47080439729\") " pod="openshift-network-operator/iptables-alerter-xtmgz" Jul 06 16:38:28.978223 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977266 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/563411d8-43a0-49ab-b73b-c29dec275f46-cni-binary-copy\") pod \"multus-additional-cni-plugins-p9rnx\" (UID: \"563411d8-43a0-49ab-b73b-c29dec275f46\") " pod="openshift-multus/multus-additional-cni-plugins-p9rnx" Jul 06 16:38:28.978223 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977294 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-multus-cni-dir\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.978223 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977316 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-hostroot\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.978223 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977339 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrd5d\" (UniqueName: \"kubernetes.io/projected/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-kube-api-access-xrd5d\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.978223 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977365 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-host-run-netns\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.978223 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977389 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqb2m\" (UniqueName: \"kubernetes.io/projected/5aa71c0b-c28a-43a4-92d7-3d617a604f55-kube-api-access-kqb2m\") pod \"node-resolver-nf6xc\" (UID: \"5aa71c0b-c28a-43a4-92d7-3d617a604f55\") " pod="openshift-dns/node-resolver-nf6xc" Jul 06 16:38:28.978223 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977412 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-log-socket\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.978223 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977261 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-host\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.978223 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977454 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/563411d8-43a0-49ab-b73b-c29dec275f46-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p9rnx\" (UID: \"563411d8-43a0-49ab-b73b-c29dec275f46\") " pod="openshift-multus/multus-additional-cni-plugins-p9rnx" Jul 06 16:38:28.978223 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977439 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.978223 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977496 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9ce13f97-b996-4f40-86fb-e0453bd3a476-dbus\") pod \"global-pull-secret-syncer-nnd4r\" (UID: \"9ce13f97-b996-4f40-86fb-e0453bd3a476\") " pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:28.978223 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977546 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/563411d8-43a0-49ab-b73b-c29dec275f46-cnibin\") pod \"multus-additional-cni-plugins-p9rnx\" (UID: \"563411d8-43a0-49ab-b73b-c29dec275f46\") " pod="openshift-multus/multus-additional-cni-plugins-p9rnx" Jul 06 16:38:28.978223 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977582 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 16:38:28.978223 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977813 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/079df086-1654-4d96-8776-36ff7f1c8f13-ovnkube-script-lib\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.978953 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977885 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-host-run-netns\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.978953 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977948 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.978953 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977983 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ce13f97-b996-4f40-86fb-e0453bd3a476-original-pull-secret\") pod \"global-pull-secret-syncer-nnd4r\" (UID: \"9ce13f97-b996-4f40-86fb-e0453bd3a476\") " pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:28.978953 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.977998 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/563411d8-43a0-49ab-b73b-c29dec275f46-cni-binary-copy\") pod \"multus-additional-cni-plugins-p9rnx\" (UID: \"563411d8-43a0-49ab-b73b-c29dec275f46\") " pod="openshift-multus/multus-additional-cni-plugins-p9rnx" Jul 06 16:38:28.978953 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978025 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-etc-kubernetes\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.978953 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978051 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/563411d8-43a0-49ab-b73b-c29dec275f46-system-cni-dir\") pod \"multus-additional-cni-plugins-p9rnx\" (UID: \"563411d8-43a0-49ab-b73b-c29dec275f46\") " pod="openshift-multus/multus-additional-cni-plugins-p9rnx" Jul 06 16:38:28.978953 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978073 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsdtc\" (UniqueName: \"kubernetes.io/projected/e8ac8a0d-d06c-4b68-a3d3-c47080439729-kube-api-access-qsdtc\") pod \"iptables-alerter-xtmgz\" (UID: \"e8ac8a0d-d06c-4b68-a3d3-c47080439729\") " pod="openshift-network-operator/iptables-alerter-xtmgz" Jul 06 16:38:28.978953 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978099 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/413730d5-8dc6-44e3-890f-fa03e3339ada-multus-daemon-config\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.978953 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978102 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/94a73cbd-9f23-4895-9127-fe752b38a3a5-konnectivity-ca\") pod \"konnectivity-agent-xswlf\" (UID: \"94a73cbd-9f23-4895-9127-fe752b38a3a5\") " pod="kube-system/konnectivity-agent-xswlf" Jul 06 16:38:28.978953 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978126 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-host-var-lib-kubelet\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.978953 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978120 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7gqsc\" (UID: \"36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" Jul 06 16:38:28.978953 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978158 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-var-lib-kubelet\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.978953 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978171 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-multus-cni-dir\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.978953 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978180 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/563411d8-43a0-49ab-b73b-c29dec275f46-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p9rnx\" (UID: \"563411d8-43a0-49ab-b73b-c29dec275f46\") " pod="openshift-multus/multus-additional-cni-plugins-p9rnx" Jul 06 16:38:28.979636 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978187 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-log-socket\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.979636 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978204 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tlwb\" (UniqueName: \"kubernetes.io/projected/0f6a8b1b-224d-459c-aad2-1d86d1c38c79-kube-api-access-4tlwb\") pod \"node-ca-lxbqr\" (UID: \"0f6a8b1b-224d-459c-aad2-1d86d1c38c79\") " pod="openshift-image-registry/node-ca-lxbqr" Jul 06 16:38:28.979636 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978235 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/563411d8-43a0-49ab-b73b-c29dec275f46-system-cni-dir\") pod \"multus-additional-cni-plugins-p9rnx\" (UID: \"563411d8-43a0-49ab-b73b-c29dec275f46\") " pod="openshift-multus/multus-additional-cni-plugins-p9rnx" Jul 06 16:38:28.979636 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978240 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-run-openvswitch\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.979636 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978249 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-etc-sysconfig\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.979636 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:28.978267 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:38:28.979636 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978275 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-run-ovn\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.979636 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978287 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-etc-kubernetes\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.979636 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978298 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-node-log\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.979636 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978319 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-hostroot\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.979636 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978336 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-node-log\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.979636 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:28.978347 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39a924d0-6e9a-48bb-9972-6d40dd4c50dc-metrics-certs podName:39a924d0-6e9a-48bb-9972-6d40dd4c50dc nodeName:}" failed. No retries permitted until 2026-07-06 16:38:29.478307404 +0000 UTC m=+2.997798504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39a924d0-6e9a-48bb-9972-6d40dd4c50dc-metrics-certs") pod "network-metrics-daemon-n9kmw" (UID: "39a924d0-6e9a-48bb-9972-6d40dd4c50dc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:38:28.979636 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978357 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-run-ovn\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.979636 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978367 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-host-cni-netd\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.979636 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978372 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9ce13f97-b996-4f40-86fb-e0453bd3a476-dbus\") pod \"global-pull-secret-syncer-nnd4r\" (UID: \"9ce13f97-b996-4f40-86fb-e0453bd3a476\") " pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:28.980335 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978395 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/413730d5-8dc6-44e3-890f-fa03e3339ada-cni-binary-copy\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.980335 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978412 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-host-cni-netd\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.980335 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978439 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf-device-dir\") pod \"aws-ebs-csi-driver-node-7gqsc\" (UID: \"36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" Jul 06 16:38:28.980335 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978463 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-etc-sysctl-d\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.980335 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978483 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-etc-sysctl-conf\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.980335 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:28.978494 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 16:38:28.980335 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978508 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf-etc-selinux\") pod \"aws-ebs-csi-driver-node-7gqsc\" (UID: \"36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" Jul 06 16:38:28.980335 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978509 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8ac8a0d-d06c-4b68-a3d3-c47080439729-host-slash\") pod \"iptables-alerter-xtmgz\" (UID: \"e8ac8a0d-d06c-4b68-a3d3-c47080439729\") " pod="openshift-network-operator/iptables-alerter-xtmgz" Jul 06 16:38:28.980335 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978180 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf-registration-dir\") pod \"aws-ebs-csi-driver-node-7gqsc\" (UID: \"36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" Jul 06 16:38:28.980335 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:28.978561 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce13f97-b996-4f40-86fb-e0453bd3a476-original-pull-secret podName:9ce13f97-b996-4f40-86fb-e0453bd3a476 nodeName:}" failed. No retries permitted until 2026-07-06 16:38:29.478545943 +0000 UTC m=+2.998037040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9ce13f97-b996-4f40-86fb-e0453bd3a476-original-pull-secret") pod "global-pull-secret-syncer-nnd4r" (UID: "9ce13f97-b996-4f40-86fb-e0453bd3a476") : object "kube-system"/"original-pull-secret" not registered Jul 06 16:38:28.980335 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978586 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-host-run-k8s-cni-cncf-io\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.980335 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978595 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-run-openvswitch\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.980335 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978618 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-host-var-lib-cni-bin\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.980335 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978641 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-tmp\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.980925 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978665 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/563411d8-43a0-49ab-b73b-c29dec275f46-os-release\") pod \"multus-additional-cni-plugins-p9rnx\" (UID: \"563411d8-43a0-49ab-b73b-c29dec275f46\") " pod="openshift-multus/multus-additional-cni-plugins-p9rnx" Jul 06 16:38:28.980925 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978690 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5aa71c0b-c28a-43a4-92d7-3d617a604f55-hosts-file\") pod \"node-resolver-nf6xc\" (UID: \"5aa71c0b-c28a-43a4-92d7-3d617a604f55\") " pod="openshift-dns/node-resolver-nf6xc" Jul 06 16:38:28.980925 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978715 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-systemd-units\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.980925 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978740 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-var-lib-openvswitch\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.980925 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978751 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-etc-sysctl-d\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.980925 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978767 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-etc-openvswitch\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.980925 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978769 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/563411d8-43a0-49ab-b73b-c29dec275f46-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p9rnx\" (UID: \"563411d8-43a0-49ab-b73b-c29dec275f46\") " pod="openshift-multus/multus-additional-cni-plugins-p9rnx" Jul 06 16:38:28.980925 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978792 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9ce13f97-b996-4f40-86fb-e0453bd3a476-kubelet-config\") pod \"global-pull-secret-syncer-nnd4r\" (UID: \"9ce13f97-b996-4f40-86fb-e0453bd3a476\") " pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:28.980925 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978812 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf-device-dir\") pod \"aws-ebs-csi-driver-node-7gqsc\" (UID: \"36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" Jul 06 16:38:28.980925 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978852 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9ce13f97-b996-4f40-86fb-e0453bd3a476-kubelet-config\") pod \"global-pull-secret-syncer-nnd4r\" (UID: \"9ce13f97-b996-4f40-86fb-e0453bd3a476\") " pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:28.980925 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978914 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/563411d8-43a0-49ab-b73b-c29dec275f46-os-release\") pod \"multus-additional-cni-plugins-p9rnx\" (UID: \"563411d8-43a0-49ab-b73b-c29dec275f46\") " pod="openshift-multus/multus-additional-cni-plugins-p9rnx" Jul 06 16:38:28.980925 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978957 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5aa71c0b-c28a-43a4-92d7-3d617a604f55-hosts-file\") pod \"node-resolver-nf6xc\" (UID: \"5aa71c0b-c28a-43a4-92d7-3d617a604f55\") " pod="openshift-dns/node-resolver-nf6xc" Jul 06 16:38:28.980925 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978991 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-systemd-units\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.980925 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979032 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-var-lib-openvswitch\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.981494 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979082 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-etc-openvswitch\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.981494 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979082 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/079df086-1654-4d96-8776-36ff7f1c8f13-ovnkube-config\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.981494 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979082 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/413730d5-8dc6-44e3-890f-fa03e3339ada-cni-binary-copy\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.981494 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979132 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7gqsc\" (UID: \"36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" Jul 06 16:38:28.981494 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979143 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-host-run-k8s-cni-cncf-io\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.981494 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.978561 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e8ac8a0d-d06c-4b68-a3d3-c47080439729-host-slash\") pod \"iptables-alerter-xtmgz\" (UID: \"e8ac8a0d-d06c-4b68-a3d3-c47080439729\") " pod="openshift-network-operator/iptables-alerter-xtmgz" Jul 06 16:38:28.981494 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979159 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-var-lib-kubelet\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.981494 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979179 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-etc-sysctl-conf\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.981494 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979186 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-host-var-lib-cni-bin\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.981494 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979188 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/413730d5-8dc6-44e3-890f-fa03e3339ada-multus-daemon-config\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.981494 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979180 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-os-release\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.981494 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979220 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-os-release\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.981494 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979240 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-etc-modprobe-d\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.981494 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979265 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-etc-kubernetes\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.981494 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979310 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7rsk\" (UniqueName: \"kubernetes.io/projected/079df086-1654-4d96-8776-36ff7f1c8f13-kube-api-access-d7rsk\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.982251 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979334 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-cnibin\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.982251 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979339 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-etc-modprobe-d\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.982251 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979342 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-etc-kubernetes\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.982251 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979376 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qw74\" (UniqueName: \"kubernetes.io/projected/413730d5-8dc6-44e3-890f-fa03e3339ada-kube-api-access-8qw74\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.982251 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979390 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-cnibin\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.982251 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979405 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/563411d8-43a0-49ab-b73b-c29dec275f46-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p9rnx\" (UID: \"563411d8-43a0-49ab-b73b-c29dec275f46\") " pod="openshift-multus/multus-additional-cni-plugins-p9rnx" Jul 06 16:38:28.982251 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979430 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f6a8b1b-224d-459c-aad2-1d86d1c38c79-host\") pod \"node-ca-lxbqr\" (UID: \"0f6a8b1b-224d-459c-aad2-1d86d1c38c79\") " pod="openshift-image-registry/node-ca-lxbqr" Jul 06 16:38:28.982251 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979455 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-host-run-ovn-kubernetes\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.982251 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979478 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-host-cni-bin\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.982251 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979501 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/079df086-1654-4d96-8776-36ff7f1c8f13-env-overrides\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.982251 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979541 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf-sys-fs\") pod \"aws-ebs-csi-driver-node-7gqsc\" (UID: \"36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" Jul 06 16:38:28.982251 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979564 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-lib-modules\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.982251 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979574 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-host-run-ovn-kubernetes\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.982251 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979586 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-etc-tuned\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.982251 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979609 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa71c0b-c28a-43a4-92d7-3d617a604f55-tmp-dir\") pod \"node-resolver-nf6xc\" (UID: \"5aa71c0b-c28a-43a4-92d7-3d617a604f55\") " pod="openshift-dns/node-resolver-nf6xc" Jul 06 16:38:28.982986 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979634 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-etc-systemd\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.982986 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979655 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-run\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.982986 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979656 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf-sys-fs\") pod \"aws-ebs-csi-driver-node-7gqsc\" (UID: \"36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" Jul 06 16:38:28.982986 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979693 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-sys\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.982986 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979705 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/563411d8-43a0-49ab-b73b-c29dec275f46-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p9rnx\" (UID: \"563411d8-43a0-49ab-b73b-c29dec275f46\") " pod="openshift-multus/multus-additional-cni-plugins-p9rnx" Jul 06 16:38:28.982986 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979713 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-run\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.982986 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979720 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-multus-socket-dir-parent\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.982986 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979749 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ssc2\" (UniqueName: \"kubernetes.io/projected/563411d8-43a0-49ab-b73b-c29dec275f46-kube-api-access-9ssc2\") pod \"multus-additional-cni-plugins-p9rnx\" (UID: \"563411d8-43a0-49ab-b73b-c29dec275f46\") " pod="openshift-multus/multus-additional-cni-plugins-p9rnx" Jul 06 16:38:28.982986 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979778 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szlc4\" (UniqueName: \"kubernetes.io/projected/51bffaee-68a0-486d-8a62-a04c17b4ae9d-kube-api-access-szlc4\") pod \"network-check-target-j5khq\" (UID: \"51bffaee-68a0-486d-8a62-a04c17b4ae9d\") " pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:38:28.982986 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979801 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-system-cni-dir\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.982986 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979875 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-system-cni-dir\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.982986 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979922 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/079df086-1654-4d96-8776-36ff7f1c8f13-host-cni-bin\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.982986 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979987 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa71c0b-c28a-43a4-92d7-3d617a604f55-tmp-dir\") pod \"node-resolver-nf6xc\" (UID: \"5aa71c0b-c28a-43a4-92d7-3d617a604f55\") " pod="openshift-dns/node-resolver-nf6xc" Jul 06 16:38:28.982986 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.980012 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-lib-modules\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.982986 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.980085 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-etc-systemd\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.983701 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.979754 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f6a8b1b-224d-459c-aad2-1d86d1c38c79-host\") pod \"node-ca-lxbqr\" (UID: \"0f6a8b1b-224d-459c-aad2-1d86d1c38c79\") " pod="openshift-image-registry/node-ca-lxbqr" Jul 06 16:38:28.983701 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.980126 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-sys\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.983701 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.980170 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/413730d5-8dc6-44e3-890f-fa03e3339ada-multus-socket-dir-parent\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:28.983701 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.980426 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/079df086-1654-4d96-8776-36ff7f1c8f13-env-overrides\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.983701 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.981556 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/079df086-1654-4d96-8776-36ff7f1c8f13-ovn-node-metrics-cert\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:28.983701 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.981707 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/94a73cbd-9f23-4895-9127-fe752b38a3a5-agent-certs\") pod \"konnectivity-agent-xswlf\" (UID: \"94a73cbd-9f23-4895-9127-fe752b38a3a5\") " pod="kube-system/konnectivity-agent-xswlf" Jul 06 16:38:28.983701 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.982007 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-etc-tuned\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:28.983701 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:28.982057 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-tmp\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:29.012473 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:29.012436 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ls6b\" (UniqueName: \"kubernetes.io/projected/36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf-kube-api-access-6ls6b\") pod \"aws-ebs-csi-driver-node-7gqsc\" (UID: \"36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" Jul 06 16:38:29.025820 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:29.025768 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 16:38:29.025820 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:29.025802 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 16:38:29.025820 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:29.025820 2574 projected.go:196] Error preparing data for projected volume kube-api-access-szlc4 for pod openshift-network-diagnostics/network-check-target-j5khq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:38:29.026569 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:29.025902 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51bffaee-68a0-486d-8a62-a04c17b4ae9d-kube-api-access-szlc4 podName:51bffaee-68a0-486d-8a62-a04c17b4ae9d nodeName:}" failed. No retries permitted until 2026-07-06 16:38:29.52588048 +0000 UTC m=+3.045371609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-szlc4" (UniqueName: "kubernetes.io/projected/51bffaee-68a0-486d-8a62-a04c17b4ae9d-kube-api-access-szlc4") pod "network-check-target-j5khq" (UID: "51bffaee-68a0-486d-8a62-a04c17b4ae9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:38:29.027337 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:29.027310 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qw74\" (UniqueName: \"kubernetes.io/projected/413730d5-8dc6-44e3-890f-fa03e3339ada-kube-api-access-8qw74\") pod \"multus-bwztk\" (UID: \"413730d5-8dc6-44e3-890f-fa03e3339ada\") " pod="openshift-multus/multus-bwztk" Jul 06 16:38:29.027456 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:29.027433 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ssc2\" (UniqueName: \"kubernetes.io/projected/563411d8-43a0-49ab-b73b-c29dec275f46-kube-api-access-9ssc2\") pod \"multus-additional-cni-plugins-p9rnx\" (UID: \"563411d8-43a0-49ab-b73b-c29dec275f46\") " pod="openshift-multus/multus-additional-cni-plugins-p9rnx" Jul 06 16:38:29.027540 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:29.027459 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7rsk\" (UniqueName: \"kubernetes.io/projected/079df086-1654-4d96-8776-36ff7f1c8f13-kube-api-access-d7rsk\") pod \"ovnkube-node-zwzvg\" (UID: \"079df086-1654-4d96-8776-36ff7f1c8f13\") " pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:29.027540 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:29.027498 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tlwb\" (UniqueName: \"kubernetes.io/projected/0f6a8b1b-224d-459c-aad2-1d86d1c38c79-kube-api-access-4tlwb\") pod \"node-ca-lxbqr\" (UID: \"0f6a8b1b-224d-459c-aad2-1d86d1c38c79\") " pod="openshift-image-registry/node-ca-lxbqr" Jul 06 16:38:29.028977 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:29.028948 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqb2m\" (UniqueName: \"kubernetes.io/projected/5aa71c0b-c28a-43a4-92d7-3d617a604f55-kube-api-access-kqb2m\") pod \"node-resolver-nf6xc\" (UID: \"5aa71c0b-c28a-43a4-92d7-3d617a604f55\") " pod="openshift-dns/node-resolver-nf6xc" Jul 06 16:38:29.029545 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:29.029508 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsdtc\" (UniqueName: \"kubernetes.io/projected/e8ac8a0d-d06c-4b68-a3d3-c47080439729-kube-api-access-qsdtc\") pod \"iptables-alerter-xtmgz\" (UID: \"e8ac8a0d-d06c-4b68-a3d3-c47080439729\") " pod="openshift-network-operator/iptables-alerter-xtmgz" Jul 06 16:38:29.029697 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:29.029672 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrd5d\" (UniqueName: \"kubernetes.io/projected/2a168e85-8f35-497e-bc2e-ba3a5bc8396d-kube-api-access-xrd5d\") pod \"tuned-s9xmf\" (UID: \"2a168e85-8f35-497e-bc2e-ba3a5bc8396d\") " pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:29.033864 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:29.033847 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtln9\" (UniqueName: \"kubernetes.io/projected/39a924d0-6e9a-48bb-9972-6d40dd4c50dc-kube-api-access-wtln9\") pod \"network-metrics-daemon-n9kmw\" (UID: \"39a924d0-6e9a-48bb-9972-6d40dd4c50dc\") " pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:38:29.160326 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:29.160245 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lxbqr" Jul 06 16:38:29.167869 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:29.167844 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" Jul 06 16:38:29.178384 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:29.178358 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" Jul 06 16:38:29.184238 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:29.184215 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:29.191811 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:29.191793 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p9rnx" Jul 06 16:38:29.198588 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:29.198570 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xswlf" Jul 06 16:38:29.209506 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:29.209486 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xtmgz" Jul 06 16:38:29.215334 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:29.215316 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nf6xc" Jul 06 16:38:29.219814 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:29.219790 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bwztk" Jul 06 16:38:29.484241 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:29.484155 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39a924d0-6e9a-48bb-9972-6d40dd4c50dc-metrics-certs\") pod \"network-metrics-daemon-n9kmw\" (UID: \"39a924d0-6e9a-48bb-9972-6d40dd4c50dc\") " pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:38:29.484241 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:29.484208 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ce13f97-b996-4f40-86fb-e0453bd3a476-original-pull-secret\") pod \"global-pull-secret-syncer-nnd4r\" (UID: \"9ce13f97-b996-4f40-86fb-e0453bd3a476\") " pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:29.484449 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:29.484300 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:38:29.484449 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:29.484319 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 16:38:29.484449 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:29.484371 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39a924d0-6e9a-48bb-9972-6d40dd4c50dc-metrics-certs podName:39a924d0-6e9a-48bb-9972-6d40dd4c50dc nodeName:}" failed. No retries permitted until 2026-07-06 16:38:30.4843515 +0000 UTC m=+4.003842597 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39a924d0-6e9a-48bb-9972-6d40dd4c50dc-metrics-certs") pod "network-metrics-daemon-n9kmw" (UID: "39a924d0-6e9a-48bb-9972-6d40dd4c50dc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:38:29.484449 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:29.484390 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce13f97-b996-4f40-86fb-e0453bd3a476-original-pull-secret podName:9ce13f97-b996-4f40-86fb-e0453bd3a476 nodeName:}" failed. No retries permitted until 2026-07-06 16:38:30.4843812 +0000 UTC m=+4.003872296 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9ce13f97-b996-4f40-86fb-e0453bd3a476-original-pull-secret") pod "global-pull-secret-syncer-nnd4r" (UID: "9ce13f97-b996-4f40-86fb-e0453bd3a476") : object "kube-system"/"original-pull-secret" not registered Jul 06 16:38:29.584669 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:29.584638 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szlc4\" (UniqueName: \"kubernetes.io/projected/51bffaee-68a0-486d-8a62-a04c17b4ae9d-kube-api-access-szlc4\") pod \"network-check-target-j5khq\" (UID: \"51bffaee-68a0-486d-8a62-a04c17b4ae9d\") " pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:38:29.584801 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:29.584755 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 16:38:29.584801 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:29.584771 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 16:38:29.584801 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:29.584779 2574 projected.go:196] Error preparing data for projected volume kube-api-access-szlc4 for pod openshift-network-diagnostics/network-check-target-j5khq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:38:29.584929 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:29.584828 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51bffaee-68a0-486d-8a62-a04c17b4ae9d-kube-api-access-szlc4 podName:51bffaee-68a0-486d-8a62-a04c17b4ae9d nodeName:}" failed. No retries permitted until 2026-07-06 16:38:30.584815131 +0000 UTC m=+4.104306224 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-szlc4" (UniqueName: "kubernetes.io/projected/51bffaee-68a0-486d-8a62-a04c17b4ae9d-kube-api-access-szlc4") pod "network-check-target-j5khq" (UID: "51bffaee-68a0-486d-8a62-a04c17b4ae9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:38:29.936537 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:29.936466 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 16:33:28 +0000 UTC" deadline="2028-02-23 17:06:06.820518503 +0000 UTC" Jul 06 16:38:29.936963 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:29.936539 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14328h27m36.883985313s" Jul 06 16:38:29.940485 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:29.940454 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d8a2a0_2014_4a6f_b4b6_94a844fa8dcf.slice/crio-3a621aa61a15beae43a437a51fdf3a9aafd4a7cc71e0aec9b12d89c38da3238f WatchSource:0}: Error finding container 3a621aa61a15beae43a437a51fdf3a9aafd4a7cc71e0aec9b12d89c38da3238f: Status 404 returned error can't find the container with id 3a621aa61a15beae43a437a51fdf3a9aafd4a7cc71e0aec9b12d89c38da3238f Jul 06 16:38:29.942022 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:29.941996 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079df086_1654_4d96_8776_36ff7f1c8f13.slice/crio-f8768fb14a9a7efe145cbd861a116c4a20d5a0c1a440008367d3296c5fbe0477 WatchSource:0}: Error finding container f8768fb14a9a7efe145cbd861a116c4a20d5a0c1a440008367d3296c5fbe0477: Status 404 returned error can't find the container with id f8768fb14a9a7efe145cbd861a116c4a20d5a0c1a440008367d3296c5fbe0477 Jul 06 16:38:29.945396 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:29.945301 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413730d5_8dc6_44e3_890f_fa03e3339ada.slice/crio-c8a2f53af270fbe171d010db71c81483a6635e55a70cea950173acd1cf89c6fe WatchSource:0}: Error finding container c8a2f53af270fbe171d010db71c81483a6635e55a70cea950173acd1cf89c6fe: Status 404 returned error can't find the container with id c8a2f53af270fbe171d010db71c81483a6635e55a70cea950173acd1cf89c6fe Jul 06 16:38:29.947985 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:29.947961 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a73cbd_9f23_4895_9127_fe752b38a3a5.slice/crio-f49e852e2373b58febe4e2da0984aa30cd4926f43c6ba49de1ecfe13cfc3e60b WatchSource:0}: Error finding container f49e852e2373b58febe4e2da0984aa30cd4926f43c6ba49de1ecfe13cfc3e60b: Status 404 returned error can't find the container with id f49e852e2373b58febe4e2da0984aa30cd4926f43c6ba49de1ecfe13cfc3e60b Jul 06 16:38:29.948458 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:29.948436 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ac8a0d_d06c_4b68_a3d3_c47080439729.slice/crio-def7e010e54765ebd9429f322b99468faca574544083aebcd820b21af760e17f WatchSource:0}: Error finding container def7e010e54765ebd9429f322b99468faca574544083aebcd820b21af760e17f: Status 404 returned error can't find the container with id def7e010e54765ebd9429f322b99468faca574544083aebcd820b21af760e17f Jul 06 16:38:29.949443 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:29.949238 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6a8b1b_224d_459c_aad2_1d86d1c38c79.slice/crio-e290d57f4dcf3bd1cd534be2be02d4e710849f68a213e802f89aa0636ca22c07 WatchSource:0}: Error finding container e290d57f4dcf3bd1cd534be2be02d4e710849f68a213e802f89aa0636ca22c07: Status 404 returned error can't find the container with id e290d57f4dcf3bd1cd534be2be02d4e710849f68a213e802f89aa0636ca22c07 Jul 06 16:38:29.950560 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:29.950484 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563411d8_43a0_49ab_b73b_c29dec275f46.slice/crio-4ea9180e0c0096ff0903bb9c50c67607a3e13e4a4170b99a5876dcb3fc22ca10 WatchSource:0}: Error finding container 4ea9180e0c0096ff0903bb9c50c67607a3e13e4a4170b99a5876dcb3fc22ca10: Status 404 returned error can't find the container with id 4ea9180e0c0096ff0903bb9c50c67607a3e13e4a4170b99a5876dcb3fc22ca10 Jul 06 16:38:29.951055 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:29.950777 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a168e85_8f35_497e_bc2e_ba3a5bc8396d.slice/crio-691316dd9580b36eb3a1cb516729a74b53fb0fc185de261871ce0a1ee5385065 WatchSource:0}: Error finding container 691316dd9580b36eb3a1cb516729a74b53fb0fc185de261871ce0a1ee5385065: Status 404 returned error can't find the container with id 691316dd9580b36eb3a1cb516729a74b53fb0fc185de261871ce0a1ee5385065 Jul 06 16:38:29.953073 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:38:29.953048 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa71c0b_c28a_43a4_92d7_3d617a604f55.slice/crio-1689e942391a4ff50c0199b8816119308fff13663cc238d9948f1bd9277d810b WatchSource:0}: Error finding container 1689e942391a4ff50c0199b8816119308fff13663cc238d9948f1bd9277d810b: Status 404 returned error can't find the container with id 1689e942391a4ff50c0199b8816119308fff13663cc238d9948f1bd9277d810b Jul 06 16:38:30.070085 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:30.069903 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:30.070253 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:30.069991 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:38:30.070253 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:30.070167 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnd4r" podUID="9ce13f97-b996-4f40-86fb-e0453bd3a476" Jul 06 16:38:30.070451 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:30.070273 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9kmw" podUID="39a924d0-6e9a-48bb-9972-6d40dd4c50dc" Jul 06 16:38:30.078907 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:30.078882 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bwztk" event={"ID":"413730d5-8dc6-44e3-890f-fa03e3339ada","Type":"ContainerStarted","Data":"c8a2f53af270fbe171d010db71c81483a6635e55a70cea950173acd1cf89c6fe"} Jul 06 16:38:30.079861 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:30.079838 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" event={"ID":"36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf","Type":"ContainerStarted","Data":"3a621aa61a15beae43a437a51fdf3a9aafd4a7cc71e0aec9b12d89c38da3238f"} Jul 06 16:38:30.081051 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:30.081023 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" event={"ID":"2a168e85-8f35-497e-bc2e-ba3a5bc8396d","Type":"ContainerStarted","Data":"691316dd9580b36eb3a1cb516729a74b53fb0fc185de261871ce0a1ee5385065"} Jul 06 16:38:30.082122 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:30.082095 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p9rnx" event={"ID":"563411d8-43a0-49ab-b73b-c29dec275f46","Type":"ContainerStarted","Data":"4ea9180e0c0096ff0903bb9c50c67607a3e13e4a4170b99a5876dcb3fc22ca10"} Jul 06 16:38:30.083276 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:30.083251 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xtmgz" event={"ID":"e8ac8a0d-d06c-4b68-a3d3-c47080439729","Type":"ContainerStarted","Data":"def7e010e54765ebd9429f322b99468faca574544083aebcd820b21af760e17f"} Jul 06 16:38:30.084446 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:30.084426 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xswlf" event={"ID":"94a73cbd-9f23-4895-9127-fe752b38a3a5","Type":"ContainerStarted","Data":"f49e852e2373b58febe4e2da0984aa30cd4926f43c6ba49de1ecfe13cfc3e60b"} Jul 06 16:38:30.085717 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:30.085679 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" event={"ID":"079df086-1654-4d96-8776-36ff7f1c8f13","Type":"ContainerStarted","Data":"f8768fb14a9a7efe145cbd861a116c4a20d5a0c1a440008367d3296c5fbe0477"} Jul 06 16:38:30.087609 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:30.087589 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-71.ec2.internal" event={"ID":"03b7a04aec66663a9ae8155170654101","Type":"ContainerStarted","Data":"e87a0a7e9f22851251fd2377636ebc2cc31721865437003bda9b9010638cfdf8"} Jul 06 16:38:30.088729 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:30.088707 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nf6xc" event={"ID":"5aa71c0b-c28a-43a4-92d7-3d617a604f55","Type":"ContainerStarted","Data":"1689e942391a4ff50c0199b8816119308fff13663cc238d9948f1bd9277d810b"} Jul 06 16:38:30.089876 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:30.089858 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lxbqr" event={"ID":"0f6a8b1b-224d-459c-aad2-1d86d1c38c79","Type":"ContainerStarted","Data":"e290d57f4dcf3bd1cd534be2be02d4e710849f68a213e802f89aa0636ca22c07"} Jul 06 16:38:30.492575 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:30.492455 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39a924d0-6e9a-48bb-9972-6d40dd4c50dc-metrics-certs\") pod \"network-metrics-daemon-n9kmw\" (UID: \"39a924d0-6e9a-48bb-9972-6d40dd4c50dc\") " pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:38:30.492575 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:30.492532 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ce13f97-b996-4f40-86fb-e0453bd3a476-original-pull-secret\") pod \"global-pull-secret-syncer-nnd4r\" (UID: \"9ce13f97-b996-4f40-86fb-e0453bd3a476\") " pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:30.492788 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:30.492667 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 16:38:30.492788 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:30.492670 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:38:30.492788 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:30.492727 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce13f97-b996-4f40-86fb-e0453bd3a476-original-pull-secret podName:9ce13f97-b996-4f40-86fb-e0453bd3a476 nodeName:}" failed. No retries permitted until 2026-07-06 16:38:32.492709806 +0000 UTC m=+6.012200913 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9ce13f97-b996-4f40-86fb-e0453bd3a476-original-pull-secret") pod "global-pull-secret-syncer-nnd4r" (UID: "9ce13f97-b996-4f40-86fb-e0453bd3a476") : object "kube-system"/"original-pull-secret" not registered Jul 06 16:38:30.492788 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:30.492746 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39a924d0-6e9a-48bb-9972-6d40dd4c50dc-metrics-certs podName:39a924d0-6e9a-48bb-9972-6d40dd4c50dc nodeName:}" failed. No retries permitted until 2026-07-06 16:38:32.49273667 +0000 UTC m=+6.012227771 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39a924d0-6e9a-48bb-9972-6d40dd4c50dc-metrics-certs") pod "network-metrics-daemon-n9kmw" (UID: "39a924d0-6e9a-48bb-9972-6d40dd4c50dc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:38:30.594579 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:30.593277 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szlc4\" (UniqueName: \"kubernetes.io/projected/51bffaee-68a0-486d-8a62-a04c17b4ae9d-kube-api-access-szlc4\") pod \"network-check-target-j5khq\" (UID: \"51bffaee-68a0-486d-8a62-a04c17b4ae9d\") " pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:38:30.594579 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:30.593428 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 16:38:30.594579 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:30.593448 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 16:38:30.594579 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:30.593461 2574 projected.go:196] Error preparing data for projected volume kube-api-access-szlc4 for pod openshift-network-diagnostics/network-check-target-j5khq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:38:30.594579 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:30.593533 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51bffaee-68a0-486d-8a62-a04c17b4ae9d-kube-api-access-szlc4 podName:51bffaee-68a0-486d-8a62-a04c17b4ae9d nodeName:}" failed. No retries permitted until 2026-07-06 16:38:32.593502196 +0000 UTC m=+6.112993312 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-szlc4" (UniqueName: "kubernetes.io/projected/51bffaee-68a0-486d-8a62-a04c17b4ae9d-kube-api-access-szlc4") pod "network-check-target-j5khq" (UID: "51bffaee-68a0-486d-8a62-a04c17b4ae9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:38:30.937432 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:30.937340 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 16:33:28 +0000 UTC" deadline="2028-04-19 00:41:40.374160008 +0000 UTC" Jul 06 16:38:30.937432 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:30.937386 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15656h3m9.436780484s" Jul 06 16:38:31.073156 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:31.073128 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:38:31.073305 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:31.073260 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5khq" podUID="51bffaee-68a0-486d-8a62-a04c17b4ae9d" Jul 06 16:38:31.096243 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:31.096206 2574 generic.go:356] "Generic (PLEG): container finished" podID="17554f49b63e35a815e8bf1b98f97477" containerID="9e5ed94dd6696b5606f1a0cc4e62e85cdad1df8f4b863abf0bb6fa82a867eefb" exitCode=0 Jul 06 16:38:31.097117 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:31.097094 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal" event={"ID":"17554f49b63e35a815e8bf1b98f97477","Type":"ContainerDied","Data":"9e5ed94dd6696b5606f1a0cc4e62e85cdad1df8f4b863abf0bb6fa82a867eefb"} Jul 06 16:38:31.132855 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:31.132797 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-71.ec2.internal" podStartSLOduration=3.132772387 podStartE2EDuration="3.132772387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:38:30.118562302 +0000 UTC m=+3.638053417" watchObservedRunningTime="2026-07-06 16:38:31.132772387 +0000 UTC m=+4.652263519" Jul 06 16:38:32.071114 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:32.070364 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:38:32.071114 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:32.070511 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9kmw" podUID="39a924d0-6e9a-48bb-9972-6d40dd4c50dc" Jul 06 16:38:32.071114 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:32.070980 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:32.071114 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:32.071069 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnd4r" podUID="9ce13f97-b996-4f40-86fb-e0453bd3a476" Jul 06 16:38:32.122221 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:32.122182 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal" event={"ID":"17554f49b63e35a815e8bf1b98f97477","Type":"ContainerStarted","Data":"f7984ba9a5f7b0eaf78118a87310bf652c73b98b87a0c83d4a4ebad8b1dfa819"} Jul 06 16:38:32.508498 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:32.508409 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ce13f97-b996-4f40-86fb-e0453bd3a476-original-pull-secret\") pod \"global-pull-secret-syncer-nnd4r\" (UID: \"9ce13f97-b996-4f40-86fb-e0453bd3a476\") " pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:32.508683 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:32.508545 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39a924d0-6e9a-48bb-9972-6d40dd4c50dc-metrics-certs\") pod \"network-metrics-daemon-n9kmw\" (UID: \"39a924d0-6e9a-48bb-9972-6d40dd4c50dc\") " pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:38:32.508683 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:32.508668 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:38:32.508902 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:32.508745 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39a924d0-6e9a-48bb-9972-6d40dd4c50dc-metrics-certs podName:39a924d0-6e9a-48bb-9972-6d40dd4c50dc nodeName:}" failed. No retries permitted until 2026-07-06 16:38:36.508726063 +0000 UTC m=+10.028217161 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39a924d0-6e9a-48bb-9972-6d40dd4c50dc-metrics-certs") pod "network-metrics-daemon-n9kmw" (UID: "39a924d0-6e9a-48bb-9972-6d40dd4c50dc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:38:32.509143 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:32.509123 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 16:38:32.509273 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:32.509177 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce13f97-b996-4f40-86fb-e0453bd3a476-original-pull-secret podName:9ce13f97-b996-4f40-86fb-e0453bd3a476 nodeName:}" failed. No retries permitted until 2026-07-06 16:38:36.509161555 +0000 UTC m=+10.028652660 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9ce13f97-b996-4f40-86fb-e0453bd3a476-original-pull-secret") pod "global-pull-secret-syncer-nnd4r" (UID: "9ce13f97-b996-4f40-86fb-e0453bd3a476") : object "kube-system"/"original-pull-secret" not registered Jul 06 16:38:32.609921 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:32.609793 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szlc4\" (UniqueName: \"kubernetes.io/projected/51bffaee-68a0-486d-8a62-a04c17b4ae9d-kube-api-access-szlc4\") pod \"network-check-target-j5khq\" (UID: \"51bffaee-68a0-486d-8a62-a04c17b4ae9d\") " pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:38:32.610102 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:32.610033 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 16:38:32.610102 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:32.610064 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 16:38:32.610102 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:32.610078 2574 projected.go:196] Error preparing data for projected volume kube-api-access-szlc4 for pod openshift-network-diagnostics/network-check-target-j5khq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:38:32.610254 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:32.610140 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51bffaee-68a0-486d-8a62-a04c17b4ae9d-kube-api-access-szlc4 podName:51bffaee-68a0-486d-8a62-a04c17b4ae9d nodeName:}" failed. No retries permitted until 2026-07-06 16:38:36.610121959 +0000 UTC m=+10.129613069 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-szlc4" (UniqueName: "kubernetes.io/projected/51bffaee-68a0-486d-8a62-a04c17b4ae9d-kube-api-access-szlc4") pod "network-check-target-j5khq" (UID: "51bffaee-68a0-486d-8a62-a04c17b4ae9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:38:33.072963 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:33.072507 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:38:33.072963 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:33.072654 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5khq" podUID="51bffaee-68a0-486d-8a62-a04c17b4ae9d" Jul 06 16:38:34.070099 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:34.070061 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:34.070269 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:34.070213 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnd4r" podUID="9ce13f97-b996-4f40-86fb-e0453bd3a476" Jul 06 16:38:34.070331 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:34.070285 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:38:34.070439 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:34.070414 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9kmw" podUID="39a924d0-6e9a-48bb-9972-6d40dd4c50dc" Jul 06 16:38:35.070547 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:35.070453 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:38:35.071007 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:35.070600 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5khq" podUID="51bffaee-68a0-486d-8a62-a04c17b4ae9d" Jul 06 16:38:36.070543 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:36.070189 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:38:36.070543 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:36.070334 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9kmw" podUID="39a924d0-6e9a-48bb-9972-6d40dd4c50dc" Jul 06 16:38:36.071038 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:36.070774 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:36.071038 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:36.070874 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnd4r" podUID="9ce13f97-b996-4f40-86fb-e0453bd3a476" Jul 06 16:38:36.543577 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:36.543444 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39a924d0-6e9a-48bb-9972-6d40dd4c50dc-metrics-certs\") pod \"network-metrics-daemon-n9kmw\" (UID: \"39a924d0-6e9a-48bb-9972-6d40dd4c50dc\") " pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:38:36.543577 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:36.543503 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ce13f97-b996-4f40-86fb-e0453bd3a476-original-pull-secret\") pod \"global-pull-secret-syncer-nnd4r\" (UID: \"9ce13f97-b996-4f40-86fb-e0453bd3a476\") " pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:36.543788 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:36.543645 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 16:38:36.543788 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:36.543701 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce13f97-b996-4f40-86fb-e0453bd3a476-original-pull-secret podName:9ce13f97-b996-4f40-86fb-e0453bd3a476 nodeName:}" failed. No retries permitted until 2026-07-06 16:38:44.54368392 +0000 UTC m=+18.063175014 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9ce13f97-b996-4f40-86fb-e0453bd3a476-original-pull-secret") pod "global-pull-secret-syncer-nnd4r" (UID: "9ce13f97-b996-4f40-86fb-e0453bd3a476") : object "kube-system"/"original-pull-secret" not registered Jul 06 16:38:36.544099 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:36.544079 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:38:36.544194 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:36.544134 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39a924d0-6e9a-48bb-9972-6d40dd4c50dc-metrics-certs podName:39a924d0-6e9a-48bb-9972-6d40dd4c50dc nodeName:}" failed. No retries permitted until 2026-07-06 16:38:44.544118272 +0000 UTC m=+18.063609369 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39a924d0-6e9a-48bb-9972-6d40dd4c50dc-metrics-certs") pod "network-metrics-daemon-n9kmw" (UID: "39a924d0-6e9a-48bb-9972-6d40dd4c50dc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:38:36.644645 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:36.644501 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szlc4\" (UniqueName: \"kubernetes.io/projected/51bffaee-68a0-486d-8a62-a04c17b4ae9d-kube-api-access-szlc4\") pod \"network-check-target-j5khq\" (UID: \"51bffaee-68a0-486d-8a62-a04c17b4ae9d\") " pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:38:36.644810 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:36.644662 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 16:38:36.644810 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:36.644688 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 16:38:36.644810 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:36.644701 2574 projected.go:196] Error preparing data for projected volume kube-api-access-szlc4 for pod openshift-network-diagnostics/network-check-target-j5khq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:38:36.644810 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:36.644764 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51bffaee-68a0-486d-8a62-a04c17b4ae9d-kube-api-access-szlc4 podName:51bffaee-68a0-486d-8a62-a04c17b4ae9d nodeName:}" failed. No retries permitted until 2026-07-06 16:38:44.644745508 +0000 UTC m=+18.164236620 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-szlc4" (UniqueName: "kubernetes.io/projected/51bffaee-68a0-486d-8a62-a04c17b4ae9d-kube-api-access-szlc4") pod "network-check-target-j5khq" (UID: "51bffaee-68a0-486d-8a62-a04c17b4ae9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:38:37.072499 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:37.071930 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:38:37.072499 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:37.072038 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5khq" podUID="51bffaee-68a0-486d-8a62-a04c17b4ae9d" Jul 06 16:38:38.071206 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:38.070729 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:38.071206 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:38.070884 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnd4r" podUID="9ce13f97-b996-4f40-86fb-e0453bd3a476" Jul 06 16:38:38.071206 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:38.071027 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:38:38.071206 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:38.071121 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9kmw" podUID="39a924d0-6e9a-48bb-9972-6d40dd4c50dc" Jul 06 16:38:39.070342 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:39.070310 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:38:39.070782 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:39.070433 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5khq" podUID="51bffaee-68a0-486d-8a62-a04c17b4ae9d" Jul 06 16:38:40.070911 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:40.070859 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:40.070911 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:40.070905 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:38:40.071403 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:40.071001 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnd4r" podUID="9ce13f97-b996-4f40-86fb-e0453bd3a476" Jul 06 16:38:40.071403 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:40.071117 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9kmw" podUID="39a924d0-6e9a-48bb-9972-6d40dd4c50dc" Jul 06 16:38:41.070953 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:41.070922 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:38:41.071370 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:41.071033 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5khq" podUID="51bffaee-68a0-486d-8a62-a04c17b4ae9d" Jul 06 16:38:42.070360 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:42.070323 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:42.070586 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:42.070325 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:38:42.070586 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:42.070482 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnd4r" podUID="9ce13f97-b996-4f40-86fb-e0453bd3a476" Jul 06 16:38:42.070717 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:42.070615 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9kmw" podUID="39a924d0-6e9a-48bb-9972-6d40dd4c50dc" Jul 06 16:38:43.070667 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:43.070638 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:38:43.071205 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:43.070763 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5khq" podUID="51bffaee-68a0-486d-8a62-a04c17b4ae9d" Jul 06 16:38:44.070050 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:44.070011 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:44.070279 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:44.070145 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnd4r" podUID="9ce13f97-b996-4f40-86fb-e0453bd3a476" Jul 06 16:38:44.070279 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:44.070199 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:38:44.070394 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:44.070312 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9kmw" podUID="39a924d0-6e9a-48bb-9972-6d40dd4c50dc" Jul 06 16:38:44.607175 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:44.607140 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39a924d0-6e9a-48bb-9972-6d40dd4c50dc-metrics-certs\") pod \"network-metrics-daemon-n9kmw\" (UID: \"39a924d0-6e9a-48bb-9972-6d40dd4c50dc\") " pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:38:44.607620 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:44.607189 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ce13f97-b996-4f40-86fb-e0453bd3a476-original-pull-secret\") pod \"global-pull-secret-syncer-nnd4r\" (UID: \"9ce13f97-b996-4f40-86fb-e0453bd3a476\") " pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:44.607620 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:44.607268 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:38:44.607620 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:44.607309 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 16:38:44.607620 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:44.607348 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39a924d0-6e9a-48bb-9972-6d40dd4c50dc-metrics-certs podName:39a924d0-6e9a-48bb-9972-6d40dd4c50dc nodeName:}" failed. No retries permitted until 2026-07-06 16:39:00.607331802 +0000 UTC m=+34.126822895 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39a924d0-6e9a-48bb-9972-6d40dd4c50dc-metrics-certs") pod "network-metrics-daemon-n9kmw" (UID: "39a924d0-6e9a-48bb-9972-6d40dd4c50dc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:38:44.607620 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:44.607364 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce13f97-b996-4f40-86fb-e0453bd3a476-original-pull-secret podName:9ce13f97-b996-4f40-86fb-e0453bd3a476 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:00.607357525 +0000 UTC m=+34.126848618 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9ce13f97-b996-4f40-86fb-e0453bd3a476-original-pull-secret") pod "global-pull-secret-syncer-nnd4r" (UID: "9ce13f97-b996-4f40-86fb-e0453bd3a476") : object "kube-system"/"original-pull-secret" not registered Jul 06 16:38:44.708358 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:44.708325 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szlc4\" (UniqueName: \"kubernetes.io/projected/51bffaee-68a0-486d-8a62-a04c17b4ae9d-kube-api-access-szlc4\") pod \"network-check-target-j5khq\" (UID: \"51bffaee-68a0-486d-8a62-a04c17b4ae9d\") " pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:38:44.708571 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:44.708511 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 16:38:44.708571 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:44.708553 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 16:38:44.708571 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:44.708566 2574 projected.go:196] Error preparing data for projected volume kube-api-access-szlc4 for pod openshift-network-diagnostics/network-check-target-j5khq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:38:44.708712 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:44.708632 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51bffaee-68a0-486d-8a62-a04c17b4ae9d-kube-api-access-szlc4 podName:51bffaee-68a0-486d-8a62-a04c17b4ae9d nodeName:}" failed. No retries permitted until 2026-07-06 16:39:00.708613287 +0000 UTC m=+34.228104398 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-szlc4" (UniqueName: "kubernetes.io/projected/51bffaee-68a0-486d-8a62-a04c17b4ae9d-kube-api-access-szlc4") pod "network-check-target-j5khq" (UID: "51bffaee-68a0-486d-8a62-a04c17b4ae9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:38:45.073444 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:45.073371 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:38:45.073617 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:45.073489 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5khq" podUID="51bffaee-68a0-486d-8a62-a04c17b4ae9d" Jul 06 16:38:46.070888 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:46.070844 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:38:46.071310 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:46.070902 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:46.071310 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:46.070991 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9kmw" podUID="39a924d0-6e9a-48bb-9972-6d40dd4c50dc" Jul 06 16:38:46.071310 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:46.071131 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnd4r" podUID="9ce13f97-b996-4f40-86fb-e0453bd3a476" Jul 06 16:38:47.072527 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:47.072488 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:38:47.072887 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:47.072613 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5khq" podUID="51bffaee-68a0-486d-8a62-a04c17b4ae9d" Jul 06 16:38:48.070408 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:48.070373 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:48.070408 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:48.070405 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:38:48.070670 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:48.070530 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnd4r" podUID="9ce13f97-b996-4f40-86fb-e0453bd3a476" Jul 06 16:38:48.070670 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:48.070624 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9kmw" podUID="39a924d0-6e9a-48bb-9972-6d40dd4c50dc" Jul 06 16:38:49.070839 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:49.070544 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:38:49.071444 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:49.070927 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5khq" podUID="51bffaee-68a0-486d-8a62-a04c17b4ae9d" Jul 06 16:38:49.155174 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:49.155088 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nf6xc" event={"ID":"5aa71c0b-c28a-43a4-92d7-3d617a604f55","Type":"ContainerStarted","Data":"72c5c0b612f985e05b4b0769ac3afb69b530eff4dd507eef38f0edfe417f19b9"} Jul 06 16:38:49.156566 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:49.156539 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lxbqr" event={"ID":"0f6a8b1b-224d-459c-aad2-1d86d1c38c79","Type":"ContainerStarted","Data":"bcf80551c1500c1a71572c20b6da98f04265234eb0b0308a688b2548c6be1325"} Jul 06 16:38:49.157855 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:49.157833 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bwztk" event={"ID":"413730d5-8dc6-44e3-890f-fa03e3339ada","Type":"ContainerStarted","Data":"ba4aec70f13b1ad474ba30bf07de89cd45cfc17a3ba0014e36892a3e51072832"} Jul 06 16:38:49.159715 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:49.159455 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" event={"ID":"36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf","Type":"ContainerStarted","Data":"f8236baa99445d5e7d3a8c9f3a5c3231c16323ac9196f8872959617856ecd47a"} Jul 06 16:38:49.161536 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:49.161491 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" event={"ID":"2a168e85-8f35-497e-bc2e-ba3a5bc8396d","Type":"ContainerStarted","Data":"2d929807374269c83492431f99900d4ba72e77357d827d7840a6c27b8ced51a1"} Jul 06 16:38:49.163310 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:49.163288 2574 generic.go:356] "Generic (PLEG): container finished" podID="563411d8-43a0-49ab-b73b-c29dec275f46" containerID="e50546350ef0b19f7cf36a7b49588916981ebe4c99651f4eee30be9c4e35f9a3" exitCode=0 Jul 06 16:38:49.163406 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:49.163362 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p9rnx" event={"ID":"563411d8-43a0-49ab-b73b-c29dec275f46","Type":"ContainerDied","Data":"e50546350ef0b19f7cf36a7b49588916981ebe4c99651f4eee30be9c4e35f9a3"} Jul 06 16:38:49.164837 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:49.164819 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xswlf" event={"ID":"94a73cbd-9f23-4895-9127-fe752b38a3a5","Type":"ContainerStarted","Data":"e37ab2d2b7849232370baf88aeddf3a9910d952363d22eab7880c440ff52c4ae"} Jul 06 16:38:49.167634 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:49.167616 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" event={"ID":"079df086-1654-4d96-8776-36ff7f1c8f13","Type":"ContainerStarted","Data":"f7b620503bb0b2519f97f99ffc331aaada0c919b12b6101464116cfea4ebafcd"} Jul 06 16:38:49.167708 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:49.167670 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" event={"ID":"079df086-1654-4d96-8776-36ff7f1c8f13","Type":"ContainerStarted","Data":"03052c032c79c81aa83edb0ae9949ea110a5f351469232f6b31c21491a833513"} Jul 06 16:38:49.167708 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:49.167702 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" event={"ID":"079df086-1654-4d96-8776-36ff7f1c8f13","Type":"ContainerStarted","Data":"78ec7b248f5c6f8c6e4cb1c74d7a1c08c45b708243a3815c23eef561587da3b6"} Jul 06 16:38:49.167775 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:49.167729 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" event={"ID":"079df086-1654-4d96-8776-36ff7f1c8f13","Type":"ContainerStarted","Data":"ce2d8373e2ba09e043d07c101e0000b24657c7229e680dab18762af29c41cb41"} Jul 06 16:38:49.167775 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:49.167762 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" event={"ID":"079df086-1654-4d96-8776-36ff7f1c8f13","Type":"ContainerStarted","Data":"e18c156e3f5dd6b69249bb27c734693f62a7c6d27e702b69b4bebff61c243482"} Jul 06 16:38:49.167847 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:49.167789 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" event={"ID":"079df086-1654-4d96-8776-36ff7f1c8f13","Type":"ContainerStarted","Data":"dc2d6351cedba38db64377a94840cb148c0a1858cf0fccf79a6e969e580c55c4"} Jul 06 16:38:49.182593 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:49.182555 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-71.ec2.internal" podStartSLOduration=21.182543423 podStartE2EDuration="21.182543423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:38:32.155258745 +0000 UTC m=+5.674749862" watchObservedRunningTime="2026-07-06 16:38:49.182543423 +0000 UTC m=+22.702034539" Jul 06 16:38:49.182817 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:49.182796 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nf6xc" podStartSLOduration=3.761951898 podStartE2EDuration="22.182789517s" totalImagesPullingTime="18.420837619s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:38:49.182483418 +0000 UTC m=+22.701974534" watchObservedRunningTime="2026-07-06 16:38:49.182789517 +0000 UTC m=+22.702280633" Jul 06 16:38:49.254462 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:49.254418 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bwztk" podStartSLOduration=3.7750524800000003 podStartE2EDuration="22.254403502s" totalImagesPullingTime="18.479351022s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:38:49.222021906 +0000 UTC m=+22.741513022" watchObservedRunningTime="2026-07-06 16:38:49.254403502 +0000 UTC m=+22.773894677" Jul 06 16:38:49.308827 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:49.308770 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lxbqr" podStartSLOduration=3.894230693 podStartE2EDuration="22.308755329s" totalImagesPullingTime="18.414524636s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:38:49.254534969 +0000 UTC m=+22.774026080" watchObservedRunningTime="2026-07-06 16:38:49.308755329 +0000 UTC m=+22.828246445" Jul 06 16:38:49.355202 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:49.355149 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-s9xmf" podStartSLOduration=3.935472798 podStartE2EDuration="22.355133389s" totalImagesPullingTime="18.419660591s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:38:49.308651683 +0000 UTC m=+22.828142800" watchObservedRunningTime="2026-07-06 16:38:49.355133389 +0000 UTC m=+22.874624505" Jul 06 16:38:49.402004 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:49.401977 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xswlf" Jul 06 16:38:49.402503 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:49.402487 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xswlf" Jul 06 16:38:49.436667 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:49.436628 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xswlf" podStartSLOduration=9.197355554 podStartE2EDuration="22.43661291s" totalImagesPullingTime="13.239257356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:38:49.392678387 +0000 UTC m=+22.912169516" watchObservedRunningTime="2026-07-06 16:38:49.43661291 +0000 UTC m=+22.956104026" Jul 06 16:38:50.070953 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:50.070921 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:50.071367 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:50.070921 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:38:50.071367 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:50.071038 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnd4r" podUID="9ce13f97-b996-4f40-86fb-e0453bd3a476" Jul 06 16:38:50.071367 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:50.071103 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9kmw" podUID="39a924d0-6e9a-48bb-9972-6d40dd4c50dc" Jul 06 16:38:50.170880 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:50.170839 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xtmgz" event={"ID":"e8ac8a0d-d06c-4b68-a3d3-c47080439729","Type":"ContainerStarted","Data":"fd4e5983f2080484c4de834e45d5e1017ab44708b34a29d1dc6924106ba46b4a"} Jul 06 16:38:51.073366 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:51.073340 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:38:51.073766 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:51.073431 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5khq" podUID="51bffaee-68a0-486d-8a62-a04c17b4ae9d" Jul 06 16:38:51.177893 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:51.177858 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" event={"ID":"079df086-1654-4d96-8776-36ff7f1c8f13","Type":"ContainerStarted","Data":"494b2994d418a5d6be66a2dc59080f5d052d0b8b962cb666b831d5c5c92f9b7c"} Jul 06 16:38:51.178070 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:51.178005 2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 06 16:38:51.763099 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:51.762926 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 16:38:51.990112 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:51.990013 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T16:38:51.762949081Z","UUID":"0458ec98-21a9-4c06-a6dc-bc06fadc1319","Handler":null,"Name":"","Endpoint":""} Jul 06 16:38:51.992213 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:51.992191 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 16:38:51.992377 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:51.992233 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 16:38:52.022975 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:52.022944 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nf6xc_5aa71c0b-c28a-43a4-92d7-3d617a604f55/dns-node-resolver/0.log" Jul 06 16:38:52.070764 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:52.070731 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:38:52.070956 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:52.070771 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:52.070956 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:52.070842 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9kmw" podUID="39a924d0-6e9a-48bb-9972-6d40dd4c50dc" Jul 06 16:38:52.071065 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:52.070986 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnd4r" podUID="9ce13f97-b996-4f40-86fb-e0453bd3a476" Jul 06 16:38:52.182186 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:52.182148 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" event={"ID":"36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf","Type":"ContainerStarted","Data":"a821fc473984eb014bb53795e24520a2164771c8a8b16a4a79ffdc6098b5f119"} Jul 06 16:38:52.575003 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:52.574975 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lxbqr_0f6a8b1b-224d-459c-aad2-1d86d1c38c79/node-ca/0.log" Jul 06 16:38:53.070570 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:53.070332 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:38:53.070706 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:53.070681 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5khq" podUID="51bffaee-68a0-486d-8a62-a04c17b4ae9d" Jul 06 16:38:53.187623 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:53.187586 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" event={"ID":"36d8a2a0-2014-4a6f-b4b6-94a844fa8dcf","Type":"ContainerStarted","Data":"5e8bcbf57d10742fc4a0623c3ddbf38cbc3dacf4c51ef67ce5fc56cd7e1662c6"} Jul 06 16:38:53.221491 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:53.221441 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xtmgz" podStartSLOduration=7.803532359 podStartE2EDuration="26.22142044s" totalImagesPullingTime="18.417888081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:38:50.201760077 +0000 UTC m=+23.721251193" watchObservedRunningTime="2026-07-06 16:38:53.22142044 +0000 UTC m=+26.740911559" Jul 06 16:38:53.221852 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:53.221818 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7gqsc" podStartSLOduration=3.576064434 podStartE2EDuration="26.221804561s" totalImagesPullingTime="22.645740127s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:38:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:38:53.220944777 +0000 UTC m=+26.740435895" watchObservedRunningTime="2026-07-06 16:38:53.221804561 +0000 UTC m=+26.741295678" Jul 06 16:38:54.070223 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:54.070197 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:54.070364 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:54.070201 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:38:54.070364 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:54.070282 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnd4r" podUID="9ce13f97-b996-4f40-86fb-e0453bd3a476" Jul 06 16:38:54.070440 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:54.070360 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9kmw" podUID="39a924d0-6e9a-48bb-9972-6d40dd4c50dc" Jul 06 16:38:54.192552 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:54.192497 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" event={"ID":"079df086-1654-4d96-8776-36ff7f1c8f13","Type":"ContainerStarted","Data":"08a3468a04e0feac570578c19178972ee6ec78e6523fb24d31361ae288c8687e"} Jul 06 16:38:54.331927 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:54.331818 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" podStartSLOduration=8.835866283 podStartE2EDuration="27.331799219s" totalImagesPullingTime="18.495932936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:38:54.27626238 +0000 UTC m=+27.795753507" watchObservedRunningTime="2026-07-06 16:38:54.331799219 +0000 UTC m=+27.851290335" Jul 06 16:38:55.074668 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:55.074455 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:38:55.074828 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:55.074762 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5khq" podUID="51bffaee-68a0-486d-8a62-a04c17b4ae9d" Jul 06 16:38:55.195445 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:55.195388 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:55.195920 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:55.195475 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:55.195920 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:55.195548 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:55.214193 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:55.214166 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:55.214411 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:55.214281 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:38:55.926252 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:55.926211 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n9kmw"] Jul 06 16:38:55.926420 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:55.926374 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:38:55.926512 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:55.926490 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9kmw" podUID="39a924d0-6e9a-48bb-9972-6d40dd4c50dc" Jul 06 16:38:55.931185 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:55.931155 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nnd4r"] Jul 06 16:38:55.931346 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:55.931264 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:55.931429 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:55.931359 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnd4r" podUID="9ce13f97-b996-4f40-86fb-e0453bd3a476" Jul 06 16:38:55.932197 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:55.932154 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j5khq"] Jul 06 16:38:55.932302 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:55.932257 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:38:55.932370 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:55.932344 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5khq" podUID="51bffaee-68a0-486d-8a62-a04c17b4ae9d" Jul 06 16:38:57.070722 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:57.070634 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:38:57.071071 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:57.070724 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9kmw" podUID="39a924d0-6e9a-48bb-9972-6d40dd4c50dc" Jul 06 16:38:57.201028 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:57.200998 2574 generic.go:356] "Generic (PLEG): container finished" podID="563411d8-43a0-49ab-b73b-c29dec275f46" containerID="1703c9f595d7ea20ec73fe70ef8d37ede98de507e890891ead81cfaef1134cc6" exitCode=0 Jul 06 16:38:57.201172 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:57.201092 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p9rnx" event={"ID":"563411d8-43a0-49ab-b73b-c29dec275f46","Type":"ContainerDied","Data":"1703c9f595d7ea20ec73fe70ef8d37ede98de507e890891ead81cfaef1134cc6"} Jul 06 16:38:57.607072 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:57.607007 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xswlf" Jul 06 16:38:57.607214 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:57.607134 2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 06 16:38:57.607574 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:57.607529 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xswlf" Jul 06 16:38:58.070709 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:58.070527 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:38:58.070827 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:58.070583 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:38:58.070827 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:58.070789 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnd4r" podUID="9ce13f97-b996-4f40-86fb-e0453bd3a476" Jul 06 16:38:58.071104 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:58.070853 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5khq" podUID="51bffaee-68a0-486d-8a62-a04c17b4ae9d" Jul 06 16:38:58.204809 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:58.204775 2574 generic.go:356] "Generic (PLEG): container finished" podID="563411d8-43a0-49ab-b73b-c29dec275f46" containerID="237644c8863ad7a88d2fbd81b1490af545868be57fb44a9e8437d4f7a6772249" exitCode=0 Jul 06 16:38:58.204965 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:58.204858 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p9rnx" event={"ID":"563411d8-43a0-49ab-b73b-c29dec275f46","Type":"ContainerDied","Data":"237644c8863ad7a88d2fbd81b1490af545868be57fb44a9e8437d4f7a6772249"} Jul 06 16:38:59.073080 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:59.073011 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:38:59.073532 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:38:59.073103 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9kmw" podUID="39a924d0-6e9a-48bb-9972-6d40dd4c50dc" Jul 06 16:38:59.209456 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:59.209423 2574 generic.go:356] "Generic (PLEG): container finished" podID="563411d8-43a0-49ab-b73b-c29dec275f46" containerID="1a0e378205fa699bae26fde6e31604af915ddd26d442a0ba311b7115895e6ec6" exitCode=0 Jul 06 16:38:59.209638 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:38:59.209495 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p9rnx" event={"ID":"563411d8-43a0-49ab-b73b-c29dec275f46","Type":"ContainerDied","Data":"1a0e378205fa699bae26fde6e31604af915ddd26d442a0ba311b7115895e6ec6"} Jul 06 16:39:00.070378 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:00.070341 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:39:00.070378 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:00.070375 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:39:00.070713 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:00.070475 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nnd4r" podUID="9ce13f97-b996-4f40-86fb-e0453bd3a476" Jul 06 16:39:00.070713 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:00.070598 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j5khq" podUID="51bffaee-68a0-486d-8a62-a04c17b4ae9d" Jul 06 16:39:00.631995 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:00.631958 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ce13f97-b996-4f40-86fb-e0453bd3a476-original-pull-secret\") pod \"global-pull-secret-syncer-nnd4r\" (UID: \"9ce13f97-b996-4f40-86fb-e0453bd3a476\") " pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:39:00.632409 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:00.632050 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39a924d0-6e9a-48bb-9972-6d40dd4c50dc-metrics-certs\") pod \"network-metrics-daemon-n9kmw\" (UID: \"39a924d0-6e9a-48bb-9972-6d40dd4c50dc\") " pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:39:00.632409 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:00.632087 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 16:39:00.632409 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:00.632131 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:39:00.632409 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:00.632171 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ce13f97-b996-4f40-86fb-e0453bd3a476-original-pull-secret podName:9ce13f97-b996-4f40-86fb-e0453bd3a476 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:32.632148958 +0000 UTC m=+66.151640056 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9ce13f97-b996-4f40-86fb-e0453bd3a476-original-pull-secret") pod "global-pull-secret-syncer-nnd4r" (UID: "9ce13f97-b996-4f40-86fb-e0453bd3a476") : object "kube-system"/"original-pull-secret" not registered Jul 06 16:39:00.632409 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:00.632193 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39a924d0-6e9a-48bb-9972-6d40dd4c50dc-metrics-certs podName:39a924d0-6e9a-48bb-9972-6d40dd4c50dc nodeName:}" failed. No retries permitted until 2026-07-06 16:39:32.632181694 +0000 UTC m=+66.151672795 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/39a924d0-6e9a-48bb-9972-6d40dd4c50dc-metrics-certs") pod "network-metrics-daemon-n9kmw" (UID: "39a924d0-6e9a-48bb-9972-6d40dd4c50dc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:39:00.732901 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:00.732869 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szlc4\" (UniqueName: \"kubernetes.io/projected/51bffaee-68a0-486d-8a62-a04c17b4ae9d-kube-api-access-szlc4\") pod \"network-check-target-j5khq\" (UID: \"51bffaee-68a0-486d-8a62-a04c17b4ae9d\") " pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:39:00.733067 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:00.733020 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 16:39:00.733067 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:00.733036 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 16:39:00.733067 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:00.733044 2574 projected.go:196] Error preparing data for projected volume kube-api-access-szlc4 for pod openshift-network-diagnostics/network-check-target-j5khq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:39:00.733208 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:00.733093 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/51bffaee-68a0-486d-8a62-a04c17b4ae9d-kube-api-access-szlc4 podName:51bffaee-68a0-486d-8a62-a04c17b4ae9d nodeName:}" failed. No retries permitted until 2026-07-06 16:39:32.733077138 +0000 UTC m=+66.252568255 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-szlc4" (UniqueName: "kubernetes.io/projected/51bffaee-68a0-486d-8a62-a04c17b4ae9d-kube-api-access-szlc4") pod "network-check-target-j5khq" (UID: "51bffaee-68a0-486d-8a62-a04c17b4ae9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:39:01.073224 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.073198 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:39:01.073359 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:01.073303 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-n9kmw" podUID="39a924d0-6e9a-48bb-9972-6d40dd4c50dc" Jul 06 16:39:01.275423 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.275354 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-71.ec2.internal" event="NodeReady" Jul 06 16:39:01.275568 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.275461 2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 16:39:01.398224 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.398191 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-n82pj"] Jul 06 16:39:01.400341 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.400322 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n82pj" Jul 06 16:39:01.407943 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.407832 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 16:39:01.407943 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.407893 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-db7nl\"" Jul 06 16:39:01.408187 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.407943 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 16:39:01.408872 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.408853 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pdh4h"] Jul 06 16:39:01.410827 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.410808 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pdh4h" Jul 06 16:39:01.427787 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.427745 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 16:39:01.427913 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.427798 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 16:39:01.427913 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.427858 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 16:39:01.428153 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.428138 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-g8n7p\"" Jul 06 16:39:01.482455 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.482427 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pdh4h"] Jul 06 16:39:01.530071 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.529973 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fqkj2"] Jul 06 16:39:01.532303 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.532273 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fqkj2" Jul 06 16:39:01.539532 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.539498 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2d0b8d6-be30-423c-840e-f20cebd694fd-metrics-tls\") pod \"dns-default-n82pj\" (UID: \"f2d0b8d6-be30-423c-840e-f20cebd694fd\") " pod="openshift-dns/dns-default-n82pj" Jul 06 16:39:01.539660 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.539617 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d0b8d6-be30-423c-840e-f20cebd694fd-tmp-dir\") pod \"dns-default-n82pj\" (UID: \"f2d0b8d6-be30-423c-840e-f20cebd694fd\") " pod="openshift-dns/dns-default-n82pj" Jul 06 16:39:01.539728 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.539699 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b350fbc-4cba-421e-89d3-1f0df3318c34-cert\") pod \"ingress-canary-pdh4h\" (UID: \"5b350fbc-4cba-421e-89d3-1f0df3318c34\") " pod="openshift-ingress-canary/ingress-canary-pdh4h" Jul 06 16:39:01.539782 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.539726 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d0b8d6-be30-423c-840e-f20cebd694fd-config-volume\") pod \"dns-default-n82pj\" (UID: \"f2d0b8d6-be30-423c-840e-f20cebd694fd\") " pod="openshift-dns/dns-default-n82pj" Jul 06 16:39:01.539839 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.539793 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxzpr\" (UniqueName: \"kubernetes.io/projected/5b350fbc-4cba-421e-89d3-1f0df3318c34-kube-api-access-nxzpr\") pod \"ingress-canary-pdh4h\" (UID: \"5b350fbc-4cba-421e-89d3-1f0df3318c34\") " pod="openshift-ingress-canary/ingress-canary-pdh4h" Jul 06 16:39:01.539893 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.539844 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7x2n\" (UniqueName: \"kubernetes.io/projected/f2d0b8d6-be30-423c-840e-f20cebd694fd-kube-api-access-x7x2n\") pod \"dns-default-n82pj\" (UID: \"f2d0b8d6-be30-423c-840e-f20cebd694fd\") " pod="openshift-dns/dns-default-n82pj" Jul 06 16:39:01.560577 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.560548 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fqkj2"] Jul 06 16:39:01.568588 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.568566 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n82pj"] Jul 06 16:39:01.577818 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.577799 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 16:39:01.577942 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.577872 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 16:39:01.578004 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.577964 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nl5jr\"" Jul 06 16:39:01.578091 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.578077 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 16:39:01.578477 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.578454 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 16:39:01.640344 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.640308 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48e427b3-b218-4b68-90ba-0a4750c49f43-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fqkj2\" (UID: \"48e427b3-b218-4b68-90ba-0a4750c49f43\") " pod="openshift-insights/insights-runtime-extractor-fqkj2" Jul 06 16:39:01.641002 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.640359 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxzpr\" (UniqueName: \"kubernetes.io/projected/5b350fbc-4cba-421e-89d3-1f0df3318c34-kube-api-access-nxzpr\") pod \"ingress-canary-pdh4h\" (UID: \"5b350fbc-4cba-421e-89d3-1f0df3318c34\") " pod="openshift-ingress-canary/ingress-canary-pdh4h" Jul 06 16:39:01.641002 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.640384 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/48e427b3-b218-4b68-90ba-0a4750c49f43-data-volume\") pod \"insights-runtime-extractor-fqkj2\" (UID: \"48e427b3-b218-4b68-90ba-0a4750c49f43\") " pod="openshift-insights/insights-runtime-extractor-fqkj2" Jul 06 16:39:01.641002 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.640563 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48e427b3-b218-4b68-90ba-0a4750c49f43-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fqkj2\" (UID: \"48e427b3-b218-4b68-90ba-0a4750c49f43\") " pod="openshift-insights/insights-runtime-extractor-fqkj2" Jul 06 16:39:01.641002 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.640614 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7x2n\" (UniqueName: \"kubernetes.io/projected/f2d0b8d6-be30-423c-840e-f20cebd694fd-kube-api-access-x7x2n\") pod \"dns-default-n82pj\" (UID: \"f2d0b8d6-be30-423c-840e-f20cebd694fd\") " pod="openshift-dns/dns-default-n82pj" Jul 06 16:39:01.641002 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.640639 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5njq\" (UniqueName: \"kubernetes.io/projected/48e427b3-b218-4b68-90ba-0a4750c49f43-kube-api-access-l5njq\") pod \"insights-runtime-extractor-fqkj2\" (UID: \"48e427b3-b218-4b68-90ba-0a4750c49f43\") " pod="openshift-insights/insights-runtime-extractor-fqkj2" Jul 06 16:39:01.641002 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.640687 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2d0b8d6-be30-423c-840e-f20cebd694fd-metrics-tls\") pod \"dns-default-n82pj\" (UID: \"f2d0b8d6-be30-423c-840e-f20cebd694fd\") " pod="openshift-dns/dns-default-n82pj" Jul 06 16:39:01.641002 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.640759 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/48e427b3-b218-4b68-90ba-0a4750c49f43-crio-socket\") pod \"insights-runtime-extractor-fqkj2\" (UID: \"48e427b3-b218-4b68-90ba-0a4750c49f43\") " pod="openshift-insights/insights-runtime-extractor-fqkj2" Jul 06 16:39:01.641002 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:01.640792 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 16:39:01.641002 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.640792 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d0b8d6-be30-423c-840e-f20cebd694fd-tmp-dir\") pod \"dns-default-n82pj\" (UID: \"f2d0b8d6-be30-423c-840e-f20cebd694fd\") " pod="openshift-dns/dns-default-n82pj" Jul 06 16:39:01.641002 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:01.640870 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2d0b8d6-be30-423c-840e-f20cebd694fd-metrics-tls podName:f2d0b8d6-be30-423c-840e-f20cebd694fd nodeName:}" failed. No retries permitted until 2026-07-06 16:39:02.140847389 +0000 UTC m=+35.660338497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2d0b8d6-be30-423c-840e-f20cebd694fd-metrics-tls") pod "dns-default-n82pj" (UID: "f2d0b8d6-be30-423c-840e-f20cebd694fd") : secret "dns-default-metrics-tls" not found Jul 06 16:39:01.641002 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.640929 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b350fbc-4cba-421e-89d3-1f0df3318c34-cert\") pod \"ingress-canary-pdh4h\" (UID: \"5b350fbc-4cba-421e-89d3-1f0df3318c34\") " pod="openshift-ingress-canary/ingress-canary-pdh4h" Jul 06 16:39:01.641002 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.640954 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d0b8d6-be30-423c-840e-f20cebd694fd-config-volume\") pod \"dns-default-n82pj\" (UID: \"f2d0b8d6-be30-423c-840e-f20cebd694fd\") " pod="openshift-dns/dns-default-n82pj" Jul 06 16:39:01.641695 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.641102 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d0b8d6-be30-423c-840e-f20cebd694fd-tmp-dir\") pod \"dns-default-n82pj\" (UID: \"f2d0b8d6-be30-423c-840e-f20cebd694fd\") " pod="openshift-dns/dns-default-n82pj" Jul 06 16:39:01.641695 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.641462 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f2d0b8d6-be30-423c-840e-f20cebd694fd-config-volume\") pod \"dns-default-n82pj\" (UID: \"f2d0b8d6-be30-423c-840e-f20cebd694fd\") " pod="openshift-dns/dns-default-n82pj" Jul 06 16:39:01.645921 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.645897 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b350fbc-4cba-421e-89d3-1f0df3318c34-cert\") pod \"ingress-canary-pdh4h\" (UID: \"5b350fbc-4cba-421e-89d3-1f0df3318c34\") " pod="openshift-ingress-canary/ingress-canary-pdh4h" Jul 06 16:39:01.663023 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.663002 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7x2n\" (UniqueName: \"kubernetes.io/projected/f2d0b8d6-be30-423c-840e-f20cebd694fd-kube-api-access-x7x2n\") pod \"dns-default-n82pj\" (UID: \"f2d0b8d6-be30-423c-840e-f20cebd694fd\") " pod="openshift-dns/dns-default-n82pj" Jul 06 16:39:01.703160 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.703135 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxzpr\" (UniqueName: \"kubernetes.io/projected/5b350fbc-4cba-421e-89d3-1f0df3318c34-kube-api-access-nxzpr\") pod \"ingress-canary-pdh4h\" (UID: \"5b350fbc-4cba-421e-89d3-1f0df3318c34\") " pod="openshift-ingress-canary/ingress-canary-pdh4h" Jul 06 16:39:01.720075 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.720045 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pdh4h" Jul 06 16:39:01.741469 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.741439 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/48e427b3-b218-4b68-90ba-0a4750c49f43-data-volume\") pod \"insights-runtime-extractor-fqkj2\" (UID: \"48e427b3-b218-4b68-90ba-0a4750c49f43\") " pod="openshift-insights/insights-runtime-extractor-fqkj2" Jul 06 16:39:01.741632 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.741479 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48e427b3-b218-4b68-90ba-0a4750c49f43-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fqkj2\" (UID: \"48e427b3-b218-4b68-90ba-0a4750c49f43\") " pod="openshift-insights/insights-runtime-extractor-fqkj2" Jul 06 16:39:01.741632 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:01.741586 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 16:39:01.741745 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.741644 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5njq\" (UniqueName: \"kubernetes.io/projected/48e427b3-b218-4b68-90ba-0a4750c49f43-kube-api-access-l5njq\") pod \"insights-runtime-extractor-fqkj2\" (UID: \"48e427b3-b218-4b68-90ba-0a4750c49f43\") " pod="openshift-insights/insights-runtime-extractor-fqkj2" Jul 06 16:39:01.741745 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:01.741663 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/48e427b3-b218-4b68-90ba-0a4750c49f43-kube-rbac-proxy-cm podName:48e427b3-b218-4b68-90ba-0a4750c49f43 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:02.241643211 +0000 UTC m=+35.761134330 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/48e427b3-b218-4b68-90ba-0a4750c49f43-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fqkj2" (UID: "48e427b3-b218-4b68-90ba-0a4750c49f43") : configmap "kube-rbac-proxy" not found Jul 06 16:39:01.742246 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.742038 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/48e427b3-b218-4b68-90ba-0a4750c49f43-crio-socket\") pod \"insights-runtime-extractor-fqkj2\" (UID: \"48e427b3-b218-4b68-90ba-0a4750c49f43\") " pod="openshift-insights/insights-runtime-extractor-fqkj2" Jul 06 16:39:01.742246 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.742144 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48e427b3-b218-4b68-90ba-0a4750c49f43-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fqkj2\" (UID: \"48e427b3-b218-4b68-90ba-0a4750c49f43\") " pod="openshift-insights/insights-runtime-extractor-fqkj2" Jul 06 16:39:01.742246 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.742221 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/48e427b3-b218-4b68-90ba-0a4750c49f43-data-volume\") pod \"insights-runtime-extractor-fqkj2\" (UID: \"48e427b3-b218-4b68-90ba-0a4750c49f43\") " pod="openshift-insights/insights-runtime-extractor-fqkj2" Jul 06 16:39:01.742465 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.742247 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/48e427b3-b218-4b68-90ba-0a4750c49f43-crio-socket\") pod \"insights-runtime-extractor-fqkj2\" (UID: \"48e427b3-b218-4b68-90ba-0a4750c49f43\") " pod="openshift-insights/insights-runtime-extractor-fqkj2" Jul 06 16:39:01.742465 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:01.742299 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 16:39:01.742465 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:01.742375 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48e427b3-b218-4b68-90ba-0a4750c49f43-insights-runtime-extractor-tls podName:48e427b3-b218-4b68-90ba-0a4750c49f43 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:02.242360419 +0000 UTC m=+35.761851516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/48e427b3-b218-4b68-90ba-0a4750c49f43-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fqkj2" (UID: "48e427b3-b218-4b68-90ba-0a4750c49f43") : secret "insights-runtime-extractor-tls" not found Jul 06 16:39:01.764655 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.764617 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5njq\" (UniqueName: \"kubernetes.io/projected/48e427b3-b218-4b68-90ba-0a4750c49f43-kube-api-access-l5njq\") pod \"insights-runtime-extractor-fqkj2\" (UID: \"48e427b3-b218-4b68-90ba-0a4750c49f43\") " pod="openshift-insights/insights-runtime-extractor-fqkj2" Jul 06 16:39:01.942672 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:01.942458 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pdh4h"] Jul 06 16:39:01.946779 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:39:01.946750 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b350fbc_4cba_421e_89d3_1f0df3318c34.slice/crio-e2b26701eaa5a966c332412cad79afa765172d6326573a265e12b236fded9b3d WatchSource:0}: Error finding container e2b26701eaa5a966c332412cad79afa765172d6326573a265e12b236fded9b3d: Status 404 returned error can't find the container with id e2b26701eaa5a966c332412cad79afa765172d6326573a265e12b236fded9b3d Jul 06 16:39:02.070019 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:02.069912 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:39:02.070019 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:02.069941 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:39:02.078592 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:02.078562 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 16:39:02.078976 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:02.078958 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q4r89\"" Jul 06 16:39:02.079337 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:02.079308 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 16:39:02.079831 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:02.079711 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 16:39:02.145785 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:02.145760 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2d0b8d6-be30-423c-840e-f20cebd694fd-metrics-tls\") pod \"dns-default-n82pj\" (UID: \"f2d0b8d6-be30-423c-840e-f20cebd694fd\") " pod="openshift-dns/dns-default-n82pj" Jul 06 16:39:02.145945 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:02.145929 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 16:39:02.146021 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:02.146011 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2d0b8d6-be30-423c-840e-f20cebd694fd-metrics-tls podName:f2d0b8d6-be30-423c-840e-f20cebd694fd nodeName:}" failed. No retries permitted until 2026-07-06 16:39:03.145989786 +0000 UTC m=+36.665480884 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2d0b8d6-be30-423c-840e-f20cebd694fd-metrics-tls") pod "dns-default-n82pj" (UID: "f2d0b8d6-be30-423c-840e-f20cebd694fd") : secret "dns-default-metrics-tls" not found Jul 06 16:39:02.217486 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:02.217454 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pdh4h" event={"ID":"5b350fbc-4cba-421e-89d3-1f0df3318c34","Type":"ContainerStarted","Data":"e2b26701eaa5a966c332412cad79afa765172d6326573a265e12b236fded9b3d"} Jul 06 16:39:02.246962 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:02.246925 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48e427b3-b218-4b68-90ba-0a4750c49f43-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fqkj2\" (UID: \"48e427b3-b218-4b68-90ba-0a4750c49f43\") " pod="openshift-insights/insights-runtime-extractor-fqkj2" Jul 06 16:39:02.247106 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:02.246981 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48e427b3-b218-4b68-90ba-0a4750c49f43-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fqkj2\" (UID: \"48e427b3-b218-4b68-90ba-0a4750c49f43\") " pod="openshift-insights/insights-runtime-extractor-fqkj2" Jul 06 16:39:02.247106 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:02.247081 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 16:39:02.247106 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:02.247091 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 16:39:02.247258 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:02.247143 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/48e427b3-b218-4b68-90ba-0a4750c49f43-kube-rbac-proxy-cm podName:48e427b3-b218-4b68-90ba-0a4750c49f43 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:03.247125773 +0000 UTC m=+36.766616871 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/48e427b3-b218-4b68-90ba-0a4750c49f43-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fqkj2" (UID: "48e427b3-b218-4b68-90ba-0a4750c49f43") : configmap "kube-rbac-proxy" not found Jul 06 16:39:02.247258 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:02.247163 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48e427b3-b218-4b68-90ba-0a4750c49f43-insights-runtime-extractor-tls podName:48e427b3-b218-4b68-90ba-0a4750c49f43 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:03.247153096 +0000 UTC m=+36.766644191 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/48e427b3-b218-4b68-90ba-0a4750c49f43-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fqkj2" (UID: "48e427b3-b218-4b68-90ba-0a4750c49f43") : secret "insights-runtime-extractor-tls" not found Jul 06 16:39:02.467207 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:02.467173 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-b2rtp"] Jul 06 16:39:02.472979 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:02.472957 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2rtp" Jul 06 16:39:02.476721 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:02.476701 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 06 16:39:02.477050 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:02.477030 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-7pwwg\"" Jul 06 16:39:02.477309 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:02.477291 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 06 16:39:02.495211 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:02.495189 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-b2rtp"] Jul 06 16:39:02.550321 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:02.550292 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9e75cc4-a9a1-4be1-b52b-0522e2f176d3-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-b2rtp\" (UID: \"a9e75cc4-a9a1-4be1-b52b-0522e2f176d3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2rtp" Jul 06 16:39:02.550455 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:02.550325 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a9e75cc4-a9a1-4be1-b52b-0522e2f176d3-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-b2rtp\" (UID: \"a9e75cc4-a9a1-4be1-b52b-0522e2f176d3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2rtp" Jul 06 16:39:02.651384 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:02.651348 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9e75cc4-a9a1-4be1-b52b-0522e2f176d3-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-b2rtp\" (UID: \"a9e75cc4-a9a1-4be1-b52b-0522e2f176d3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2rtp" Jul 06 16:39:02.651953 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:02.651407 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a9e75cc4-a9a1-4be1-b52b-0522e2f176d3-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-b2rtp\" (UID: \"a9e75cc4-a9a1-4be1-b52b-0522e2f176d3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2rtp" Jul 06 16:39:02.651953 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:02.651778 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 16:39:02.651953 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:02.651921 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9e75cc4-a9a1-4be1-b52b-0522e2f176d3-networking-console-plugin-cert podName:a9e75cc4-a9a1-4be1-b52b-0522e2f176d3 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:03.151898026 +0000 UTC m=+36.671389132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a9e75cc4-a9a1-4be1-b52b-0522e2f176d3-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-b2rtp" (UID: "a9e75cc4-a9a1-4be1-b52b-0522e2f176d3") : secret "networking-console-plugin-cert" not found Jul 06 16:39:02.652351 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:02.652328 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a9e75cc4-a9a1-4be1-b52b-0522e2f176d3-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-b2rtp\" (UID: \"a9e75cc4-a9a1-4be1-b52b-0522e2f176d3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2rtp" Jul 06 16:39:03.071046 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:03.070977 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:39:03.074713 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:03.074500 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 16:39:03.075044 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:03.075023 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ntdtl\"" Jul 06 16:39:03.156380 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:03.156333 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2d0b8d6-be30-423c-840e-f20cebd694fd-metrics-tls\") pod \"dns-default-n82pj\" (UID: \"f2d0b8d6-be30-423c-840e-f20cebd694fd\") " pod="openshift-dns/dns-default-n82pj" Jul 06 16:39:03.156581 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:03.156423 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9e75cc4-a9a1-4be1-b52b-0522e2f176d3-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-b2rtp\" (UID: \"a9e75cc4-a9a1-4be1-b52b-0522e2f176d3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2rtp" Jul 06 16:39:03.156581 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:03.156538 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 16:39:03.156581 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:03.156568 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 16:39:03.156759 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:03.156633 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2d0b8d6-be30-423c-840e-f20cebd694fd-metrics-tls podName:f2d0b8d6-be30-423c-840e-f20cebd694fd nodeName:}" failed. No retries permitted until 2026-07-06 16:39:05.156593227 +0000 UTC m=+38.676084322 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2d0b8d6-be30-423c-840e-f20cebd694fd-metrics-tls") pod "dns-default-n82pj" (UID: "f2d0b8d6-be30-423c-840e-f20cebd694fd") : secret "dns-default-metrics-tls" not found Jul 06 16:39:03.156759 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:03.156654 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9e75cc4-a9a1-4be1-b52b-0522e2f176d3-networking-console-plugin-cert podName:a9e75cc4-a9a1-4be1-b52b-0522e2f176d3 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:04.156644159 +0000 UTC m=+37.676135259 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a9e75cc4-a9a1-4be1-b52b-0522e2f176d3-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-b2rtp" (UID: "a9e75cc4-a9a1-4be1-b52b-0522e2f176d3") : secret "networking-console-plugin-cert" not found Jul 06 16:39:03.257348 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:03.257315 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48e427b3-b218-4b68-90ba-0a4750c49f43-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fqkj2\" (UID: \"48e427b3-b218-4b68-90ba-0a4750c49f43\") " pod="openshift-insights/insights-runtime-extractor-fqkj2" Jul 06 16:39:03.257553 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:03.257368 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48e427b3-b218-4b68-90ba-0a4750c49f43-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fqkj2\" (UID: \"48e427b3-b218-4b68-90ba-0a4750c49f43\") " pod="openshift-insights/insights-runtime-extractor-fqkj2" Jul 06 16:39:03.257553 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:03.257484 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 16:39:03.257679 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:03.257560 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48e427b3-b218-4b68-90ba-0a4750c49f43-insights-runtime-extractor-tls podName:48e427b3-b218-4b68-90ba-0a4750c49f43 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:05.257542394 +0000 UTC m=+38.777033508 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/48e427b3-b218-4b68-90ba-0a4750c49f43-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fqkj2" (UID: "48e427b3-b218-4b68-90ba-0a4750c49f43") : secret "insights-runtime-extractor-tls" not found Jul 06 16:39:03.257743 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:03.257696 2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 16:39:03.257810 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:03.257754 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/48e427b3-b218-4b68-90ba-0a4750c49f43-kube-rbac-proxy-cm podName:48e427b3-b218-4b68-90ba-0a4750c49f43 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:05.257736359 +0000 UTC m=+38.777227456 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/48e427b3-b218-4b68-90ba-0a4750c49f43-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fqkj2" (UID: "48e427b3-b218-4b68-90ba-0a4750c49f43") : configmap "kube-rbac-proxy" not found Jul 06 16:39:03.484376 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:03.484326 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-jdv5k"] Jul 06 16:39:03.513410 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:03.513379 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-jdv5k"] Jul 06 16:39:03.513580 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:03.513500 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-jdv5k" Jul 06 16:39:03.524209 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:03.524185 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 06 16:39:03.524902 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:03.524688 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-2f8g4\"" Jul 06 16:39:03.524902 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:03.524732 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 06 16:39:03.525179 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:03.524936 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 06 16:39:03.525263 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:03.525242 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 06 16:39:03.660862 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:03.660814 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1e9cbbfa-03e2-4cfe-8e6f-a11c252a27be-signing-cabundle\") pod \"service-ca-758f599c5-jdv5k\" (UID: \"1e9cbbfa-03e2-4cfe-8e6f-a11c252a27be\") " pod="openshift-service-ca/service-ca-758f599c5-jdv5k" Jul 06 16:39:03.660862 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:03.660865 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsxh2\" (UniqueName: \"kubernetes.io/projected/1e9cbbfa-03e2-4cfe-8e6f-a11c252a27be-kube-api-access-lsxh2\") pod \"service-ca-758f599c5-jdv5k\" (UID: \"1e9cbbfa-03e2-4cfe-8e6f-a11c252a27be\") " pod="openshift-service-ca/service-ca-758f599c5-jdv5k" Jul 06 16:39:03.661357 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:03.660949 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1e9cbbfa-03e2-4cfe-8e6f-a11c252a27be-signing-key\") pod \"service-ca-758f599c5-jdv5k\" (UID: \"1e9cbbfa-03e2-4cfe-8e6f-a11c252a27be\") " pod="openshift-service-ca/service-ca-758f599c5-jdv5k" Jul 06 16:39:03.762384 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:03.762296 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1e9cbbfa-03e2-4cfe-8e6f-a11c252a27be-signing-cabundle\") pod \"service-ca-758f599c5-jdv5k\" (UID: \"1e9cbbfa-03e2-4cfe-8e6f-a11c252a27be\") " pod="openshift-service-ca/service-ca-758f599c5-jdv5k" Jul 06 16:39:03.762384 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:03.762352 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsxh2\" (UniqueName: \"kubernetes.io/projected/1e9cbbfa-03e2-4cfe-8e6f-a11c252a27be-kube-api-access-lsxh2\") pod \"service-ca-758f599c5-jdv5k\" (UID: \"1e9cbbfa-03e2-4cfe-8e6f-a11c252a27be\") " pod="openshift-service-ca/service-ca-758f599c5-jdv5k" Jul 06 16:39:03.762625 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:03.762428 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1e9cbbfa-03e2-4cfe-8e6f-a11c252a27be-signing-key\") pod \"service-ca-758f599c5-jdv5k\" (UID: \"1e9cbbfa-03e2-4cfe-8e6f-a11c252a27be\") " pod="openshift-service-ca/service-ca-758f599c5-jdv5k" Jul 06 16:39:03.763137 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:03.763111 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1e9cbbfa-03e2-4cfe-8e6f-a11c252a27be-signing-cabundle\") pod \"service-ca-758f599c5-jdv5k\" (UID: \"1e9cbbfa-03e2-4cfe-8e6f-a11c252a27be\") " pod="openshift-service-ca/service-ca-758f599c5-jdv5k" Jul 06 16:39:03.765912 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:03.765884 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1e9cbbfa-03e2-4cfe-8e6f-a11c252a27be-signing-key\") pod \"service-ca-758f599c5-jdv5k\" (UID: \"1e9cbbfa-03e2-4cfe-8e6f-a11c252a27be\") " pod="openshift-service-ca/service-ca-758f599c5-jdv5k" Jul 06 16:39:03.773232 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:03.773208 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsxh2\" (UniqueName: \"kubernetes.io/projected/1e9cbbfa-03e2-4cfe-8e6f-a11c252a27be-kube-api-access-lsxh2\") pod \"service-ca-758f599c5-jdv5k\" (UID: \"1e9cbbfa-03e2-4cfe-8e6f-a11c252a27be\") " pod="openshift-service-ca/service-ca-758f599c5-jdv5k" Jul 06 16:39:03.825350 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:03.825313 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-jdv5k" Jul 06 16:39:04.165909 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:04.165826 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9e75cc4-a9a1-4be1-b52b-0522e2f176d3-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-b2rtp\" (UID: \"a9e75cc4-a9a1-4be1-b52b-0522e2f176d3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2rtp" Jul 06 16:39:04.166071 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:04.165983 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 16:39:04.166071 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:04.166055 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9e75cc4-a9a1-4be1-b52b-0522e2f176d3-networking-console-plugin-cert podName:a9e75cc4-a9a1-4be1-b52b-0522e2f176d3 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:06.166038948 +0000 UTC m=+39.685530042 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a9e75cc4-a9a1-4be1-b52b-0522e2f176d3-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-b2rtp" (UID: "a9e75cc4-a9a1-4be1-b52b-0522e2f176d3") : secret "networking-console-plugin-cert" not found Jul 06 16:39:05.173279 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:05.173241 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2d0b8d6-be30-423c-840e-f20cebd694fd-metrics-tls\") pod \"dns-default-n82pj\" (UID: \"f2d0b8d6-be30-423c-840e-f20cebd694fd\") " pod="openshift-dns/dns-default-n82pj" Jul 06 16:39:05.173823 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:05.173388 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 16:39:05.173823 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:05.173459 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2d0b8d6-be30-423c-840e-f20cebd694fd-metrics-tls podName:f2d0b8d6-be30-423c-840e-f20cebd694fd nodeName:}" failed. No retries permitted until 2026-07-06 16:39:09.173438004 +0000 UTC m=+42.692929101 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2d0b8d6-be30-423c-840e-f20cebd694fd-metrics-tls") pod "dns-default-n82pj" (UID: "f2d0b8d6-be30-423c-840e-f20cebd694fd") : secret "dns-default-metrics-tls" not found Jul 06 16:39:05.274015 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:05.273978 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48e427b3-b218-4b68-90ba-0a4750c49f43-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fqkj2\" (UID: \"48e427b3-b218-4b68-90ba-0a4750c49f43\") " pod="openshift-insights/insights-runtime-extractor-fqkj2" Jul 06 16:39:05.274158 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:05.274022 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48e427b3-b218-4b68-90ba-0a4750c49f43-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fqkj2\" (UID: \"48e427b3-b218-4b68-90ba-0a4750c49f43\") " pod="openshift-insights/insights-runtime-extractor-fqkj2" Jul 06 16:39:05.274158 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:05.274134 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 16:39:05.274224 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:05.274199 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48e427b3-b218-4b68-90ba-0a4750c49f43-insights-runtime-extractor-tls podName:48e427b3-b218-4b68-90ba-0a4750c49f43 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:09.274183216 +0000 UTC m=+42.793674312 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/48e427b3-b218-4b68-90ba-0a4750c49f43-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fqkj2" (UID: "48e427b3-b218-4b68-90ba-0a4750c49f43") : secret "insights-runtime-extractor-tls" not found Jul 06 16:39:05.274399 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:05.274385 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/48e427b3-b218-4b68-90ba-0a4750c49f43-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fqkj2\" (UID: \"48e427b3-b218-4b68-90ba-0a4750c49f43\") " pod="openshift-insights/insights-runtime-extractor-fqkj2" Jul 06 16:39:05.755850 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:05.755618 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-jdv5k"] Jul 06 16:39:05.761265 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:39:05.761230 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e9cbbfa_03e2_4cfe_8e6f_a11c252a27be.slice/crio-c51d5c8ce02186a406e9c4f93c30579db146363457d6f672b57b0776ab9fd1f9 WatchSource:0}: Error finding container c51d5c8ce02186a406e9c4f93c30579db146363457d6f672b57b0776ab9fd1f9: Status 404 returned error can't find the container with id c51d5c8ce02186a406e9c4f93c30579db146363457d6f672b57b0776ab9fd1f9 Jul 06 16:39:06.179812 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:06.179776 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9e75cc4-a9a1-4be1-b52b-0522e2f176d3-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-b2rtp\" (UID: \"a9e75cc4-a9a1-4be1-b52b-0522e2f176d3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2rtp" Jul 06 16:39:06.180252 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:06.179898 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 16:39:06.180252 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:06.179952 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9e75cc4-a9a1-4be1-b52b-0522e2f176d3-networking-console-plugin-cert podName:a9e75cc4-a9a1-4be1-b52b-0522e2f176d3 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:10.17993845 +0000 UTC m=+43.699429549 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a9e75cc4-a9a1-4be1-b52b-0522e2f176d3-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-b2rtp" (UID: "a9e75cc4-a9a1-4be1-b52b-0522e2f176d3") : secret "networking-console-plugin-cert" not found Jul 06 16:39:06.231108 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:06.231072 2574 generic.go:356] "Generic (PLEG): container finished" podID="563411d8-43a0-49ab-b73b-c29dec275f46" containerID="e092df7b7c812fa8e61040b05b5ab1ffe57668369f0ddfb00e25e56da35e29ba" exitCode=0 Jul 06 16:39:06.231288 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:06.231161 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p9rnx" event={"ID":"563411d8-43a0-49ab-b73b-c29dec275f46","Type":"ContainerDied","Data":"e092df7b7c812fa8e61040b05b5ab1ffe57668369f0ddfb00e25e56da35e29ba"} Jul 06 16:39:06.232680 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:06.232643 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pdh4h" event={"ID":"5b350fbc-4cba-421e-89d3-1f0df3318c34","Type":"ContainerStarted","Data":"c607b4dadce94c17cd1022cf6b1a79e620572907658b09d07aeeeba68b29cee3"} Jul 06 16:39:06.233703 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:06.233680 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-jdv5k" event={"ID":"1e9cbbfa-03e2-4cfe-8e6f-a11c252a27be","Type":"ContainerStarted","Data":"c51d5c8ce02186a406e9c4f93c30579db146363457d6f672b57b0776ab9fd1f9"} Jul 06 16:39:06.315809 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:06.315758 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pdh4h" podStartSLOduration=1.665209886 podStartE2EDuration="5.315741641s" totalImagesPullingTime="3.650531755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:39:06.314673988 +0000 UTC m=+39.834165107" watchObservedRunningTime="2026-07-06 16:39:06.315741641 +0000 UTC m=+39.835232756" Jul 06 16:39:07.238442 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:07.238400 2574 generic.go:356] "Generic (PLEG): container finished" podID="563411d8-43a0-49ab-b73b-c29dec275f46" containerID="0d5740093f4fc3d1fab32306c45a9d7114296ec75272b20af81161d12a596947" exitCode=0 Jul 06 16:39:07.239122 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:07.238481 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p9rnx" event={"ID":"563411d8-43a0-49ab-b73b-c29dec275f46","Type":"ContainerDied","Data":"0d5740093f4fc3d1fab32306c45a9d7114296ec75272b20af81161d12a596947"} Jul 06 16:39:08.245090 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:08.245050 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p9rnx" event={"ID":"563411d8-43a0-49ab-b73b-c29dec275f46","Type":"ContainerStarted","Data":"2668ae9c270b6396e30541040e4375e091b85200e1f0ef7f4808b6f8a8c5b1a8"} Jul 06 16:39:08.281717 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:08.281668 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-p9rnx" podStartSLOduration=7.653978403 podStartE2EDuration="41.281654149s" totalImagesPullingTime="33.627675746s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:38:27 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:39:08.281456419 +0000 UTC m=+41.800947535" watchObservedRunningTime="2026-07-06 16:39:08.281654149 +0000 UTC m=+41.801145257" Jul 06 16:39:09.204912 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:09.204878 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2d0b8d6-be30-423c-840e-f20cebd694fd-metrics-tls\") pod \"dns-default-n82pj\" (UID: \"f2d0b8d6-be30-423c-840e-f20cebd694fd\") " pod="openshift-dns/dns-default-n82pj" Jul 06 16:39:09.205069 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:09.205038 2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 16:39:09.205111 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:09.205098 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2d0b8d6-be30-423c-840e-f20cebd694fd-metrics-tls podName:f2d0b8d6-be30-423c-840e-f20cebd694fd nodeName:}" failed. No retries permitted until 2026-07-06 16:39:17.205081812 +0000 UTC m=+50.724572916 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f2d0b8d6-be30-423c-840e-f20cebd694fd-metrics-tls") pod "dns-default-n82pj" (UID: "f2d0b8d6-be30-423c-840e-f20cebd694fd") : secret "dns-default-metrics-tls" not found Jul 06 16:39:09.305925 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:09.305895 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48e427b3-b218-4b68-90ba-0a4750c49f43-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fqkj2\" (UID: \"48e427b3-b218-4b68-90ba-0a4750c49f43\") " pod="openshift-insights/insights-runtime-extractor-fqkj2" Jul 06 16:39:09.306343 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:09.306051 2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 16:39:09.306343 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:09.306128 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48e427b3-b218-4b68-90ba-0a4750c49f43-insights-runtime-extractor-tls podName:48e427b3-b218-4b68-90ba-0a4750c49f43 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:17.306106796 +0000 UTC m=+50.825597895 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/48e427b3-b218-4b68-90ba-0a4750c49f43-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fqkj2" (UID: "48e427b3-b218-4b68-90ba-0a4750c49f43") : secret "insights-runtime-extractor-tls" not found Jul 06 16:39:10.212649 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:10.212620 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9e75cc4-a9a1-4be1-b52b-0522e2f176d3-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-b2rtp\" (UID: \"a9e75cc4-a9a1-4be1-b52b-0522e2f176d3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2rtp" Jul 06 16:39:10.212747 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:10.212737 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 16:39:10.212794 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:10.212785 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9e75cc4-a9a1-4be1-b52b-0522e2f176d3-networking-console-plugin-cert podName:a9e75cc4-a9a1-4be1-b52b-0522e2f176d3 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:18.212773019 +0000 UTC m=+51.732264113 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a9e75cc4-a9a1-4be1-b52b-0522e2f176d3-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-b2rtp" (UID: "a9e75cc4-a9a1-4be1-b52b-0522e2f176d3") : secret "networking-console-plugin-cert" not found Jul 06 16:39:11.258102 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:11.258066 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-jdv5k" event={"ID":"1e9cbbfa-03e2-4cfe-8e6f-a11c252a27be","Type":"ContainerStarted","Data":"a6ccb73e9748b1edc9330fe0ba36404df4b119ffbe1028e8619e7cbee77b64a1"} Jul 06 16:39:11.304794 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:11.304744 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-jdv5k" podStartSLOduration=3.90039835 podStartE2EDuration="8.30472977s" totalImagesPullingTime="4.40433142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:39:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:39:11.304244113 +0000 UTC m=+44.823735229" watchObservedRunningTime="2026-07-06 16:39:11.30472977 +0000 UTC m=+44.824220905" Jul 06 16:39:17.267559 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:17.267500 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2d0b8d6-be30-423c-840e-f20cebd694fd-metrics-tls\") pod \"dns-default-n82pj\" (UID: \"f2d0b8d6-be30-423c-840e-f20cebd694fd\") " pod="openshift-dns/dns-default-n82pj" Jul 06 16:39:17.270977 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:17.270953 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f2d0b8d6-be30-423c-840e-f20cebd694fd-metrics-tls\") pod \"dns-default-n82pj\" (UID: \"f2d0b8d6-be30-423c-840e-f20cebd694fd\") " pod="openshift-dns/dns-default-n82pj" Jul 06 16:39:17.311595 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:17.311561 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n82pj" Jul 06 16:39:17.368677 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:17.368644 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48e427b3-b218-4b68-90ba-0a4750c49f43-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fqkj2\" (UID: \"48e427b3-b218-4b68-90ba-0a4750c49f43\") " pod="openshift-insights/insights-runtime-extractor-fqkj2" Jul 06 16:39:17.371732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:17.371266 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/48e427b3-b218-4b68-90ba-0a4750c49f43-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fqkj2\" (UID: \"48e427b3-b218-4b68-90ba-0a4750c49f43\") " pod="openshift-insights/insights-runtime-extractor-fqkj2" Jul 06 16:39:17.441919 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:17.441886 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fqkj2" Jul 06 16:39:17.473110 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:17.473086 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n82pj"] Jul 06 16:39:17.630394 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:17.630357 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fqkj2"] Jul 06 16:39:17.635048 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:39:17.635017 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e427b3_b218_4b68_90ba_0a4750c49f43.slice/crio-377d2124a29b95fbb76ddc15c15e32d1478b1631f7c19bd58a4b82597fe600dd WatchSource:0}: Error finding container 377d2124a29b95fbb76ddc15c15e32d1478b1631f7c19bd58a4b82597fe600dd: Status 404 returned error can't find the container with id 377d2124a29b95fbb76ddc15c15e32d1478b1631f7c19bd58a4b82597fe600dd Jul 06 16:39:18.276222 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:18.276187 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9e75cc4-a9a1-4be1-b52b-0522e2f176d3-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-b2rtp\" (UID: \"a9e75cc4-a9a1-4be1-b52b-0522e2f176d3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2rtp" Jul 06 16:39:18.276690 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:18.276332 2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 06 16:39:18.276690 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:18.276406 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9e75cc4-a9a1-4be1-b52b-0522e2f176d3-networking-console-plugin-cert podName:a9e75cc4-a9a1-4be1-b52b-0522e2f176d3 nodeName:}" failed. No retries permitted until 2026-07-06 16:39:34.276383803 +0000 UTC m=+67.795874913 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/a9e75cc4-a9a1-4be1-b52b-0522e2f176d3-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-b2rtp" (UID: "a9e75cc4-a9a1-4be1-b52b-0522e2f176d3") : secret "networking-console-plugin-cert" not found Jul 06 16:39:18.280840 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:18.280801 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fqkj2" event={"ID":"48e427b3-b218-4b68-90ba-0a4750c49f43","Type":"ContainerStarted","Data":"377d2124a29b95fbb76ddc15c15e32d1478b1631f7c19bd58a4b82597fe600dd"} Jul 06 16:39:18.282362 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:18.282305 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n82pj" event={"ID":"f2d0b8d6-be30-423c-840e-f20cebd694fd","Type":"ContainerStarted","Data":"625f0f48eb25b7c5619819c08df9826447958500de88c09cacb63c8191aef5dd"} Jul 06 16:39:20.291827 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:20.291679 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n82pj" event={"ID":"f2d0b8d6-be30-423c-840e-f20cebd694fd","Type":"ContainerStarted","Data":"cc99f6258c02d98f8d1ad50f7e3791d2ac215ccc1f5a9901348e4ac9eae9cbc7"} Jul 06 16:39:20.291827 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:20.291753 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n82pj" event={"ID":"f2d0b8d6-be30-423c-840e-f20cebd694fd","Type":"ContainerStarted","Data":"854b92bd590ff8188521d5e5c7e81c678ff356a83b78fef5bb5d93f79b3df1de"} Jul 06 16:39:20.291827 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:20.291828 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-n82pj" Jul 06 16:39:20.357055 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:20.356999 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-n82pj" podStartSLOduration=17.131876005 podStartE2EDuration="19.356978309s" totalImagesPullingTime="2.225102304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:39:20.356850219 +0000 UTC m=+53.876341338" watchObservedRunningTime="2026-07-06 16:39:20.356978309 +0000 UTC m=+53.876469428" Jul 06 16:39:22.297479 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:22.297444 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fqkj2" event={"ID":"48e427b3-b218-4b68-90ba-0a4750c49f43","Type":"ContainerStarted","Data":"af4b7fe013217ecaa1d7051faa8dce676842545013dee30578fd5c428673f317"} Jul 06 16:39:23.301857 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:23.301824 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fqkj2" event={"ID":"48e427b3-b218-4b68-90ba-0a4750c49f43","Type":"ContainerStarted","Data":"a94590dc06852cf40b9952667df3b8e4c034586cd68d39f186581071b42fa3dd"} Jul 06 16:39:24.457440 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.457408 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r"] Jul 06 16:39:24.459510 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.459495 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r" Jul 06 16:39:24.465428 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.465402 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 06 16:39:24.465862 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.465403 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 06 16:39:24.466055 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.466038 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 16:39:24.471479 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.471459 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 16:39:24.492427 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.492407 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 16:39:24.508861 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.508841 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 06 16:39:24.510964 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.510949 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 06 16:39:24.524671 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.524649 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/221cd668-894b-4a2e-a15d-a68621f29485-hub\") pod \"cluster-proxy-proxy-agent-7c474dd899-tpx5r\" (UID: \"221cd668-894b-4a2e-a15d-a68621f29485\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r" Jul 06 16:39:24.524730 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.524680 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/221cd668-894b-4a2e-a15d-a68621f29485-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c474dd899-tpx5r\" (UID: \"221cd668-894b-4a2e-a15d-a68621f29485\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r" Jul 06 16:39:24.524730 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.524697 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skhxv\" (UniqueName: \"kubernetes.io/projected/221cd668-894b-4a2e-a15d-a68621f29485-kube-api-access-skhxv\") pod \"cluster-proxy-proxy-agent-7c474dd899-tpx5r\" (UID: \"221cd668-894b-4a2e-a15d-a68621f29485\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r" Jul 06 16:39:24.524819 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.524748 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/221cd668-894b-4a2e-a15d-a68621f29485-ca\") pod \"cluster-proxy-proxy-agent-7c474dd899-tpx5r\" (UID: \"221cd668-894b-4a2e-a15d-a68621f29485\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r" Jul 06 16:39:24.524819 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.524778 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/221cd668-894b-4a2e-a15d-a68621f29485-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c474dd899-tpx5r\" (UID: \"221cd668-894b-4a2e-a15d-a68621f29485\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r" Jul 06 16:39:24.524885 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.524868 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/221cd668-894b-4a2e-a15d-a68621f29485-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c474dd899-tpx5r\" (UID: \"221cd668-894b-4a2e-a15d-a68621f29485\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r" Jul 06 16:39:24.626135 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.626081 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/221cd668-894b-4a2e-a15d-a68621f29485-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c474dd899-tpx5r\" (UID: \"221cd668-894b-4a2e-a15d-a68621f29485\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r" Jul 06 16:39:24.626321 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.626166 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/221cd668-894b-4a2e-a15d-a68621f29485-hub\") pod \"cluster-proxy-proxy-agent-7c474dd899-tpx5r\" (UID: \"221cd668-894b-4a2e-a15d-a68621f29485\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r" Jul 06 16:39:24.626321 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.626190 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/221cd668-894b-4a2e-a15d-a68621f29485-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c474dd899-tpx5r\" (UID: \"221cd668-894b-4a2e-a15d-a68621f29485\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r" Jul 06 16:39:24.626321 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.626231 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skhxv\" (UniqueName: \"kubernetes.io/projected/221cd668-894b-4a2e-a15d-a68621f29485-kube-api-access-skhxv\") pod \"cluster-proxy-proxy-agent-7c474dd899-tpx5r\" (UID: \"221cd668-894b-4a2e-a15d-a68621f29485\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r" Jul 06 16:39:24.626321 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.626269 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/221cd668-894b-4a2e-a15d-a68621f29485-ca\") pod \"cluster-proxy-proxy-agent-7c474dd899-tpx5r\" (UID: \"221cd668-894b-4a2e-a15d-a68621f29485\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r" Jul 06 16:39:24.626321 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.626292 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/221cd668-894b-4a2e-a15d-a68621f29485-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c474dd899-tpx5r\" (UID: \"221cd668-894b-4a2e-a15d-a68621f29485\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r" Jul 06 16:39:24.626977 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.626950 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/221cd668-894b-4a2e-a15d-a68621f29485-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7c474dd899-tpx5r\" (UID: \"221cd668-894b-4a2e-a15d-a68621f29485\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r" Jul 06 16:39:24.628819 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.628789 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/221cd668-894b-4a2e-a15d-a68621f29485-ca\") pod \"cluster-proxy-proxy-agent-7c474dd899-tpx5r\" (UID: \"221cd668-894b-4a2e-a15d-a68621f29485\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r" Jul 06 16:39:24.628976 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.628890 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/221cd668-894b-4a2e-a15d-a68621f29485-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7c474dd899-tpx5r\" (UID: \"221cd668-894b-4a2e-a15d-a68621f29485\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r" Jul 06 16:39:24.628976 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.628906 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/221cd668-894b-4a2e-a15d-a68621f29485-hub\") pod \"cluster-proxy-proxy-agent-7c474dd899-tpx5r\" (UID: \"221cd668-894b-4a2e-a15d-a68621f29485\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r" Jul 06 16:39:24.628976 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.628970 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/221cd668-894b-4a2e-a15d-a68621f29485-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7c474dd899-tpx5r\" (UID: \"221cd668-894b-4a2e-a15d-a68621f29485\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r" Jul 06 16:39:24.655991 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.655962 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-nh4dr"] Jul 06 16:39:24.657928 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.657914 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-nh4dr" Jul 06 16:39:24.676290 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:24.676250 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"downloads-685bfb5f64-nh4dr\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="832a849a-0ae6-4242-8ba1-f64b0f12038a" pod="openshift-console/downloads-685bfb5f64-nh4dr" Jul 06 16:39:24.676404 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:24.676351 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"default-dockercfg-c9t6d\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"default-dockercfg-c9t6d\"" type="*v1.Secret" Jul 06 16:39:24.676404 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:24.676349 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 06 16:39:24.676485 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:39:24.676422 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-console\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 06 16:39:24.691989 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.691944 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skhxv\" (UniqueName: \"kubernetes.io/projected/221cd668-894b-4a2e-a15d-a68621f29485-kube-api-access-skhxv\") pod \"cluster-proxy-proxy-agent-7c474dd899-tpx5r\" (UID: \"221cd668-894b-4a2e-a15d-a68621f29485\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r" Jul 06 16:39:24.692385 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.692369 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-nh4dr"] Jul 06 16:39:24.717863 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.717784 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r"] Jul 06 16:39:24.727095 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.727076 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfk4b\" (UniqueName: \"kubernetes.io/projected/832a849a-0ae6-4242-8ba1-f64b0f12038a-kube-api-access-cfk4b\") pod \"downloads-685bfb5f64-nh4dr\" (UID: \"832a849a-0ae6-4242-8ba1-f64b0f12038a\") " pod="openshift-console/downloads-685bfb5f64-nh4dr" Jul 06 16:39:24.781161 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.781122 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r" Jul 06 16:39:24.827986 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.827538 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfk4b\" (UniqueName: \"kubernetes.io/projected/832a849a-0ae6-4242-8ba1-f64b0f12038a-kube-api-access-cfk4b\") pod \"downloads-685bfb5f64-nh4dr\" (UID: \"832a849a-0ae6-4242-8ba1-f64b0f12038a\") " pod="openshift-console/downloads-685bfb5f64-nh4dr" Jul 06 16:39:24.951398 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:24.951368 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r"] Jul 06 16:39:24.954299 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:39:24.954271 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod221cd668_894b_4a2e_a15d_a68621f29485.slice/crio-2516070ab4704e8ec0505e8001e57bdeb908cb2b6fe275a3349febc4bd3b05b3 WatchSource:0}: Error finding container 2516070ab4704e8ec0505e8001e57bdeb908cb2b6fe275a3349febc4bd3b05b3: Status 404 returned error can't find the container with id 2516070ab4704e8ec0505e8001e57bdeb908cb2b6fe275a3349febc4bd3b05b3 Jul 06 16:39:25.308601 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:25.308568 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r" event={"ID":"221cd668-894b-4a2e-a15d-a68621f29485","Type":"ContainerStarted","Data":"2516070ab4704e8ec0505e8001e57bdeb908cb2b6fe275a3349febc4bd3b05b3"} Jul 06 16:39:25.547560 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:25.547530 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 06 16:39:25.559022 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:25.558950 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-c9t6d\"" Jul 06 16:39:25.617060 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:25.617032 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 06 16:39:25.628379 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:25.628353 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfk4b\" (UniqueName: \"kubernetes.io/projected/832a849a-0ae6-4242-8ba1-f64b0f12038a-kube-api-access-cfk4b\") pod \"downloads-685bfb5f64-nh4dr\" (UID: \"832a849a-0ae6-4242-8ba1-f64b0f12038a\") " pod="openshift-console/downloads-685bfb5f64-nh4dr" Jul 06 16:39:25.866803 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:25.866714 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-nh4dr" Jul 06 16:39:26.044193 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:26.044113 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-nh4dr"] Jul 06 16:39:26.049986 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:39:26.049572 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod832a849a_0ae6_4242_8ba1_f64b0f12038a.slice/crio-449505814fe412eff4e9927d277cc285a42ef3e922ac72c7ce84f497898f0782 WatchSource:0}: Error finding container 449505814fe412eff4e9927d277cc285a42ef3e922ac72c7ce84f497898f0782: Status 404 returned error can't find the container with id 449505814fe412eff4e9927d277cc285a42ef3e922ac72c7ce84f497898f0782 Jul 06 16:39:26.313696 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:26.313646 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-nh4dr" event={"ID":"832a849a-0ae6-4242-8ba1-f64b0f12038a","Type":"ContainerStarted","Data":"449505814fe412eff4e9927d277cc285a42ef3e922ac72c7ce84f497898f0782"} Jul 06 16:39:27.221511 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:27.220172 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zwzvg" Jul 06 16:39:29.330786 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:29.330690 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r" event={"ID":"221cd668-894b-4a2e-a15d-a68621f29485","Type":"ContainerStarted","Data":"bac8891336fc9c7e42e1da946109534bb1f401b0fdbef5b7a0406fadc8048420"} Jul 06 16:39:29.330786 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:29.330736 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r" event={"ID":"221cd668-894b-4a2e-a15d-a68621f29485","Type":"ContainerStarted","Data":"44ffd4fcd1be46ca62aa2446f1f03f746f0cf354852d6857464a4be0bcc02c88"} Jul 06 16:39:29.330786 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:29.330752 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r" event={"ID":"221cd668-894b-4a2e-a15d-a68621f29485","Type":"ContainerStarted","Data":"c7dbdcb04eb9f5e76f503c7db3c0997cab18745e217f4ce419b4e4e277555be6"} Jul 06 16:39:29.358508 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:29.358461 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7c474dd899-tpx5r" podStartSLOduration=1.304847072 podStartE2EDuration="5.35844677s" totalImagesPullingTime="4.053599698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:39:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:39:29.358210051 +0000 UTC m=+62.877701167" watchObservedRunningTime="2026-07-06 16:39:29.35844677 +0000 UTC m=+62.877937885" Jul 06 16:39:30.296912 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:30.296883 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-n82pj" Jul 06 16:39:32.344715 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:32.344672 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fqkj2" event={"ID":"48e427b3-b218-4b68-90ba-0a4750c49f43","Type":"ContainerStarted","Data":"d013c937525921ab169f8c2529caede69226db28ebb6641090cc19dccd6fe7da"} Jul 06 16:39:32.372738 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:32.372694 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fqkj2" podStartSLOduration=17.540289846 podStartE2EDuration="31.372679601s" totalImagesPullingTime="13.832389755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:39:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:39:32.372121946 +0000 UTC m=+65.891613075" watchObservedRunningTime="2026-07-06 16:39:32.372679601 +0000 UTC m=+65.892170717" Jul 06 16:39:32.694049 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:32.694015 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39a924d0-6e9a-48bb-9972-6d40dd4c50dc-metrics-certs\") pod \"network-metrics-daemon-n9kmw\" (UID: \"39a924d0-6e9a-48bb-9972-6d40dd4c50dc\") " pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:39:32.694225 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:32.694069 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ce13f97-b996-4f40-86fb-e0453bd3a476-original-pull-secret\") pod \"global-pull-secret-syncer-nnd4r\" (UID: \"9ce13f97-b996-4f40-86fb-e0453bd3a476\") " pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:39:32.697989 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:32.697964 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 16:39:32.698125 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:32.698057 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 16:39:32.707636 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:32.707588 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ce13f97-b996-4f40-86fb-e0453bd3a476-original-pull-secret\") pod \"global-pull-secret-syncer-nnd4r\" (UID: \"9ce13f97-b996-4f40-86fb-e0453bd3a476\") " pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:39:32.707745 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:32.707639 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39a924d0-6e9a-48bb-9972-6d40dd4c50dc-metrics-certs\") pod \"network-metrics-daemon-n9kmw\" (UID: \"39a924d0-6e9a-48bb-9972-6d40dd4c50dc\") " pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:39:32.787736 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:32.787703 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ntdtl\"" Jul 06 16:39:32.792798 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:32.792778 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-n9kmw" Jul 06 16:39:32.794557 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:32.794511 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szlc4\" (UniqueName: \"kubernetes.io/projected/51bffaee-68a0-486d-8a62-a04c17b4ae9d-kube-api-access-szlc4\") pod \"network-check-target-j5khq\" (UID: \"51bffaee-68a0-486d-8a62-a04c17b4ae9d\") " pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:39:32.799235 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:32.799211 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 16:39:32.808593 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:32.808570 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 16:39:32.818105 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:32.818084 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szlc4\" (UniqueName: \"kubernetes.io/projected/51bffaee-68a0-486d-8a62-a04c17b4ae9d-kube-api-access-szlc4\") pod \"network-check-target-j5khq\" (UID: \"51bffaee-68a0-486d-8a62-a04c17b4ae9d\") " pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:39:32.929213 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:32.929180 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-n9kmw"] Jul 06 16:39:32.933930 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:39:32.933905 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a924d0_6e9a_48bb_9972_6d40dd4c50dc.slice/crio-9cd78d72d1ddc62018ae4b27ecec982fb17daee44393c547f47d5dbb848d31f6 WatchSource:0}: Error finding container 9cd78d72d1ddc62018ae4b27ecec982fb17daee44393c547f47d5dbb848d31f6: Status 404 returned error can't find the container with id 9cd78d72d1ddc62018ae4b27ecec982fb17daee44393c547f47d5dbb848d31f6 Jul 06 16:39:32.986868 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:32.986747 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q4r89\"" Jul 06 16:39:32.989010 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:32.988984 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nnd4r" Jul 06 16:39:32.994599 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:32.994573 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:39:33.148580 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:33.148543 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j5khq"] Jul 06 16:39:33.153136 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:39:33.153106 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51bffaee_68a0_486d_8a62_a04c17b4ae9d.slice/crio-eb3de5aa4b51cd73a5112353938c282297d9a33e5d0ba5bc535f2c98fad60509 WatchSource:0}: Error finding container eb3de5aa4b51cd73a5112353938c282297d9a33e5d0ba5bc535f2c98fad60509: Status 404 returned error can't find the container with id eb3de5aa4b51cd73a5112353938c282297d9a33e5d0ba5bc535f2c98fad60509 Jul 06 16:39:33.172612 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:33.172591 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nnd4r"] Jul 06 16:39:33.175920 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:39:33.175898 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ce13f97_b996_4f40_86fb_e0453bd3a476.slice/crio-2f8bd28f85b1522ba4b10d9803c1e49cbd997015983243e619000a2282e59a67 WatchSource:0}: Error finding container 2f8bd28f85b1522ba4b10d9803c1e49cbd997015983243e619000a2282e59a67: Status 404 returned error can't find the container with id 2f8bd28f85b1522ba4b10d9803c1e49cbd997015983243e619000a2282e59a67 Jul 06 16:39:33.348739 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:33.348651 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nnd4r" event={"ID":"9ce13f97-b996-4f40-86fb-e0453bd3a476","Type":"ContainerStarted","Data":"2f8bd28f85b1522ba4b10d9803c1e49cbd997015983243e619000a2282e59a67"} Jul 06 16:39:33.349735 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:33.349712 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n9kmw" event={"ID":"39a924d0-6e9a-48bb-9972-6d40dd4c50dc","Type":"ContainerStarted","Data":"9cd78d72d1ddc62018ae4b27ecec982fb17daee44393c547f47d5dbb848d31f6"} Jul 06 16:39:33.350926 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:33.350898 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j5khq" event={"ID":"51bffaee-68a0-486d-8a62-a04c17b4ae9d","Type":"ContainerStarted","Data":"eb3de5aa4b51cd73a5112353938c282297d9a33e5d0ba5bc535f2c98fad60509"} Jul 06 16:39:34.306573 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:34.306478 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9e75cc4-a9a1-4be1-b52b-0522e2f176d3-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-b2rtp\" (UID: \"a9e75cc4-a9a1-4be1-b52b-0522e2f176d3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2rtp" Jul 06 16:39:34.312083 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:34.312028 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a9e75cc4-a9a1-4be1-b52b-0522e2f176d3-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-b2rtp\" (UID: \"a9e75cc4-a9a1-4be1-b52b-0522e2f176d3\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2rtp" Jul 06 16:39:34.586329 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:34.586248 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-7pwwg\"" Jul 06 16:39:34.593598 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:34.593562 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2rtp" Jul 06 16:39:34.766952 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:34.766865 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-b2rtp"] Jul 06 16:39:34.772128 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:39:34.772095 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e75cc4_a9a1_4be1_b52b_0522e2f176d3.slice/crio-06efccb2f2e58f93cf81db70de52d9fa83c29df2a8951c5ea3015f886ab0fe13 WatchSource:0}: Error finding container 06efccb2f2e58f93cf81db70de52d9fa83c29df2a8951c5ea3015f886ab0fe13: Status 404 returned error can't find the container with id 06efccb2f2e58f93cf81db70de52d9fa83c29df2a8951c5ea3015f886ab0fe13 Jul 06 16:39:35.361508 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:35.361466 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2rtp" event={"ID":"a9e75cc4-a9a1-4be1-b52b-0522e2f176d3","Type":"ContainerStarted","Data":"06efccb2f2e58f93cf81db70de52d9fa83c29df2a8951c5ea3015f886ab0fe13"} Jul 06 16:39:36.369281 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:36.369235 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n9kmw" event={"ID":"39a924d0-6e9a-48bb-9972-6d40dd4c50dc","Type":"ContainerStarted","Data":"94bde73ae7ef81af35bea9fbdbe710e71f96073b3c8bb75898ba7c0a4cf0387f"} Jul 06 16:39:36.369761 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:36.369296 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-n9kmw" event={"ID":"39a924d0-6e9a-48bb-9972-6d40dd4c50dc","Type":"ContainerStarted","Data":"baa54b902ee722bfe8548a3bf2f07184d55dd67779c74cdc25c1d8d622fd9fb3"} Jul 06 16:39:36.405481 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:36.405426 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-n9kmw" podStartSLOduration=66.754076297 podStartE2EDuration="1m9.405406297s" totalImagesPullingTime="2.65133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:39:36.40289237 +0000 UTC m=+69.922383489" watchObservedRunningTime="2026-07-06 16:39:36.405406297 +0000 UTC m=+69.924897415" Jul 06 16:39:39.382185 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:39.382147 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nnd4r" event={"ID":"9ce13f97-b996-4f40-86fb-e0453bd3a476","Type":"ContainerStarted","Data":"b54d15ebc3b39e6c5cbf0f2dfe133af485ab3f7e62c38ceab9a0843d3a3a6bf0"} Jul 06 16:39:39.384024 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:39.383992 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j5khq" event={"ID":"51bffaee-68a0-486d-8a62-a04c17b4ae9d","Type":"ContainerStarted","Data":"9a19d473df788302d26b22e355ca80d1b7eec0d1a254cf654b994b89ff764c59"} Jul 06 16:39:39.384156 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:39.384052 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:39:39.385623 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:39.385596 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2rtp" event={"ID":"a9e75cc4-a9a1-4be1-b52b-0522e2f176d3","Type":"ContainerStarted","Data":"fe0b8f69ae572d735e87c041b72c8c1cc63b3438ff5acf228bdcabdaabd4940a"} Jul 06 16:39:39.427810 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:39.427756 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nnd4r" podStartSLOduration=66.615494259 podStartE2EDuration="1m12.427738003s" totalImagesPullingTime="5.812243744s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:39:39.427669564 +0000 UTC m=+72.947160680" watchObservedRunningTime="2026-07-06 16:39:39.427738003 +0000 UTC m=+72.947229118" Jul 06 16:39:39.474788 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:39.474742 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j5khq" podStartSLOduration=66.648893384 podStartE2EDuration="1m12.474718931s" totalImagesPullingTime="5.825825547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:39:39.473371016 +0000 UTC m=+72.992862155" watchObservedRunningTime="2026-07-06 16:39:39.474718931 +0000 UTC m=+72.994210046" Jul 06 16:39:49.423200 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:49.423120 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-nh4dr" event={"ID":"832a849a-0ae6-4242-8ba1-f64b0f12038a","Type":"ContainerStarted","Data":"13fca1148257a84755c03e419533c59cc01164edf7a40951341bdf61ba0ede40"} Jul 06 16:39:49.423614 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:49.423281 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-nh4dr" Jul 06 16:39:49.424550 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:49.424526 2574 patch_prober.go:28] interesting pod/downloads-685bfb5f64-nh4dr container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.12:8080/\": dial tcp 10.134.0.12:8080: connect: connection refused" start-of-body= Jul 06 16:39:49.424654 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:49.424573 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-685bfb5f64-nh4dr" podUID="832a849a-0ae6-4242-8ba1-f64b0f12038a" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.12:8080/\": dial tcp 10.134.0.12:8080: connect: connection refused" Jul 06 16:39:49.458687 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:49.458631 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-nh4dr" podStartSLOduration=2.370164211 podStartE2EDuration="25.458615369s" totalImagesPullingTime="23.088451158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:39:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:39:49.457931299 +0000 UTC m=+82.977422416" watchObservedRunningTime="2026-07-06 16:39:49.458615369 +0000 UTC m=+82.978106493" Jul 06 16:39:49.459226 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:49.459194 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2rtp" podStartSLOduration=43.250784459 podStartE2EDuration="47.459182938s" totalImagesPullingTime="4.208398479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:39:39.529898433 +0000 UTC m=+73.049389550" watchObservedRunningTime="2026-07-06 16:39:49.459182938 +0000 UTC m=+82.978674054" Jul 06 16:39:50.446406 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:50.446368 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-nh4dr" Jul 06 16:39:52.060601 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.060565 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-864c7cc647-fr9wh"] Jul 06 16:39:52.090915 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.090878 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-864c7cc647-fr9wh"] Jul 06 16:39:52.091083 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.091005 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:39:52.097126 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.096930 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 06 16:39:52.097126 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.097018 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 06 16:39:52.097323 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.097176 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 06 16:39:52.097323 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.097188 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 06 16:39:52.097592 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.097571 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-klg6g\"" Jul 06 16:39:52.097751 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.097688 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 06 16:39:52.103430 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.103411 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 06 16:39:52.143512 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.143479 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96417ce0-8b2f-49ad-8052-63687d8a46dc-console-oauth-config\") pod \"console-864c7cc647-fr9wh\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:39:52.143512 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.143531 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96417ce0-8b2f-49ad-8052-63687d8a46dc-console-config\") pod \"console-864c7cc647-fr9wh\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:39:52.143749 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.143560 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmtgw\" (UniqueName: \"kubernetes.io/projected/96417ce0-8b2f-49ad-8052-63687d8a46dc-kube-api-access-vmtgw\") pod \"console-864c7cc647-fr9wh\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:39:52.143749 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.143636 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96417ce0-8b2f-49ad-8052-63687d8a46dc-oauth-serving-cert\") pod \"console-864c7cc647-fr9wh\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:39:52.143749 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.143694 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96417ce0-8b2f-49ad-8052-63687d8a46dc-trusted-ca-bundle\") pod \"console-864c7cc647-fr9wh\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:39:52.143897 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.143748 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96417ce0-8b2f-49ad-8052-63687d8a46dc-service-ca\") pod \"console-864c7cc647-fr9wh\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:39:52.143897 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.143777 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96417ce0-8b2f-49ad-8052-63687d8a46dc-console-serving-cert\") pod \"console-864c7cc647-fr9wh\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:39:52.244729 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.244690 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96417ce0-8b2f-49ad-8052-63687d8a46dc-trusted-ca-bundle\") pod \"console-864c7cc647-fr9wh\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:39:52.244729 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.244736 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96417ce0-8b2f-49ad-8052-63687d8a46dc-service-ca\") pod \"console-864c7cc647-fr9wh\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:39:52.244987 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.244763 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96417ce0-8b2f-49ad-8052-63687d8a46dc-console-serving-cert\") pod \"console-864c7cc647-fr9wh\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:39:52.244987 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.244797 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96417ce0-8b2f-49ad-8052-63687d8a46dc-console-oauth-config\") pod \"console-864c7cc647-fr9wh\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:39:52.244987 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.244818 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96417ce0-8b2f-49ad-8052-63687d8a46dc-console-config\") pod \"console-864c7cc647-fr9wh\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:39:52.244987 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.244840 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmtgw\" (UniqueName: \"kubernetes.io/projected/96417ce0-8b2f-49ad-8052-63687d8a46dc-kube-api-access-vmtgw\") pod \"console-864c7cc647-fr9wh\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:39:52.244987 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.244880 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96417ce0-8b2f-49ad-8052-63687d8a46dc-oauth-serving-cert\") pod \"console-864c7cc647-fr9wh\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:39:52.245654 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.245600 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96417ce0-8b2f-49ad-8052-63687d8a46dc-service-ca\") pod \"console-864c7cc647-fr9wh\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:39:52.245868 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.245839 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96417ce0-8b2f-49ad-8052-63687d8a46dc-oauth-serving-cert\") pod \"console-864c7cc647-fr9wh\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:39:52.246038 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.245980 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96417ce0-8b2f-49ad-8052-63687d8a46dc-trusted-ca-bundle\") pod \"console-864c7cc647-fr9wh\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:39:52.246038 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.245858 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96417ce0-8b2f-49ad-8052-63687d8a46dc-console-config\") pod \"console-864c7cc647-fr9wh\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:39:52.247729 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.247710 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96417ce0-8b2f-49ad-8052-63687d8a46dc-console-serving-cert\") pod \"console-864c7cc647-fr9wh\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:39:52.247909 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.247880 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96417ce0-8b2f-49ad-8052-63687d8a46dc-console-oauth-config\") pod \"console-864c7cc647-fr9wh\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:39:52.255472 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.255446 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmtgw\" (UniqueName: \"kubernetes.io/projected/96417ce0-8b2f-49ad-8052-63687d8a46dc-kube-api-access-vmtgw\") pod \"console-864c7cc647-fr9wh\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:39:52.401953 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.401861 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:39:52.606787 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:52.606753 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-864c7cc647-fr9wh"] Jul 06 16:39:52.610615 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:39:52.610581 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96417ce0_8b2f_49ad_8052_63687d8a46dc.slice/crio-f5240a0b7d759d29f92f730cb6e44e01465c4fab51bb4a7fc8919d5f6c692eba WatchSource:0}: Error finding container f5240a0b7d759d29f92f730cb6e44e01465c4fab51bb4a7fc8919d5f6c692eba: Status 404 returned error can't find the container with id f5240a0b7d759d29f92f730cb6e44e01465c4fab51bb4a7fc8919d5f6c692eba Jul 06 16:39:53.451492 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:53.451450 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-864c7cc647-fr9wh" event={"ID":"96417ce0-8b2f-49ad-8052-63687d8a46dc","Type":"ContainerStarted","Data":"f5240a0b7d759d29f92f730cb6e44e01465c4fab51bb4a7fc8919d5f6c692eba"} Jul 06 16:39:56.465549 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:56.465498 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-864c7cc647-fr9wh" event={"ID":"96417ce0-8b2f-49ad-8052-63687d8a46dc","Type":"ContainerStarted","Data":"a2fde0b3426f009d0f05a912defd570c972a37c63a9e4beeea7f305f8a9f3540"} Jul 06 16:39:56.559397 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:56.559339 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-864c7cc647-fr9wh" podStartSLOduration=1.117069143 podStartE2EDuration="4.559321513s" totalImagesPullingTime="3.44225237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:39:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:39:56.558932375 +0000 UTC m=+90.078423491" watchObservedRunningTime="2026-07-06 16:39:56.559321513 +0000 UTC m=+90.078812629" Jul 06 16:39:57.527617 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:57.527583 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-b2rtp_a9e75cc4-a9a1-4be1-b52b-0522e2f176d3/networking-console-plugin/0.log" Jul 06 16:39:58.285973 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:58.285950 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-864c7cc647-fr9wh_96417ce0-8b2f-49ad-8052-63687d8a46dc/console/0.log" Jul 06 16:39:58.490676 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:58.490644 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-nh4dr_832a849a-0ae6-4242-8ba1-f64b0f12038a/download-server/0.log" Jul 06 16:39:59.302062 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:39:59.302039 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pdh4h_5b350fbc-4cba-421e-89d3-1f0df3318c34/serve-healthcheck-canary/0.log" Jul 06 16:40:02.402886 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:02.402856 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:40:02.403356 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:02.402914 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:40:02.407581 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:02.407559 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:40:02.489251 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:02.489222 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:40:03.689860 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.689831 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68"] Jul 06 16:40:03.763379 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.763344 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68"] Jul 06 16:40:03.763538 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.763463 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68" Jul 06 16:40:03.767620 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.767591 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 16:40:03.767763 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.767650 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-lqs5v\"" Jul 06 16:40:03.767763 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.767669 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 06 16:40:03.767763 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.767697 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 06 16:40:03.768687 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.768671 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 16:40:03.768795 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.768691 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 16:40:03.768795 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.768754 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 06 16:40:03.813971 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.813940 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mgt5v"] Jul 06 16:40:03.839313 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.839282 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2743adb0-29e5-4b14-999b-dd6b06bbab78-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-lwx68\" (UID: \"2743adb0-29e5-4b14-999b-dd6b06bbab78\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68" Jul 06 16:40:03.839456 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.839316 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2743adb0-29e5-4b14-999b-dd6b06bbab78-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-lwx68\" (UID: \"2743adb0-29e5-4b14-999b-dd6b06bbab78\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68" Jul 06 16:40:03.839456 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.839353 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2743adb0-29e5-4b14-999b-dd6b06bbab78-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-lwx68\" (UID: \"2743adb0-29e5-4b14-999b-dd6b06bbab78\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68" Jul 06 16:40:03.839456 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.839397 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2743adb0-29e5-4b14-999b-dd6b06bbab78-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-lwx68\" (UID: \"2743adb0-29e5-4b14-999b-dd6b06bbab78\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68" Jul 06 16:40:03.839456 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.839420 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn24h\" (UniqueName: \"kubernetes.io/projected/2743adb0-29e5-4b14-999b-dd6b06bbab78-kube-api-access-rn24h\") pod \"kube-state-metrics-7c9dbb8b7d-lwx68\" (UID: \"2743adb0-29e5-4b14-999b-dd6b06bbab78\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68" Jul 06 16:40:03.839456 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.839444 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2743adb0-29e5-4b14-999b-dd6b06bbab78-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-lwx68\" (UID: \"2743adb0-29e5-4b14-999b-dd6b06bbab78\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68" Jul 06 16:40:03.842776 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.842756 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:03.888016 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.887984 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 16:40:03.888182 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.888087 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 16:40:03.888286 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.888267 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 16:40:03.888376 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.888359 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-64ncg\"" Jul 06 16:40:03.940531 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.940441 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ac30c81-a78b-44ab-aa37-711dcc8be401-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mgt5v\" (UID: \"9ac30c81-a78b-44ab-aa37-711dcc8be401\") " pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:03.940531 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.940479 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2743adb0-29e5-4b14-999b-dd6b06bbab78-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-lwx68\" (UID: \"2743adb0-29e5-4b14-999b-dd6b06bbab78\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68" Jul 06 16:40:03.940711 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.940511 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2743adb0-29e5-4b14-999b-dd6b06bbab78-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-lwx68\" (UID: \"2743adb0-29e5-4b14-999b-dd6b06bbab78\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68" Jul 06 16:40:03.940711 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.940559 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzlds\" (UniqueName: \"kubernetes.io/projected/9ac30c81-a78b-44ab-aa37-711dcc8be401-kube-api-access-nzlds\") pod \"node-exporter-mgt5v\" (UID: \"9ac30c81-a78b-44ab-aa37-711dcc8be401\") " pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:03.940711 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.940585 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ac30c81-a78b-44ab-aa37-711dcc8be401-metrics-client-ca\") pod \"node-exporter-mgt5v\" (UID: \"9ac30c81-a78b-44ab-aa37-711dcc8be401\") " pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:03.940711 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.940607 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9ac30c81-a78b-44ab-aa37-711dcc8be401-node-exporter-wtmp\") pod \"node-exporter-mgt5v\" (UID: \"9ac30c81-a78b-44ab-aa37-711dcc8be401\") " pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:03.940711 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.940628 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9ac30c81-a78b-44ab-aa37-711dcc8be401-root\") pod \"node-exporter-mgt5v\" (UID: \"9ac30c81-a78b-44ab-aa37-711dcc8be401\") " pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:03.940711 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.940659 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2743adb0-29e5-4b14-999b-dd6b06bbab78-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-lwx68\" (UID: \"2743adb0-29e5-4b14-999b-dd6b06bbab78\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68" Jul 06 16:40:03.940711 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.940680 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9ac30c81-a78b-44ab-aa37-711dcc8be401-node-exporter-tls\") pod \"node-exporter-mgt5v\" (UID: \"9ac30c81-a78b-44ab-aa37-711dcc8be401\") " pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:03.940711 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.940705 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2743adb0-29e5-4b14-999b-dd6b06bbab78-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-lwx68\" (UID: \"2743adb0-29e5-4b14-999b-dd6b06bbab78\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68" Jul 06 16:40:03.941029 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.940723 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ac30c81-a78b-44ab-aa37-711dcc8be401-sys\") pod \"node-exporter-mgt5v\" (UID: \"9ac30c81-a78b-44ab-aa37-711dcc8be401\") " pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:03.941029 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.940745 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9ac30c81-a78b-44ab-aa37-711dcc8be401-node-exporter-textfile\") pod \"node-exporter-mgt5v\" (UID: \"9ac30c81-a78b-44ab-aa37-711dcc8be401\") " pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:03.941029 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.940772 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9ac30c81-a78b-44ab-aa37-711dcc8be401-node-exporter-accelerators-collector-config\") pod \"node-exporter-mgt5v\" (UID: \"9ac30c81-a78b-44ab-aa37-711dcc8be401\") " pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:03.941029 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:40:03.940796 2574 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 06 16:40:03.941029 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.940805 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn24h\" (UniqueName: \"kubernetes.io/projected/2743adb0-29e5-4b14-999b-dd6b06bbab78-kube-api-access-rn24h\") pod \"kube-state-metrics-7c9dbb8b7d-lwx68\" (UID: \"2743adb0-29e5-4b14-999b-dd6b06bbab78\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68" Jul 06 16:40:03.941029 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.940833 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2743adb0-29e5-4b14-999b-dd6b06bbab78-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-lwx68\" (UID: \"2743adb0-29e5-4b14-999b-dd6b06bbab78\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68" Jul 06 16:40:03.941029 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:40:03.940862 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2743adb0-29e5-4b14-999b-dd6b06bbab78-kube-state-metrics-tls podName:2743adb0-29e5-4b14-999b-dd6b06bbab78 nodeName:}" failed. No retries permitted until 2026-07-06 16:40:04.440847396 +0000 UTC m=+97.960338489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/2743adb0-29e5-4b14-999b-dd6b06bbab78-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-lwx68" (UID: "2743adb0-29e5-4b14-999b-dd6b06bbab78") : secret "kube-state-metrics-tls" not found Jul 06 16:40:03.941310 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.941040 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2743adb0-29e5-4b14-999b-dd6b06bbab78-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-lwx68\" (UID: \"2743adb0-29e5-4b14-999b-dd6b06bbab78\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68" Jul 06 16:40:03.941476 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.941458 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2743adb0-29e5-4b14-999b-dd6b06bbab78-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-lwx68\" (UID: \"2743adb0-29e5-4b14-999b-dd6b06bbab78\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68" Jul 06 16:40:03.941511 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.941458 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2743adb0-29e5-4b14-999b-dd6b06bbab78-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-lwx68\" (UID: \"2743adb0-29e5-4b14-999b-dd6b06bbab78\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68" Jul 06 16:40:03.943073 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:03.943056 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2743adb0-29e5-4b14-999b-dd6b06bbab78-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-lwx68\" (UID: \"2743adb0-29e5-4b14-999b-dd6b06bbab78\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68" Jul 06 16:40:04.022603 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:04.022567 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn24h\" (UniqueName: \"kubernetes.io/projected/2743adb0-29e5-4b14-999b-dd6b06bbab78-kube-api-access-rn24h\") pod \"kube-state-metrics-7c9dbb8b7d-lwx68\" (UID: \"2743adb0-29e5-4b14-999b-dd6b06bbab78\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68" Jul 06 16:40:04.041687 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:04.041657 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ac30c81-a78b-44ab-aa37-711dcc8be401-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mgt5v\" (UID: \"9ac30c81-a78b-44ab-aa37-711dcc8be401\") " pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:04.041842 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:04.041690 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzlds\" (UniqueName: \"kubernetes.io/projected/9ac30c81-a78b-44ab-aa37-711dcc8be401-kube-api-access-nzlds\") pod \"node-exporter-mgt5v\" (UID: \"9ac30c81-a78b-44ab-aa37-711dcc8be401\") " pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:04.041842 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:04.041750 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ac30c81-a78b-44ab-aa37-711dcc8be401-metrics-client-ca\") pod \"node-exporter-mgt5v\" (UID: \"9ac30c81-a78b-44ab-aa37-711dcc8be401\") " pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:04.041842 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:04.041781 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9ac30c81-a78b-44ab-aa37-711dcc8be401-node-exporter-wtmp\") pod \"node-exporter-mgt5v\" (UID: \"9ac30c81-a78b-44ab-aa37-711dcc8be401\") " pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:04.041842 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:04.041813 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9ac30c81-a78b-44ab-aa37-711dcc8be401-root\") pod \"node-exporter-mgt5v\" (UID: \"9ac30c81-a78b-44ab-aa37-711dcc8be401\") " pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:04.042054 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:04.041843 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9ac30c81-a78b-44ab-aa37-711dcc8be401-node-exporter-tls\") pod \"node-exporter-mgt5v\" (UID: \"9ac30c81-a78b-44ab-aa37-711dcc8be401\") " pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:04.042054 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:04.041890 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ac30c81-a78b-44ab-aa37-711dcc8be401-sys\") pod \"node-exporter-mgt5v\" (UID: \"9ac30c81-a78b-44ab-aa37-711dcc8be401\") " pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:04.042054 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:04.041910 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9ac30c81-a78b-44ab-aa37-711dcc8be401-node-exporter-textfile\") pod \"node-exporter-mgt5v\" (UID: \"9ac30c81-a78b-44ab-aa37-711dcc8be401\") " pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:04.042054 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:04.041911 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9ac30c81-a78b-44ab-aa37-711dcc8be401-root\") pod \"node-exporter-mgt5v\" (UID: \"9ac30c81-a78b-44ab-aa37-711dcc8be401\") " pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:04.042054 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:04.041940 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9ac30c81-a78b-44ab-aa37-711dcc8be401-node-exporter-accelerators-collector-config\") pod \"node-exporter-mgt5v\" (UID: \"9ac30c81-a78b-44ab-aa37-711dcc8be401\") " pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:04.042054 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:04.041973 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ac30c81-a78b-44ab-aa37-711dcc8be401-sys\") pod \"node-exporter-mgt5v\" (UID: \"9ac30c81-a78b-44ab-aa37-711dcc8be401\") " pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:04.042054 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:04.041983 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9ac30c81-a78b-44ab-aa37-711dcc8be401-node-exporter-wtmp\") pod \"node-exporter-mgt5v\" (UID: \"9ac30c81-a78b-44ab-aa37-711dcc8be401\") " pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:04.042390 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:04.042248 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9ac30c81-a78b-44ab-aa37-711dcc8be401-node-exporter-textfile\") pod \"node-exporter-mgt5v\" (UID: \"9ac30c81-a78b-44ab-aa37-711dcc8be401\") " pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:04.042486 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:04.042464 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ac30c81-a78b-44ab-aa37-711dcc8be401-metrics-client-ca\") pod \"node-exporter-mgt5v\" (UID: \"9ac30c81-a78b-44ab-aa37-711dcc8be401\") " pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:04.042560 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:04.042483 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9ac30c81-a78b-44ab-aa37-711dcc8be401-node-exporter-accelerators-collector-config\") pod \"node-exporter-mgt5v\" (UID: \"9ac30c81-a78b-44ab-aa37-711dcc8be401\") " pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:04.043964 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:04.043944 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ac30c81-a78b-44ab-aa37-711dcc8be401-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mgt5v\" (UID: \"9ac30c81-a78b-44ab-aa37-711dcc8be401\") " pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:04.044073 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:04.044054 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9ac30c81-a78b-44ab-aa37-711dcc8be401-node-exporter-tls\") pod \"node-exporter-mgt5v\" (UID: \"9ac30c81-a78b-44ab-aa37-711dcc8be401\") " pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:04.079590 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:04.079558 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzlds\" (UniqueName: \"kubernetes.io/projected/9ac30c81-a78b-44ab-aa37-711dcc8be401-kube-api-access-nzlds\") pod \"node-exporter-mgt5v\" (UID: \"9ac30c81-a78b-44ab-aa37-711dcc8be401\") " pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:04.155863 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:04.155834 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mgt5v" Jul 06 16:40:04.189707 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:40:04.189677 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac30c81_a78b_44ab_aa37_711dcc8be401.slice/crio-bcdc2d603022c95dff6431009c05b8d04e73d7483045ca3c49696fc898c4d351 WatchSource:0}: Error finding container bcdc2d603022c95dff6431009c05b8d04e73d7483045ca3c49696fc898c4d351: Status 404 returned error can't find the container with id bcdc2d603022c95dff6431009c05b8d04e73d7483045ca3c49696fc898c4d351 Jul 06 16:40:04.445578 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:04.445547 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2743adb0-29e5-4b14-999b-dd6b06bbab78-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-lwx68\" (UID: \"2743adb0-29e5-4b14-999b-dd6b06bbab78\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68" Jul 06 16:40:04.447967 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:04.447940 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2743adb0-29e5-4b14-999b-dd6b06bbab78-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-lwx68\" (UID: \"2743adb0-29e5-4b14-999b-dd6b06bbab78\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68" Jul 06 16:40:04.493586 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:04.493546 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mgt5v" event={"ID":"9ac30c81-a78b-44ab-aa37-711dcc8be401","Type":"ContainerStarted","Data":"bcdc2d603022c95dff6431009c05b8d04e73d7483045ca3c49696fc898c4d351"} Jul 06 16:40:04.673207 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:04.673170 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68" Jul 06 16:40:04.854079 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:04.854043 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68"] Jul 06 16:40:04.857832 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:40:04.857772 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2743adb0_29e5_4b14_999b_dd6b06bbab78.slice/crio-130d5727d4be73999888717f07f02e94520a77b777d67a66a3537db5840b1029 WatchSource:0}: Error finding container 130d5727d4be73999888717f07f02e94520a77b777d67a66a3537db5840b1029: Status 404 returned error can't find the container with id 130d5727d4be73999888717f07f02e94520a77b777d67a66a3537db5840b1029 Jul 06 16:40:05.497953 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:05.497920 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68" event={"ID":"2743adb0-29e5-4b14-999b-dd6b06bbab78","Type":"ContainerStarted","Data":"130d5727d4be73999888717f07f02e94520a77b777d67a66a3537db5840b1029"} Jul 06 16:40:06.502228 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:06.502199 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mgt5v" event={"ID":"9ac30c81-a78b-44ab-aa37-711dcc8be401","Type":"ContainerStarted","Data":"0ce763c6f7cfea2d03e7f89dd696b43571be32858955454230027471a7ac586f"} Jul 06 16:40:07.508235 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:07.508194 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68" event={"ID":"2743adb0-29e5-4b14-999b-dd6b06bbab78","Type":"ContainerStarted","Data":"f8005def97aa677446eac1fde3fdeda083513b4c2875bcda23b2c6893a707a62"} Jul 06 16:40:07.508772 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:07.508304 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68" event={"ID":"2743adb0-29e5-4b14-999b-dd6b06bbab78","Type":"ContainerStarted","Data":"a85108982b25a1682bd5d0ada5a97e131ecddc19b90c71cca7aab72f32e26780"} Jul 06 16:40:07.510283 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:07.510251 2574 generic.go:356] "Generic (PLEG): container finished" podID="9ac30c81-a78b-44ab-aa37-711dcc8be401" containerID="0ce763c6f7cfea2d03e7f89dd696b43571be32858955454230027471a7ac586f" exitCode=0 Jul 06 16:40:07.510400 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:07.510332 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mgt5v" event={"ID":"9ac30c81-a78b-44ab-aa37-711dcc8be401","Type":"ContainerDied","Data":"0ce763c6f7cfea2d03e7f89dd696b43571be32858955454230027471a7ac586f"} Jul 06 16:40:08.230622 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.230588 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-ccfdd49f9-jrnjm"] Jul 06 16:40:08.245809 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.245780 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:40:08.288179 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.288148 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ccfdd49f9-jrnjm"] Jul 06 16:40:08.379289 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.379259 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd46e22-9117-4d84-8130-4053f6d9e0ab-service-ca\") pod \"console-ccfdd49f9-jrnjm\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:40:08.379465 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.379300 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dd46e22-9117-4d84-8130-4053f6d9e0ab-trusted-ca-bundle\") pod \"console-ccfdd49f9-jrnjm\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:40:08.379465 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.379320 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gntjh\" (UniqueName: \"kubernetes.io/projected/7dd46e22-9117-4d84-8130-4053f6d9e0ab-kube-api-access-gntjh\") pod \"console-ccfdd49f9-jrnjm\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:40:08.379465 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.379342 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7dd46e22-9117-4d84-8130-4053f6d9e0ab-console-config\") pod \"console-ccfdd49f9-jrnjm\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:40:08.379465 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.379437 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7dd46e22-9117-4d84-8130-4053f6d9e0ab-console-serving-cert\") pod \"console-ccfdd49f9-jrnjm\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:40:08.379633 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.379511 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7dd46e22-9117-4d84-8130-4053f6d9e0ab-oauth-serving-cert\") pod \"console-ccfdd49f9-jrnjm\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:40:08.379633 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.379563 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7dd46e22-9117-4d84-8130-4053f6d9e0ab-console-oauth-config\") pod \"console-ccfdd49f9-jrnjm\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:40:08.480533 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.480484 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7dd46e22-9117-4d84-8130-4053f6d9e0ab-console-serving-cert\") pod \"console-ccfdd49f9-jrnjm\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:40:08.480735 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.480578 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7dd46e22-9117-4d84-8130-4053f6d9e0ab-oauth-serving-cert\") pod \"console-ccfdd49f9-jrnjm\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:40:08.480735 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.480609 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7dd46e22-9117-4d84-8130-4053f6d9e0ab-console-oauth-config\") pod \"console-ccfdd49f9-jrnjm\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:40:08.480735 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.480636 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd46e22-9117-4d84-8130-4053f6d9e0ab-service-ca\") pod \"console-ccfdd49f9-jrnjm\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:40:08.480735 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.480684 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dd46e22-9117-4d84-8130-4053f6d9e0ab-trusted-ca-bundle\") pod \"console-ccfdd49f9-jrnjm\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:40:08.480735 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.480708 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gntjh\" (UniqueName: \"kubernetes.io/projected/7dd46e22-9117-4d84-8130-4053f6d9e0ab-kube-api-access-gntjh\") pod \"console-ccfdd49f9-jrnjm\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:40:08.480735 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.480734 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7dd46e22-9117-4d84-8130-4053f6d9e0ab-console-config\") pod \"console-ccfdd49f9-jrnjm\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:40:08.481450 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.481396 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd46e22-9117-4d84-8130-4053f6d9e0ab-service-ca\") pod \"console-ccfdd49f9-jrnjm\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:40:08.481450 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.481422 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7dd46e22-9117-4d84-8130-4053f6d9e0ab-console-config\") pod \"console-ccfdd49f9-jrnjm\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:40:08.481650 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.481576 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7dd46e22-9117-4d84-8130-4053f6d9e0ab-oauth-serving-cert\") pod \"console-ccfdd49f9-jrnjm\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:40:08.481887 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.481858 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dd46e22-9117-4d84-8130-4053f6d9e0ab-trusted-ca-bundle\") pod \"console-ccfdd49f9-jrnjm\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:40:08.483241 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.483218 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7dd46e22-9117-4d84-8130-4053f6d9e0ab-console-serving-cert\") pod \"console-ccfdd49f9-jrnjm\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:40:08.483954 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.483934 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7dd46e22-9117-4d84-8130-4053f6d9e0ab-console-oauth-config\") pod \"console-ccfdd49f9-jrnjm\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:40:08.519800 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.519762 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68" event={"ID":"2743adb0-29e5-4b14-999b-dd6b06bbab78","Type":"ContainerStarted","Data":"f6d22af601b0cedecb2f7052c8656c28e5b849ccf3dfc0ccb6ed6a3207132182"} Jul 06 16:40:08.521785 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.521741 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gntjh\" (UniqueName: \"kubernetes.io/projected/7dd46e22-9117-4d84-8130-4053f6d9e0ab-kube-api-access-gntjh\") pod \"console-ccfdd49f9-jrnjm\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:40:08.522427 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.522405 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mgt5v" event={"ID":"9ac30c81-a78b-44ab-aa37-711dcc8be401","Type":"ContainerStarted","Data":"583708e0a1fb506d89329db27478c86b292b05c2b7c6d05b89b2356a936efcb2"} Jul 06 16:40:08.522548 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.522454 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mgt5v" event={"ID":"9ac30c81-a78b-44ab-aa37-711dcc8be401","Type":"ContainerStarted","Data":"599d1b262024b4162bb5c913acf3fdbd903232d8d8eb129ce09d43d51044005c"} Jul 06 16:40:08.557891 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.557857 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:40:08.596855 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.596783 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-lwx68" podStartSLOduration=3.218828842 podStartE2EDuration="5.596759712s" totalImagesPullingTime="2.37793087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:40:08.591064056 +0000 UTC m=+102.110555172" watchObservedRunningTime="2026-07-06 16:40:08.596759712 +0000 UTC m=+102.116250836" Jul 06 16:40:08.839613 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.839446 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mgt5v" podStartSLOduration=3.704372932 podStartE2EDuration="5.839421172s" totalImagesPullingTime="2.024420711s" totalInitContainerRuntime="110.627529ms" isStatefulPod=true podCreationTimestamp="2026-07-06 16:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:40:08.679064493 +0000 UTC m=+102.198555609" watchObservedRunningTime="2026-07-06 16:40:08.839421172 +0000 UTC m=+102.358912287" Jul 06 16:40:08.840259 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:08.840235 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ccfdd49f9-jrnjm"] Jul 06 16:40:08.844090 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:40:08.844062 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd46e22_9117_4d84_8130_4053f6d9e0ab.slice/crio-04acf188d9252d5abee73e6581a7c06f1bb0a1ec04bb12f8e89a7af7e8f7a3f2 WatchSource:0}: Error finding container 04acf188d9252d5abee73e6581a7c06f1bb0a1ec04bb12f8e89a7af7e8f7a3f2: Status 404 returned error can't find the container with id 04acf188d9252d5abee73e6581a7c06f1bb0a1ec04bb12f8e89a7af7e8f7a3f2 Jul 06 16:40:09.528307 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:09.528270 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ccfdd49f9-jrnjm" event={"ID":"7dd46e22-9117-4d84-8130-4053f6d9e0ab","Type":"ContainerStarted","Data":"0241f0b214d2b04ddac56b94f0b94ea95433f37feaf338040751c6fd1633367e"} Jul 06 16:40:09.528840 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:09.528329 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ccfdd49f9-jrnjm" event={"ID":"7dd46e22-9117-4d84-8130-4053f6d9e0ab","Type":"ContainerStarted","Data":"04acf188d9252d5abee73e6581a7c06f1bb0a1ec04bb12f8e89a7af7e8f7a3f2"} Jul 06 16:40:10.392036 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:10.392008 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j5khq" Jul 06 16:40:10.436709 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:10.436656 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-ccfdd49f9-jrnjm" podStartSLOduration=2.436634277 podStartE2EDuration="2.436634277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:40:09.722049351 +0000 UTC m=+103.241540480" watchObservedRunningTime="2026-07-06 16:40:10.436634277 +0000 UTC m=+103.956125393" Jul 06 16:40:18.558328 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:18.558289 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:40:18.558888 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:18.558349 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:40:18.563413 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:18.563391 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:40:18.567043 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:18.567025 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:40:18.721642 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:18.721614 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-864c7cc647-fr9wh"] Jul 06 16:40:43.740913 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:43.740861 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-864c7cc647-fr9wh" podUID="96417ce0-8b2f-49ad-8052-63687d8a46dc" containerName="console" containerID="cri-o://a2fde0b3426f009d0f05a912defd570c972a37c63a9e4beeea7f305f8a9f3540" gracePeriod=15 Jul 06 16:40:43.997758 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:43.997682 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:40:44.056381 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.056352 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96417ce0-8b2f-49ad-8052-63687d8a46dc-service-ca\") pod \"96417ce0-8b2f-49ad-8052-63687d8a46dc\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " Jul 06 16:40:44.056573 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.056391 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96417ce0-8b2f-49ad-8052-63687d8a46dc-trusted-ca-bundle\") pod \"96417ce0-8b2f-49ad-8052-63687d8a46dc\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " Jul 06 16:40:44.056573 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.056424 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96417ce0-8b2f-49ad-8052-63687d8a46dc-console-oauth-config\") pod \"96417ce0-8b2f-49ad-8052-63687d8a46dc\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " Jul 06 16:40:44.056573 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.056445 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96417ce0-8b2f-49ad-8052-63687d8a46dc-oauth-serving-cert\") pod \"96417ce0-8b2f-49ad-8052-63687d8a46dc\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " Jul 06 16:40:44.056573 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.056504 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96417ce0-8b2f-49ad-8052-63687d8a46dc-console-serving-cert\") pod \"96417ce0-8b2f-49ad-8052-63687d8a46dc\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " Jul 06 16:40:44.056777 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.056656 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmtgw\" (UniqueName: \"kubernetes.io/projected/96417ce0-8b2f-49ad-8052-63687d8a46dc-kube-api-access-vmtgw\") pod \"96417ce0-8b2f-49ad-8052-63687d8a46dc\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " Jul 06 16:40:44.056777 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.056702 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96417ce0-8b2f-49ad-8052-63687d8a46dc-console-config\") pod \"96417ce0-8b2f-49ad-8052-63687d8a46dc\" (UID: \"96417ce0-8b2f-49ad-8052-63687d8a46dc\") " Jul 06 16:40:44.056939 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.056914 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96417ce0-8b2f-49ad-8052-63687d8a46dc-service-ca" (OuterVolumeSpecName: "service-ca") pod "96417ce0-8b2f-49ad-8052-63687d8a46dc" (UID: "96417ce0-8b2f-49ad-8052-63687d8a46dc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:40:44.057091 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.057024 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96417ce0-8b2f-49ad-8052-63687d8a46dc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "96417ce0-8b2f-49ad-8052-63687d8a46dc" (UID: "96417ce0-8b2f-49ad-8052-63687d8a46dc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:40:44.057223 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.057109 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96417ce0-8b2f-49ad-8052-63687d8a46dc-console-config" (OuterVolumeSpecName: "console-config") pod "96417ce0-8b2f-49ad-8052-63687d8a46dc" (UID: "96417ce0-8b2f-49ad-8052-63687d8a46dc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:40:44.057223 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.057195 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96417ce0-8b2f-49ad-8052-63687d8a46dc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "96417ce0-8b2f-49ad-8052-63687d8a46dc" (UID: "96417ce0-8b2f-49ad-8052-63687d8a46dc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:40:44.058761 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.058728 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96417ce0-8b2f-49ad-8052-63687d8a46dc-kube-api-access-vmtgw" (OuterVolumeSpecName: "kube-api-access-vmtgw") pod "96417ce0-8b2f-49ad-8052-63687d8a46dc" (UID: "96417ce0-8b2f-49ad-8052-63687d8a46dc"). InnerVolumeSpecName "kube-api-access-vmtgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:40:44.058851 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.058791 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96417ce0-8b2f-49ad-8052-63687d8a46dc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "96417ce0-8b2f-49ad-8052-63687d8a46dc" (UID: "96417ce0-8b2f-49ad-8052-63687d8a46dc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:40:44.058851 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.058807 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96417ce0-8b2f-49ad-8052-63687d8a46dc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "96417ce0-8b2f-49ad-8052-63687d8a46dc" (UID: "96417ce0-8b2f-49ad-8052-63687d8a46dc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:40:44.157495 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.157454 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96417ce0-8b2f-49ad-8052-63687d8a46dc-console-config\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:40:44.157495 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.157487 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96417ce0-8b2f-49ad-8052-63687d8a46dc-service-ca\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:40:44.157495 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.157496 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96417ce0-8b2f-49ad-8052-63687d8a46dc-trusted-ca-bundle\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:40:44.157747 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.157507 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96417ce0-8b2f-49ad-8052-63687d8a46dc-console-oauth-config\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:40:44.157747 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.157540 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96417ce0-8b2f-49ad-8052-63687d8a46dc-oauth-serving-cert\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:40:44.157747 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.157565 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96417ce0-8b2f-49ad-8052-63687d8a46dc-console-serving-cert\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:40:44.157747 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.157575 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmtgw\" (UniqueName: \"kubernetes.io/projected/96417ce0-8b2f-49ad-8052-63687d8a46dc-kube-api-access-vmtgw\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:40:44.651207 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.651170 2574 generic.go:356] "Generic (PLEG): container finished" podID="96417ce0-8b2f-49ad-8052-63687d8a46dc" containerID="a2fde0b3426f009d0f05a912defd570c972a37c63a9e4beeea7f305f8a9f3540" exitCode=0 Jul 06 16:40:44.651386 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.651215 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-864c7cc647-fr9wh" Jul 06 16:40:44.651386 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.651253 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-864c7cc647-fr9wh" event={"ID":"96417ce0-8b2f-49ad-8052-63687d8a46dc","Type":"ContainerDied","Data":"a2fde0b3426f009d0f05a912defd570c972a37c63a9e4beeea7f305f8a9f3540"} Jul 06 16:40:44.651386 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.651303 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-864c7cc647-fr9wh" event={"ID":"96417ce0-8b2f-49ad-8052-63687d8a46dc","Type":"ContainerDied","Data":"f5240a0b7d759d29f92f730cb6e44e01465c4fab51bb4a7fc8919d5f6c692eba"} Jul 06 16:40:44.651386 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.651324 2574 scope.go:117] "RemoveContainer" containerID="a2fde0b3426f009d0f05a912defd570c972a37c63a9e4beeea7f305f8a9f3540" Jul 06 16:40:44.660614 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.660597 2574 scope.go:117] "RemoveContainer" containerID="a2fde0b3426f009d0f05a912defd570c972a37c63a9e4beeea7f305f8a9f3540" Jul 06 16:40:44.660884 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:40:44.660861 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2fde0b3426f009d0f05a912defd570c972a37c63a9e4beeea7f305f8a9f3540\": container with ID starting with a2fde0b3426f009d0f05a912defd570c972a37c63a9e4beeea7f305f8a9f3540 not found: ID does not exist" containerID="a2fde0b3426f009d0f05a912defd570c972a37c63a9e4beeea7f305f8a9f3540" Jul 06 16:40:44.660929 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.660899 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2fde0b3426f009d0f05a912defd570c972a37c63a9e4beeea7f305f8a9f3540"} err="failed to get container status \"a2fde0b3426f009d0f05a912defd570c972a37c63a9e4beeea7f305f8a9f3540\": rpc error: code = NotFound desc = could not find container \"a2fde0b3426f009d0f05a912defd570c972a37c63a9e4beeea7f305f8a9f3540\": container with ID starting with a2fde0b3426f009d0f05a912defd570c972a37c63a9e4beeea7f305f8a9f3540 not found: ID does not exist" Jul 06 16:40:44.679392 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.679368 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-864c7cc647-fr9wh"] Jul 06 16:40:44.685427 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:44.685407 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-864c7cc647-fr9wh"] Jul 06 16:40:45.074944 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:40:45.074912 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96417ce0-8b2f-49ad-8052-63687d8a46dc" path="/var/lib/kubelet/pods/96417ce0-8b2f-49ad-8052-63687d8a46dc/volumes" Jul 06 16:43:26.882587 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:43:26.882555 2574 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 16:45:18.044060 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.044028 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7777b59744-fxzz4"] Jul 06 16:45:18.044650 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.044286 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96417ce0-8b2f-49ad-8052-63687d8a46dc" containerName="console" Jul 06 16:45:18.044650 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.044301 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="96417ce0-8b2f-49ad-8052-63687d8a46dc" containerName="console" Jul 06 16:45:18.044650 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.044350 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="96417ce0-8b2f-49ad-8052-63687d8a46dc" containerName="console" Jul 06 16:45:18.046254 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.046238 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:45:18.143926 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.143884 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgcbz\" (UniqueName: \"kubernetes.io/projected/8332929f-d1b1-449a-ae88-b3912af70e00-kube-api-access-qgcbz\") pod \"console-7777b59744-fxzz4\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:45:18.143926 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.143930 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8332929f-d1b1-449a-ae88-b3912af70e00-trusted-ca-bundle\") pod \"console-7777b59744-fxzz4\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:45:18.144147 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.143947 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8332929f-d1b1-449a-ae88-b3912af70e00-service-ca\") pod \"console-7777b59744-fxzz4\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:45:18.144147 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.144005 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8332929f-d1b1-449a-ae88-b3912af70e00-console-oauth-config\") pod \"console-7777b59744-fxzz4\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:45:18.144147 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.144061 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8332929f-d1b1-449a-ae88-b3912af70e00-console-serving-cert\") pod \"console-7777b59744-fxzz4\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:45:18.144147 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.144088 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8332929f-d1b1-449a-ae88-b3912af70e00-console-config\") pod \"console-7777b59744-fxzz4\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:45:18.144147 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.144113 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8332929f-d1b1-449a-ae88-b3912af70e00-oauth-serving-cert\") pod \"console-7777b59744-fxzz4\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:45:18.244668 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.244633 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8332929f-d1b1-449a-ae88-b3912af70e00-console-config\") pod \"console-7777b59744-fxzz4\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:45:18.244864 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.244680 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8332929f-d1b1-449a-ae88-b3912af70e00-oauth-serving-cert\") pod \"console-7777b59744-fxzz4\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:45:18.244864 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.244706 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgcbz\" (UniqueName: \"kubernetes.io/projected/8332929f-d1b1-449a-ae88-b3912af70e00-kube-api-access-qgcbz\") pod \"console-7777b59744-fxzz4\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:45:18.244864 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.244746 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8332929f-d1b1-449a-ae88-b3912af70e00-trusted-ca-bundle\") pod \"console-7777b59744-fxzz4\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:45:18.244864 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.244771 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8332929f-d1b1-449a-ae88-b3912af70e00-service-ca\") pod \"console-7777b59744-fxzz4\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:45:18.244864 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.244795 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8332929f-d1b1-449a-ae88-b3912af70e00-console-oauth-config\") pod \"console-7777b59744-fxzz4\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:45:18.244864 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.244842 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8332929f-d1b1-449a-ae88-b3912af70e00-console-serving-cert\") pod \"console-7777b59744-fxzz4\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:45:18.245506 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.245466 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8332929f-d1b1-449a-ae88-b3912af70e00-service-ca\") pod \"console-7777b59744-fxzz4\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:45:18.245630 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.245567 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8332929f-d1b1-449a-ae88-b3912af70e00-oauth-serving-cert\") pod \"console-7777b59744-fxzz4\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:45:18.245719 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.245697 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8332929f-d1b1-449a-ae88-b3912af70e00-console-config\") pod \"console-7777b59744-fxzz4\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:45:18.245763 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.245697 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8332929f-d1b1-449a-ae88-b3912af70e00-trusted-ca-bundle\") pod \"console-7777b59744-fxzz4\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:45:18.247381 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.247360 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8332929f-d1b1-449a-ae88-b3912af70e00-console-serving-cert\") pod \"console-7777b59744-fxzz4\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:45:18.247469 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.247358 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8332929f-d1b1-449a-ae88-b3912af70e00-console-oauth-config\") pod \"console-7777b59744-fxzz4\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:45:18.264639 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.264615 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7777b59744-fxzz4"] Jul 06 16:45:18.266659 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.266632 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgcbz\" (UniqueName: \"kubernetes.io/projected/8332929f-d1b1-449a-ae88-b3912af70e00-kube-api-access-qgcbz\") pod \"console-7777b59744-fxzz4\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:45:18.354390 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.354310 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:45:18.488873 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.488845 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7777b59744-fxzz4"] Jul 06 16:45:18.491924 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:45:18.491896 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8332929f_d1b1_449a_ae88_b3912af70e00.slice/crio-aa73e0476746bd9155db3d124c9436f1d8c046035cce2578cb728686b83ac314 WatchSource:0}: Error finding container aa73e0476746bd9155db3d124c9436f1d8c046035cce2578cb728686b83ac314: Status 404 returned error can't find the container with id aa73e0476746bd9155db3d124c9436f1d8c046035cce2578cb728686b83ac314 Jul 06 16:45:18.529009 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:18.528982 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7777b59744-fxzz4" event={"ID":"8332929f-d1b1-449a-ae88-b3912af70e00","Type":"ContainerStarted","Data":"aa73e0476746bd9155db3d124c9436f1d8c046035cce2578cb728686b83ac314"} Jul 06 16:45:19.533958 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:19.533919 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7777b59744-fxzz4" event={"ID":"8332929f-d1b1-449a-ae88-b3912af70e00","Type":"ContainerStarted","Data":"7a5620a1d5bcea7d9f40697266f1d538430c5950e8b2ed3e00aab37896a1a83b"} Jul 06 16:45:19.567500 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:19.567453 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7777b59744-fxzz4" podStartSLOduration=1.5674386550000001 podStartE2EDuration="1.567438655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:45:19.566648429 +0000 UTC m=+413.086139544" watchObservedRunningTime="2026-07-06 16:45:19.567438655 +0000 UTC m=+413.086929770" Jul 06 16:45:28.354967 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:28.354911 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:45:28.355363 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:28.355009 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:45:28.359610 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:28.359591 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:45:28.566225 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:28.566196 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:45:28.632210 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:28.632130 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ccfdd49f9-jrnjm"] Jul 06 16:45:53.654003 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:53.653960 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-ccfdd49f9-jrnjm" podUID="7dd46e22-9117-4d84-8130-4053f6d9e0ab" containerName="console" containerID="cri-o://0241f0b214d2b04ddac56b94f0b94ea95433f37feaf338040751c6fd1633367e" gracePeriod=15 Jul 06 16:45:53.890266 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:53.890243 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:45:54.014429 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.014398 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dd46e22-9117-4d84-8130-4053f6d9e0ab-trusted-ca-bundle\") pod \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " Jul 06 16:45:54.014617 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.014438 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gntjh\" (UniqueName: \"kubernetes.io/projected/7dd46e22-9117-4d84-8130-4053f6d9e0ab-kube-api-access-gntjh\") pod \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " Jul 06 16:45:54.014617 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.014457 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7dd46e22-9117-4d84-8130-4053f6d9e0ab-console-serving-cert\") pod \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " Jul 06 16:45:54.014617 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.014488 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd46e22-9117-4d84-8130-4053f6d9e0ab-service-ca\") pod \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " Jul 06 16:45:54.014617 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.014508 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7dd46e22-9117-4d84-8130-4053f6d9e0ab-console-config\") pod \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " Jul 06 16:45:54.014617 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.014547 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7dd46e22-9117-4d84-8130-4053f6d9e0ab-oauth-serving-cert\") pod \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " Jul 06 16:45:54.014617 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.014610 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7dd46e22-9117-4d84-8130-4053f6d9e0ab-console-oauth-config\") pod \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\" (UID: \"7dd46e22-9117-4d84-8130-4053f6d9e0ab\") " Jul 06 16:45:54.014890 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.014855 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd46e22-9117-4d84-8130-4053f6d9e0ab-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7dd46e22-9117-4d84-8130-4053f6d9e0ab" (UID: "7dd46e22-9117-4d84-8130-4053f6d9e0ab"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:45:54.015051 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.014982 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd46e22-9117-4d84-8130-4053f6d9e0ab-service-ca" (OuterVolumeSpecName: "service-ca") pod "7dd46e22-9117-4d84-8130-4053f6d9e0ab" (UID: "7dd46e22-9117-4d84-8130-4053f6d9e0ab"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:45:54.015127 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.015050 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd46e22-9117-4d84-8130-4053f6d9e0ab-console-config" (OuterVolumeSpecName: "console-config") pod "7dd46e22-9117-4d84-8130-4053f6d9e0ab" (UID: "7dd46e22-9117-4d84-8130-4053f6d9e0ab"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:45:54.015127 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.015086 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dd46e22-9117-4d84-8130-4053f6d9e0ab-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7dd46e22-9117-4d84-8130-4053f6d9e0ab" (UID: "7dd46e22-9117-4d84-8130-4053f6d9e0ab"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:45:54.016699 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.016665 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dd46e22-9117-4d84-8130-4053f6d9e0ab-kube-api-access-gntjh" (OuterVolumeSpecName: "kube-api-access-gntjh") pod "7dd46e22-9117-4d84-8130-4053f6d9e0ab" (UID: "7dd46e22-9117-4d84-8130-4053f6d9e0ab"). InnerVolumeSpecName "kube-api-access-gntjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:45:54.016806 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.016749 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dd46e22-9117-4d84-8130-4053f6d9e0ab-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7dd46e22-9117-4d84-8130-4053f6d9e0ab" (UID: "7dd46e22-9117-4d84-8130-4053f6d9e0ab"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:45:54.016806 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.016774 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dd46e22-9117-4d84-8130-4053f6d9e0ab-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7dd46e22-9117-4d84-8130-4053f6d9e0ab" (UID: "7dd46e22-9117-4d84-8130-4053f6d9e0ab"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:45:54.115391 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.115357 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7dd46e22-9117-4d84-8130-4053f6d9e0ab-oauth-serving-cert\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:45:54.115391 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.115385 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7dd46e22-9117-4d84-8130-4053f6d9e0ab-console-oauth-config\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:45:54.115391 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.115394 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7dd46e22-9117-4d84-8130-4053f6d9e0ab-trusted-ca-bundle\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:45:54.115631 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.115402 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gntjh\" (UniqueName: \"kubernetes.io/projected/7dd46e22-9117-4d84-8130-4053f6d9e0ab-kube-api-access-gntjh\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:45:54.115631 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.115411 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7dd46e22-9117-4d84-8130-4053f6d9e0ab-console-serving-cert\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:45:54.115631 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.115421 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7dd46e22-9117-4d84-8130-4053f6d9e0ab-service-ca\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:45:54.115631 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.115430 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7dd46e22-9117-4d84-8130-4053f6d9e0ab-console-config\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:45:54.648787 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.648753 2574 generic.go:356] "Generic (PLEG): container finished" podID="7dd46e22-9117-4d84-8130-4053f6d9e0ab" containerID="0241f0b214d2b04ddac56b94f0b94ea95433f37feaf338040751c6fd1633367e" exitCode=0 Jul 06 16:45:54.648954 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.648790 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ccfdd49f9-jrnjm" Jul 06 16:45:54.648954 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.648817 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ccfdd49f9-jrnjm" event={"ID":"7dd46e22-9117-4d84-8130-4053f6d9e0ab","Type":"ContainerDied","Data":"0241f0b214d2b04ddac56b94f0b94ea95433f37feaf338040751c6fd1633367e"} Jul 06 16:45:54.648954 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.648854 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ccfdd49f9-jrnjm" event={"ID":"7dd46e22-9117-4d84-8130-4053f6d9e0ab","Type":"ContainerDied","Data":"04acf188d9252d5abee73e6581a7c06f1bb0a1ec04bb12f8e89a7af7e8f7a3f2"} Jul 06 16:45:54.648954 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.648874 2574 scope.go:117] "RemoveContainer" containerID="0241f0b214d2b04ddac56b94f0b94ea95433f37feaf338040751c6fd1633367e" Jul 06 16:45:54.657467 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.657279 2574 scope.go:117] "RemoveContainer" containerID="0241f0b214d2b04ddac56b94f0b94ea95433f37feaf338040751c6fd1633367e" Jul 06 16:45:54.657754 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:45:54.657561 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0241f0b214d2b04ddac56b94f0b94ea95433f37feaf338040751c6fd1633367e\": container with ID starting with 0241f0b214d2b04ddac56b94f0b94ea95433f37feaf338040751c6fd1633367e not found: ID does not exist" containerID="0241f0b214d2b04ddac56b94f0b94ea95433f37feaf338040751c6fd1633367e" Jul 06 16:45:54.657754 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.657596 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0241f0b214d2b04ddac56b94f0b94ea95433f37feaf338040751c6fd1633367e"} err="failed to get container status \"0241f0b214d2b04ddac56b94f0b94ea95433f37feaf338040751c6fd1633367e\": rpc error: code = NotFound desc = could not find container \"0241f0b214d2b04ddac56b94f0b94ea95433f37feaf338040751c6fd1633367e\": container with ID starting with 0241f0b214d2b04ddac56b94f0b94ea95433f37feaf338040751c6fd1633367e not found: ID does not exist" Jul 06 16:45:54.670705 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.670684 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ccfdd49f9-jrnjm"] Jul 06 16:45:54.673969 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:54.673951 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-ccfdd49f9-jrnjm"] Jul 06 16:45:55.078592 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:45:55.078561 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dd46e22-9117-4d84-8130-4053f6d9e0ab" path="/var/lib/kubelet/pods/7dd46e22-9117-4d84-8130-4053f6d9e0ab/volumes" Jul 06 16:46:26.655778 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.655743 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-566f7ddb96-l989k"] Jul 06 16:46:26.656254 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.655976 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dd46e22-9117-4d84-8130-4053f6d9e0ab" containerName="console" Jul 06 16:46:26.656254 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.655986 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dd46e22-9117-4d84-8130-4053f6d9e0ab" containerName="console" Jul 06 16:46:26.656254 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.656037 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dd46e22-9117-4d84-8130-4053f6d9e0ab" containerName="console" Jul 06 16:46:26.658116 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.658100 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:46:26.749069 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.749026 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97dab90b-6dee-49c5-8fc2-ef1a6061931e-oauth-serving-cert\") pod \"console-566f7ddb96-l989k\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:46:26.749069 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.749068 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97dab90b-6dee-49c5-8fc2-ef1a6061931e-console-oauth-config\") pod \"console-566f7ddb96-l989k\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:46:26.749277 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.749090 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmslq\" (UniqueName: \"kubernetes.io/projected/97dab90b-6dee-49c5-8fc2-ef1a6061931e-kube-api-access-hmslq\") pod \"console-566f7ddb96-l989k\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:46:26.749277 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.749108 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97dab90b-6dee-49c5-8fc2-ef1a6061931e-trusted-ca-bundle\") pod \"console-566f7ddb96-l989k\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:46:26.749277 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.749163 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97dab90b-6dee-49c5-8fc2-ef1a6061931e-console-serving-cert\") pod \"console-566f7ddb96-l989k\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:46:26.749277 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.749195 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97dab90b-6dee-49c5-8fc2-ef1a6061931e-console-config\") pod \"console-566f7ddb96-l989k\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:46:26.749277 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.749219 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97dab90b-6dee-49c5-8fc2-ef1a6061931e-service-ca\") pod \"console-566f7ddb96-l989k\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:46:26.760360 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.760334 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-566f7ddb96-l989k"] Jul 06 16:46:26.850462 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.850417 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmslq\" (UniqueName: \"kubernetes.io/projected/97dab90b-6dee-49c5-8fc2-ef1a6061931e-kube-api-access-hmslq\") pod \"console-566f7ddb96-l989k\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:46:26.850462 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.850466 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97dab90b-6dee-49c5-8fc2-ef1a6061931e-trusted-ca-bundle\") pod \"console-566f7ddb96-l989k\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:46:26.850716 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.850489 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97dab90b-6dee-49c5-8fc2-ef1a6061931e-console-serving-cert\") pod \"console-566f7ddb96-l989k\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:46:26.850716 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.850505 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97dab90b-6dee-49c5-8fc2-ef1a6061931e-console-config\") pod \"console-566f7ddb96-l989k\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:46:26.850716 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.850538 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97dab90b-6dee-49c5-8fc2-ef1a6061931e-service-ca\") pod \"console-566f7ddb96-l989k\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:46:26.850716 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.850593 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97dab90b-6dee-49c5-8fc2-ef1a6061931e-oauth-serving-cert\") pod \"console-566f7ddb96-l989k\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:46:26.850716 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.850613 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97dab90b-6dee-49c5-8fc2-ef1a6061931e-console-oauth-config\") pod \"console-566f7ddb96-l989k\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:46:26.851364 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.851331 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97dab90b-6dee-49c5-8fc2-ef1a6061931e-console-config\") pod \"console-566f7ddb96-l989k\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:46:26.851509 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.851336 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97dab90b-6dee-49c5-8fc2-ef1a6061931e-oauth-serving-cert\") pod \"console-566f7ddb96-l989k\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:46:26.851509 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.851345 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97dab90b-6dee-49c5-8fc2-ef1a6061931e-service-ca\") pod \"console-566f7ddb96-l989k\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:46:26.851509 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.851397 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97dab90b-6dee-49c5-8fc2-ef1a6061931e-trusted-ca-bundle\") pod \"console-566f7ddb96-l989k\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:46:26.853086 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.853054 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97dab90b-6dee-49c5-8fc2-ef1a6061931e-console-serving-cert\") pod \"console-566f7ddb96-l989k\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:46:26.853086 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.853076 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97dab90b-6dee-49c5-8fc2-ef1a6061931e-console-oauth-config\") pod \"console-566f7ddb96-l989k\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:46:26.869161 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.869138 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmslq\" (UniqueName: \"kubernetes.io/projected/97dab90b-6dee-49c5-8fc2-ef1a6061931e-kube-api-access-hmslq\") pod \"console-566f7ddb96-l989k\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:46:26.966639 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:26.966606 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:46:27.106733 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:27.106703 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-566f7ddb96-l989k"] Jul 06 16:46:27.110110 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:46:27.110086 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97dab90b_6dee_49c5_8fc2_ef1a6061931e.slice/crio-c974d66089d778290db5eee1039a89e386fbac6a76cda9ff51a69ff95641bc06 WatchSource:0}: Error finding container c974d66089d778290db5eee1039a89e386fbac6a76cda9ff51a69ff95641bc06: Status 404 returned error can't find the container with id c974d66089d778290db5eee1039a89e386fbac6a76cda9ff51a69ff95641bc06 Jul 06 16:46:27.760269 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:27.760224 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-566f7ddb96-l989k" event={"ID":"97dab90b-6dee-49c5-8fc2-ef1a6061931e","Type":"ContainerStarted","Data":"4aa73770735319b227346ce44cdcdf104e148eafb85ae1cf0e623c27b86d6d31"} Jul 06 16:46:27.760670 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:27.760283 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-566f7ddb96-l989k" event={"ID":"97dab90b-6dee-49c5-8fc2-ef1a6061931e","Type":"ContainerStarted","Data":"c974d66089d778290db5eee1039a89e386fbac6a76cda9ff51a69ff95641bc06"} Jul 06 16:46:27.790689 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:27.790644 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-566f7ddb96-l989k" podStartSLOduration=1.790628485 podStartE2EDuration="1.790628485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:46:27.78975573 +0000 UTC m=+481.309246848" watchObservedRunningTime="2026-07-06 16:46:27.790628485 +0000 UTC m=+481.310119601" Jul 06 16:46:36.967527 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:36.967466 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:46:36.967972 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:36.967572 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:46:36.972396 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:36.972373 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:46:37.804401 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:37.804370 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:46:37.867875 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:46:37.867843 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7777b59744-fxzz4"] Jul 06 16:47:02.888915 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:02.888877 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7777b59744-fxzz4" podUID="8332929f-d1b1-449a-ae88-b3912af70e00" containerName="console" containerID="cri-o://7a5620a1d5bcea7d9f40697266f1d538430c5950e8b2ed3e00aab37896a1a83b" gracePeriod=15 Jul 06 16:47:03.135694 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.135672 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:47:03.219435 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.219405 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8332929f-d1b1-449a-ae88-b3912af70e00-service-ca\") pod \"8332929f-d1b1-449a-ae88-b3912af70e00\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " Jul 06 16:47:03.219603 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.219446 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8332929f-d1b1-449a-ae88-b3912af70e00-console-oauth-config\") pod \"8332929f-d1b1-449a-ae88-b3912af70e00\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " Jul 06 16:47:03.219603 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.219474 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgcbz\" (UniqueName: \"kubernetes.io/projected/8332929f-d1b1-449a-ae88-b3912af70e00-kube-api-access-qgcbz\") pod \"8332929f-d1b1-449a-ae88-b3912af70e00\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " Jul 06 16:47:03.219603 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.219502 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8332929f-d1b1-449a-ae88-b3912af70e00-oauth-serving-cert\") pod \"8332929f-d1b1-449a-ae88-b3912af70e00\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " Jul 06 16:47:03.219603 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.219560 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8332929f-d1b1-449a-ae88-b3912af70e00-console-serving-cert\") pod \"8332929f-d1b1-449a-ae88-b3912af70e00\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " Jul 06 16:47:03.219800 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.219646 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8332929f-d1b1-449a-ae88-b3912af70e00-console-config\") pod \"8332929f-d1b1-449a-ae88-b3912af70e00\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " Jul 06 16:47:03.219800 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.219710 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8332929f-d1b1-449a-ae88-b3912af70e00-trusted-ca-bundle\") pod \"8332929f-d1b1-449a-ae88-b3912af70e00\" (UID: \"8332929f-d1b1-449a-ae88-b3912af70e00\") " Jul 06 16:47:03.219903 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.219882 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8332929f-d1b1-449a-ae88-b3912af70e00-service-ca" (OuterVolumeSpecName: "service-ca") pod "8332929f-d1b1-449a-ae88-b3912af70e00" (UID: "8332929f-d1b1-449a-ae88-b3912af70e00"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:47:03.220487 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.220365 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8332929f-d1b1-449a-ae88-b3912af70e00-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8332929f-d1b1-449a-ae88-b3912af70e00" (UID: "8332929f-d1b1-449a-ae88-b3912af70e00"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:47:03.220487 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.220445 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8332929f-d1b1-449a-ae88-b3912af70e00-console-config" (OuterVolumeSpecName: "console-config") pod "8332929f-d1b1-449a-ae88-b3912af70e00" (UID: "8332929f-d1b1-449a-ae88-b3912af70e00"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:47:03.221739 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.220491 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8332929f-d1b1-449a-ae88-b3912af70e00-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8332929f-d1b1-449a-ae88-b3912af70e00" (UID: "8332929f-d1b1-449a-ae88-b3912af70e00"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:47:03.222622 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.222583 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8332929f-d1b1-449a-ae88-b3912af70e00-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8332929f-d1b1-449a-ae88-b3912af70e00" (UID: "8332929f-d1b1-449a-ae88-b3912af70e00"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:47:03.226154 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.226127 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8332929f-d1b1-449a-ae88-b3912af70e00-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8332929f-d1b1-449a-ae88-b3912af70e00" (UID: "8332929f-d1b1-449a-ae88-b3912af70e00"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:47:03.226154 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.226137 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8332929f-d1b1-449a-ae88-b3912af70e00-kube-api-access-qgcbz" (OuterVolumeSpecName: "kube-api-access-qgcbz") pod "8332929f-d1b1-449a-ae88-b3912af70e00" (UID: "8332929f-d1b1-449a-ae88-b3912af70e00"). InnerVolumeSpecName "kube-api-access-qgcbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:47:03.320621 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.320591 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8332929f-d1b1-449a-ae88-b3912af70e00-service-ca\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:47:03.320621 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.320617 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8332929f-d1b1-449a-ae88-b3912af70e00-console-oauth-config\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:47:03.320621 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.320628 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgcbz\" (UniqueName: \"kubernetes.io/projected/8332929f-d1b1-449a-ae88-b3912af70e00-kube-api-access-qgcbz\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:47:03.320824 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.320637 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8332929f-d1b1-449a-ae88-b3912af70e00-oauth-serving-cert\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:47:03.320824 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.320646 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8332929f-d1b1-449a-ae88-b3912af70e00-console-serving-cert\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:47:03.320824 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.320655 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8332929f-d1b1-449a-ae88-b3912af70e00-console-config\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:47:03.320824 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.320663 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8332929f-d1b1-449a-ae88-b3912af70e00-trusted-ca-bundle\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:47:03.891299 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.891264 2574 generic.go:356] "Generic (PLEG): container finished" podID="8332929f-d1b1-449a-ae88-b3912af70e00" containerID="7a5620a1d5bcea7d9f40697266f1d538430c5950e8b2ed3e00aab37896a1a83b" exitCode=0 Jul 06 16:47:03.891709 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.891323 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7777b59744-fxzz4" event={"ID":"8332929f-d1b1-449a-ae88-b3912af70e00","Type":"ContainerDied","Data":"7a5620a1d5bcea7d9f40697266f1d538430c5950e8b2ed3e00aab37896a1a83b"} Jul 06 16:47:03.891709 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.891324 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7777b59744-fxzz4" Jul 06 16:47:03.891709 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.891392 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7777b59744-fxzz4" event={"ID":"8332929f-d1b1-449a-ae88-b3912af70e00","Type":"ContainerDied","Data":"aa73e0476746bd9155db3d124c9436f1d8c046035cce2578cb728686b83ac314"} Jul 06 16:47:03.891709 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.891411 2574 scope.go:117] "RemoveContainer" containerID="7a5620a1d5bcea7d9f40697266f1d538430c5950e8b2ed3e00aab37896a1a83b" Jul 06 16:47:03.900160 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.900140 2574 scope.go:117] "RemoveContainer" containerID="7a5620a1d5bcea7d9f40697266f1d538430c5950e8b2ed3e00aab37896a1a83b" Jul 06 16:47:03.900419 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:47:03.900385 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a5620a1d5bcea7d9f40697266f1d538430c5950e8b2ed3e00aab37896a1a83b\": container with ID starting with 7a5620a1d5bcea7d9f40697266f1d538430c5950e8b2ed3e00aab37896a1a83b not found: ID does not exist" containerID="7a5620a1d5bcea7d9f40697266f1d538430c5950e8b2ed3e00aab37896a1a83b" Jul 06 16:47:03.900533 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.900415 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a5620a1d5bcea7d9f40697266f1d538430c5950e8b2ed3e00aab37896a1a83b"} err="failed to get container status \"7a5620a1d5bcea7d9f40697266f1d538430c5950e8b2ed3e00aab37896a1a83b\": rpc error: code = NotFound desc = could not find container \"7a5620a1d5bcea7d9f40697266f1d538430c5950e8b2ed3e00aab37896a1a83b\": container with ID starting with 7a5620a1d5bcea7d9f40697266f1d538430c5950e8b2ed3e00aab37896a1a83b not found: ID does not exist" Jul 06 16:47:03.919868 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.919844 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7777b59744-fxzz4"] Jul 06 16:47:03.922418 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:03.922399 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7777b59744-fxzz4"] Jul 06 16:47:05.074346 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:47:05.074316 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8332929f-d1b1-449a-ae88-b3912af70e00" path="/var/lib/kubelet/pods/8332929f-d1b1-449a-ae88-b3912af70e00/volumes" Jul 06 16:48:24.696352 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:24.696314 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-45hww"] Jul 06 16:48:24.696794 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:24.696592 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8332929f-d1b1-449a-ae88-b3912af70e00" containerName="console" Jul 06 16:48:24.696794 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:24.696605 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8332929f-d1b1-449a-ae88-b3912af70e00" containerName="console" Jul 06 16:48:24.696794 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:24.696650 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8332929f-d1b1-449a-ae88-b3912af70e00" containerName="console" Jul 06 16:48:24.699265 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:24.699249 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-45hww" Jul 06 16:48:24.704742 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:24.704718 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 06 16:48:24.705802 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:24.705782 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-smtlt\"" Jul 06 16:48:24.705802 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:24.705794 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 06 16:48:24.712914 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:24.712895 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-45hww"] Jul 06 16:48:24.882130 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:24.882095 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhw7g\" (UniqueName: \"kubernetes.io/projected/4f635839-d270-44b7-a7e5-7b1d68681133-kube-api-access-zhw7g\") pod \"cert-manager-webhook-6486ffd997-45hww\" (UID: \"4f635839-d270-44b7-a7e5-7b1d68681133\") " pod="cert-manager/cert-manager-webhook-6486ffd997-45hww" Jul 06 16:48:24.882294 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:24.882135 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4f635839-d270-44b7-a7e5-7b1d68681133-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-45hww\" (UID: \"4f635839-d270-44b7-a7e5-7b1d68681133\") " pod="cert-manager/cert-manager-webhook-6486ffd997-45hww" Jul 06 16:48:24.983431 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:24.983348 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhw7g\" (UniqueName: \"kubernetes.io/projected/4f635839-d270-44b7-a7e5-7b1d68681133-kube-api-access-zhw7g\") pod \"cert-manager-webhook-6486ffd997-45hww\" (UID: \"4f635839-d270-44b7-a7e5-7b1d68681133\") " pod="cert-manager/cert-manager-webhook-6486ffd997-45hww" Jul 06 16:48:24.983431 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:24.983403 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4f635839-d270-44b7-a7e5-7b1d68681133-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-45hww\" (UID: \"4f635839-d270-44b7-a7e5-7b1d68681133\") " pod="cert-manager/cert-manager-webhook-6486ffd997-45hww" Jul 06 16:48:24.999982 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:24.999950 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4f635839-d270-44b7-a7e5-7b1d68681133-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-45hww\" (UID: \"4f635839-d270-44b7-a7e5-7b1d68681133\") " pod="cert-manager/cert-manager-webhook-6486ffd997-45hww" Jul 06 16:48:25.008948 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:25.008925 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhw7g\" (UniqueName: \"kubernetes.io/projected/4f635839-d270-44b7-a7e5-7b1d68681133-kube-api-access-zhw7g\") pod \"cert-manager-webhook-6486ffd997-45hww\" (UID: \"4f635839-d270-44b7-a7e5-7b1d68681133\") " pod="cert-manager/cert-manager-webhook-6486ffd997-45hww" Jul 06 16:48:25.307607 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:25.307503 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-45hww" Jul 06 16:48:25.448440 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:25.448411 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-45hww"] Jul 06 16:48:25.451603 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:48:25.451577 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f635839_d270_44b7_a7e5_7b1d68681133.slice/crio-1879f7ea806fe789651624ae19cace1e39d75b95d1141136feb63a5f51a5efcf WatchSource:0}: Error finding container 1879f7ea806fe789651624ae19cace1e39d75b95d1141136feb63a5f51a5efcf: Status 404 returned error can't find the container with id 1879f7ea806fe789651624ae19cace1e39d75b95d1141136feb63a5f51a5efcf Jul 06 16:48:25.453614 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:25.453598 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 16:48:26.159460 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:26.159415 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-45hww" event={"ID":"4f635839-d270-44b7-a7e5-7b1d68681133","Type":"ContainerStarted","Data":"1879f7ea806fe789651624ae19cace1e39d75b95d1141136feb63a5f51a5efcf"} Jul 06 16:48:28.136072 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:28.136034 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-p86j4"] Jul 06 16:48:28.139584 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:28.139561 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-p86j4" Jul 06 16:48:28.142442 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:28.142419 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-v5j9q\"" Jul 06 16:48:28.151437 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:28.151384 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-p86j4"] Jul 06 16:48:28.312887 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:28.312845 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bvtk\" (UniqueName: \"kubernetes.io/projected/c3a221c9-ccea-4ff7-8fc8-a768fe1c30a1-kube-api-access-7bvtk\") pod \"cert-manager-55b4d954d5-p86j4\" (UID: \"c3a221c9-ccea-4ff7-8fc8-a768fe1c30a1\") " pod="cert-manager/cert-manager-55b4d954d5-p86j4" Jul 06 16:48:28.313046 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:28.312965 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3a221c9-ccea-4ff7-8fc8-a768fe1c30a1-bound-sa-token\") pod \"cert-manager-55b4d954d5-p86j4\" (UID: \"c3a221c9-ccea-4ff7-8fc8-a768fe1c30a1\") " pod="cert-manager/cert-manager-55b4d954d5-p86j4" Jul 06 16:48:28.414454 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:28.414351 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bvtk\" (UniqueName: \"kubernetes.io/projected/c3a221c9-ccea-4ff7-8fc8-a768fe1c30a1-kube-api-access-7bvtk\") pod \"cert-manager-55b4d954d5-p86j4\" (UID: \"c3a221c9-ccea-4ff7-8fc8-a768fe1c30a1\") " pod="cert-manager/cert-manager-55b4d954d5-p86j4" Jul 06 16:48:28.414454 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:28.414407 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3a221c9-ccea-4ff7-8fc8-a768fe1c30a1-bound-sa-token\") pod \"cert-manager-55b4d954d5-p86j4\" (UID: \"c3a221c9-ccea-4ff7-8fc8-a768fe1c30a1\") " pod="cert-manager/cert-manager-55b4d954d5-p86j4" Jul 06 16:48:28.425564 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:28.425511 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bvtk\" (UniqueName: \"kubernetes.io/projected/c3a221c9-ccea-4ff7-8fc8-a768fe1c30a1-kube-api-access-7bvtk\") pod \"cert-manager-55b4d954d5-p86j4\" (UID: \"c3a221c9-ccea-4ff7-8fc8-a768fe1c30a1\") " pod="cert-manager/cert-manager-55b4d954d5-p86j4" Jul 06 16:48:28.428773 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:28.428734 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c3a221c9-ccea-4ff7-8fc8-a768fe1c30a1-bound-sa-token\") pod \"cert-manager-55b4d954d5-p86j4\" (UID: \"c3a221c9-ccea-4ff7-8fc8-a768fe1c30a1\") " pod="cert-manager/cert-manager-55b4d954d5-p86j4" Jul 06 16:48:28.449644 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:28.449607 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-p86j4" Jul 06 16:48:28.605481 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:28.605441 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-p86j4"] Jul 06 16:48:28.608438 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:48:28.608412 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3a221c9_ccea_4ff7_8fc8_a768fe1c30a1.slice/crio-d9ee23f8d184087e24ac6460882a66e9a5b138a0954e999c325b53ed8aec45ff WatchSource:0}: Error finding container d9ee23f8d184087e24ac6460882a66e9a5b138a0954e999c325b53ed8aec45ff: Status 404 returned error can't find the container with id d9ee23f8d184087e24ac6460882a66e9a5b138a0954e999c325b53ed8aec45ff Jul 06 16:48:29.171467 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:29.171434 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-45hww" event={"ID":"4f635839-d270-44b7-a7e5-7b1d68681133","Type":"ContainerStarted","Data":"aef375065292d658ac8800c9e0df4aa1b3db8620274aeb390ea3a6e452b90e3e"} Jul 06 16:48:29.171896 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:29.171527 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-45hww" Jul 06 16:48:29.172873 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:29.172849 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-p86j4" event={"ID":"c3a221c9-ccea-4ff7-8fc8-a768fe1c30a1","Type":"ContainerStarted","Data":"5653c5af07ba9e0285a6a708f7d204b9f7aba07fb7ea9158443f10625982f8ee"} Jul 06 16:48:29.172873 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:29.172877 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-p86j4" event={"ID":"c3a221c9-ccea-4ff7-8fc8-a768fe1c30a1","Type":"ContainerStarted","Data":"d9ee23f8d184087e24ac6460882a66e9a5b138a0954e999c325b53ed8aec45ff"} Jul 06 16:48:29.193450 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:29.193405 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-45hww" podStartSLOduration=2.419793623 podStartE2EDuration="5.193391615s" totalImagesPullingTime="2.773597992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:48:29.192886762 +0000 UTC m=+602.712377878" watchObservedRunningTime="2026-07-06 16:48:29.193391615 +0000 UTC m=+602.712882730" Jul 06 16:48:29.216386 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:29.216337 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-p86j4" podStartSLOduration=1.216322501 podStartE2EDuration="1.216322501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:48:29.216301656 +0000 UTC m=+602.735792772" watchObservedRunningTime="2026-07-06 16:48:29.216322501 +0000 UTC m=+602.735813618" Jul 06 16:48:35.177984 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:35.177953 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-45hww" Jul 06 16:48:51.729956 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:51.729908 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d66bcd77d-vfkkk"] Jul 06 16:48:51.733129 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:51.733114 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-vfkkk" Jul 06 16:48:51.737683 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:51.737657 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 16:48:51.737683 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:51.737662 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 16:48:51.737855 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:51.737703 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 16:48:51.738476 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:51.738458 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-kwj6w\"" Jul 06 16:48:51.740217 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:51.740202 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 16:48:51.763275 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:51.763251 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d66bcd77d-vfkkk"] Jul 06 16:48:51.782103 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:51.782073 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39214b4b-e5c5-499c-ba7b-44b81db5a24a-webhook-cert\") pod \"opendatahub-operator-controller-manager-d66bcd77d-vfkkk\" (UID: \"39214b4b-e5c5-499c-ba7b-44b81db5a24a\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-vfkkk" Jul 06 16:48:51.782227 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:51.782134 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39214b4b-e5c5-499c-ba7b-44b81db5a24a-tmp\") pod \"opendatahub-operator-controller-manager-d66bcd77d-vfkkk\" (UID: \"39214b4b-e5c5-499c-ba7b-44b81db5a24a\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-vfkkk" Jul 06 16:48:51.782227 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:51.782179 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/39214b4b-e5c5-499c-ba7b-44b81db5a24a-manifests\") pod \"opendatahub-operator-controller-manager-d66bcd77d-vfkkk\" (UID: \"39214b4b-e5c5-499c-ba7b-44b81db5a24a\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-vfkkk" Jul 06 16:48:51.782336 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:51.782256 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/39214b4b-e5c5-499c-ba7b-44b81db5a24a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d66bcd77d-vfkkk\" (UID: \"39214b4b-e5c5-499c-ba7b-44b81db5a24a\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-vfkkk" Jul 06 16:48:51.782336 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:51.782299 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt24x\" (UniqueName: \"kubernetes.io/projected/39214b4b-e5c5-499c-ba7b-44b81db5a24a-kube-api-access-gt24x\") pod \"opendatahub-operator-controller-manager-d66bcd77d-vfkkk\" (UID: \"39214b4b-e5c5-499c-ba7b-44b81db5a24a\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-vfkkk" Jul 06 16:48:51.883293 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:51.883258 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39214b4b-e5c5-499c-ba7b-44b81db5a24a-tmp\") pod \"opendatahub-operator-controller-manager-d66bcd77d-vfkkk\" (UID: \"39214b4b-e5c5-499c-ba7b-44b81db5a24a\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-vfkkk" Jul 06 16:48:51.883293 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:51.883294 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/39214b4b-e5c5-499c-ba7b-44b81db5a24a-manifests\") pod \"opendatahub-operator-controller-manager-d66bcd77d-vfkkk\" (UID: \"39214b4b-e5c5-499c-ba7b-44b81db5a24a\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-vfkkk" Jul 06 16:48:51.883512 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:51.883316 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/39214b4b-e5c5-499c-ba7b-44b81db5a24a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d66bcd77d-vfkkk\" (UID: \"39214b4b-e5c5-499c-ba7b-44b81db5a24a\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-vfkkk" Jul 06 16:48:51.883512 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:51.883335 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt24x\" (UniqueName: \"kubernetes.io/projected/39214b4b-e5c5-499c-ba7b-44b81db5a24a-kube-api-access-gt24x\") pod \"opendatahub-operator-controller-manager-d66bcd77d-vfkkk\" (UID: \"39214b4b-e5c5-499c-ba7b-44b81db5a24a\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-vfkkk" Jul 06 16:48:51.883512 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:51.883454 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39214b4b-e5c5-499c-ba7b-44b81db5a24a-webhook-cert\") pod \"opendatahub-operator-controller-manager-d66bcd77d-vfkkk\" (UID: \"39214b4b-e5c5-499c-ba7b-44b81db5a24a\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-vfkkk" Jul 06 16:48:51.883792 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:51.883764 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/39214b4b-e5c5-499c-ba7b-44b81db5a24a-manifests\") pod \"opendatahub-operator-controller-manager-d66bcd77d-vfkkk\" (UID: \"39214b4b-e5c5-499c-ba7b-44b81db5a24a\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-vfkkk" Jul 06 16:48:51.885723 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:51.885703 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/39214b4b-e5c5-499c-ba7b-44b81db5a24a-tmp\") pod \"opendatahub-operator-controller-manager-d66bcd77d-vfkkk\" (UID: \"39214b4b-e5c5-499c-ba7b-44b81db5a24a\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-vfkkk" Jul 06 16:48:51.885908 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:51.885889 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39214b4b-e5c5-499c-ba7b-44b81db5a24a-webhook-cert\") pod \"opendatahub-operator-controller-manager-d66bcd77d-vfkkk\" (UID: \"39214b4b-e5c5-499c-ba7b-44b81db5a24a\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-vfkkk" Jul 06 16:48:51.885964 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:51.885913 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/39214b4b-e5c5-499c-ba7b-44b81db5a24a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d66bcd77d-vfkkk\" (UID: \"39214b4b-e5c5-499c-ba7b-44b81db5a24a\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-vfkkk" Jul 06 16:48:51.891851 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:51.891827 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt24x\" (UniqueName: \"kubernetes.io/projected/39214b4b-e5c5-499c-ba7b-44b81db5a24a-kube-api-access-gt24x\") pod \"opendatahub-operator-controller-manager-d66bcd77d-vfkkk\" (UID: \"39214b4b-e5c5-499c-ba7b-44b81db5a24a\") " pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-vfkkk" Jul 06 16:48:52.043204 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:52.043101 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-vfkkk" Jul 06 16:48:52.193107 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:52.192962 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d66bcd77d-vfkkk"] Jul 06 16:48:52.196125 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:48:52.196104 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39214b4b_e5c5_499c_ba7b_44b81db5a24a.slice/crio-bdc201e73c1f4e9c17831267a397b49b6186ffd02c442991d3e67cfc25d969fe WatchSource:0}: Error finding container bdc201e73c1f4e9c17831267a397b49b6186ffd02c442991d3e67cfc25d969fe: Status 404 returned error can't find the container with id bdc201e73c1f4e9c17831267a397b49b6186ffd02c442991d3e67cfc25d969fe Jul 06 16:48:52.255833 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:52.255806 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-vfkkk" event={"ID":"39214b4b-e5c5-499c-ba7b-44b81db5a24a","Type":"ContainerStarted","Data":"bdc201e73c1f4e9c17831267a397b49b6186ffd02c442991d3e67cfc25d969fe"} Jul 06 16:48:56.270533 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:56.270492 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-vfkkk" event={"ID":"39214b4b-e5c5-499c-ba7b-44b81db5a24a","Type":"ContainerStarted","Data":"e19bddd9e4e2c224e9dc21c9d443c2163a06d51f84d7e2ccd0e8376615a97bb1"} Jul 06 16:48:57.277459 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:57.277425 2574 generic.go:356] "Generic (PLEG): container finished" podID="39214b4b-e5c5-499c-ba7b-44b81db5a24a" containerID="e19bddd9e4e2c224e9dc21c9d443c2163a06d51f84d7e2ccd0e8376615a97bb1" exitCode=0 Jul 06 16:48:57.277883 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:57.277490 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-vfkkk" event={"ID":"39214b4b-e5c5-499c-ba7b-44b81db5a24a","Type":"ContainerDied","Data":"e19bddd9e4e2c224e9dc21c9d443c2163a06d51f84d7e2ccd0e8376615a97bb1"} Jul 06 16:48:58.282430 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:58.282396 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-vfkkk" event={"ID":"39214b4b-e5c5-499c-ba7b-44b81db5a24a","Type":"ContainerStarted","Data":"1ee21840ff2f5575635998e181bc249efe1c94bbc30e4d374669f7bae3af19e2"} Jul 06 16:48:58.282863 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:58.282539 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-vfkkk" Jul 06 16:48:58.311500 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:48:58.311453 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-vfkkk" podStartSLOduration=2.399373806 podStartE2EDuration="7.311438619s" totalImagesPullingTime="3.824993833s" totalInitContainerRuntime="1.08707098s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:48:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:48:58.310121765 +0000 UTC m=+631.829612911" watchObservedRunningTime="2026-07-06 16:48:58.311438619 +0000 UTC m=+631.830929733" Jul 06 16:49:09.287941 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:09.287909 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d66bcd77d-vfkkk" Jul 06 16:49:11.492048 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:11.490097 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5548f7d4db-d8r2s"] Jul 06 16:49:11.493556 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:11.493440 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5548f7d4db-d8r2s" Jul 06 16:49:11.496670 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:11.496640 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 06 16:49:11.497810 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:11.497790 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 16:49:11.497932 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:11.497819 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 16:49:11.497932 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:11.497792 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-pxdww\"" Jul 06 16:49:11.497932 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:11.497895 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 06 16:49:11.513280 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:11.513255 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5548f7d4db-d8r2s"] Jul 06 16:49:11.631363 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:11.631328 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wr47\" (UniqueName: \"kubernetes.io/projected/d74d37ae-0e47-45c8-9520-b490f54badf9-kube-api-access-8wr47\") pod \"kube-auth-proxy-5548f7d4db-d8r2s\" (UID: \"d74d37ae-0e47-45c8-9520-b490f54badf9\") " pod="openshift-ingress/kube-auth-proxy-5548f7d4db-d8r2s" Jul 06 16:49:11.631363 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:11.631364 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d74d37ae-0e47-45c8-9520-b490f54badf9-tls-certs\") pod \"kube-auth-proxy-5548f7d4db-d8r2s\" (UID: \"d74d37ae-0e47-45c8-9520-b490f54badf9\") " pod="openshift-ingress/kube-auth-proxy-5548f7d4db-d8r2s" Jul 06 16:49:11.631577 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:11.631393 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d74d37ae-0e47-45c8-9520-b490f54badf9-tmp\") pod \"kube-auth-proxy-5548f7d4db-d8r2s\" (UID: \"d74d37ae-0e47-45c8-9520-b490f54badf9\") " pod="openshift-ingress/kube-auth-proxy-5548f7d4db-d8r2s" Jul 06 16:49:11.732678 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:11.732638 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wr47\" (UniqueName: \"kubernetes.io/projected/d74d37ae-0e47-45c8-9520-b490f54badf9-kube-api-access-8wr47\") pod \"kube-auth-proxy-5548f7d4db-d8r2s\" (UID: \"d74d37ae-0e47-45c8-9520-b490f54badf9\") " pod="openshift-ingress/kube-auth-proxy-5548f7d4db-d8r2s" Jul 06 16:49:11.732678 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:11.732683 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d74d37ae-0e47-45c8-9520-b490f54badf9-tls-certs\") pod \"kube-auth-proxy-5548f7d4db-d8r2s\" (UID: \"d74d37ae-0e47-45c8-9520-b490f54badf9\") " pod="openshift-ingress/kube-auth-proxy-5548f7d4db-d8r2s" Jul 06 16:49:11.732872 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:11.732721 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d74d37ae-0e47-45c8-9520-b490f54badf9-tmp\") pod \"kube-auth-proxy-5548f7d4db-d8r2s\" (UID: \"d74d37ae-0e47-45c8-9520-b490f54badf9\") " pod="openshift-ingress/kube-auth-proxy-5548f7d4db-d8r2s" Jul 06 16:49:11.734990 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:11.734969 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d74d37ae-0e47-45c8-9520-b490f54badf9-tmp\") pod \"kube-auth-proxy-5548f7d4db-d8r2s\" (UID: \"d74d37ae-0e47-45c8-9520-b490f54badf9\") " pod="openshift-ingress/kube-auth-proxy-5548f7d4db-d8r2s" Jul 06 16:49:11.735286 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:11.735264 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d74d37ae-0e47-45c8-9520-b490f54badf9-tls-certs\") pod \"kube-auth-proxy-5548f7d4db-d8r2s\" (UID: \"d74d37ae-0e47-45c8-9520-b490f54badf9\") " pod="openshift-ingress/kube-auth-proxy-5548f7d4db-d8r2s" Jul 06 16:49:11.746770 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:11.746701 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wr47\" (UniqueName: \"kubernetes.io/projected/d74d37ae-0e47-45c8-9520-b490f54badf9-kube-api-access-8wr47\") pod \"kube-auth-proxy-5548f7d4db-d8r2s\" (UID: \"d74d37ae-0e47-45c8-9520-b490f54badf9\") " pod="openshift-ingress/kube-auth-proxy-5548f7d4db-d8r2s" Jul 06 16:49:11.803739 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:11.803712 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5548f7d4db-d8r2s" Jul 06 16:49:11.947469 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:11.947441 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5548f7d4db-d8r2s"] Jul 06 16:49:11.950374 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:49:11.950334 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd74d37ae_0e47_45c8_9520_b490f54badf9.slice/crio-03aec78f1fca952c43ce50eaa9c5d5779a66347b34cc2b439b01df3f009ae387 WatchSource:0}: Error finding container 03aec78f1fca952c43ce50eaa9c5d5779a66347b34cc2b439b01df3f009ae387: Status 404 returned error can't find the container with id 03aec78f1fca952c43ce50eaa9c5d5779a66347b34cc2b439b01df3f009ae387 Jul 06 16:49:12.332881 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:12.332845 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5548f7d4db-d8r2s" event={"ID":"d74d37ae-0e47-45c8-9520-b490f54badf9","Type":"ContainerStarted","Data":"03aec78f1fca952c43ce50eaa9c5d5779a66347b34cc2b439b01df3f009ae387"} Jul 06 16:49:13.889999 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:13.889958 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf"] Jul 06 16:49:13.892423 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:13.892401 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" Jul 06 16:49:13.895125 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:13.895098 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 06 16:49:13.895253 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:13.895102 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wcv2l\"" Jul 06 16:49:13.895651 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:13.895593 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 06 16:49:13.896504 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:13.896482 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 06 16:49:13.896743 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:13.896725 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 16:49:13.909022 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:13.908999 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf"] Jul 06 16:49:13.952999 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:13.952966 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3401cccf-4a8a-4d31-9e61-0e1e43dca749-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-5r4hf\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" Jul 06 16:49:13.953168 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:13.953012 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3401cccf-4a8a-4d31-9e61-0e1e43dca749-local-certs\") pod \"istiod-openshift-gateway-7574459656-5r4hf\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" Jul 06 16:49:13.953168 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:13.953037 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3401cccf-4a8a-4d31-9e61-0e1e43dca749-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-5r4hf\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" Jul 06 16:49:13.953168 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:13.953129 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3401cccf-4a8a-4d31-9e61-0e1e43dca749-cacerts\") pod \"istiod-openshift-gateway-7574459656-5r4hf\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" Jul 06 16:49:13.953310 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:13.953179 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5pgx\" (UniqueName: \"kubernetes.io/projected/3401cccf-4a8a-4d31-9e61-0e1e43dca749-kube-api-access-q5pgx\") pod \"istiod-openshift-gateway-7574459656-5r4hf\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" Jul 06 16:49:13.953310 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:13.953213 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3401cccf-4a8a-4d31-9e61-0e1e43dca749-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-5r4hf\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" Jul 06 16:49:13.953310 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:13.953264 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3401cccf-4a8a-4d31-9e61-0e1e43dca749-istio-token\") pod \"istiod-openshift-gateway-7574459656-5r4hf\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" Jul 06 16:49:14.054246 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:14.054209 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3401cccf-4a8a-4d31-9e61-0e1e43dca749-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-5r4hf\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" Jul 06 16:49:14.054409 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:14.054260 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3401cccf-4a8a-4d31-9e61-0e1e43dca749-local-certs\") pod \"istiod-openshift-gateway-7574459656-5r4hf\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" Jul 06 16:49:14.054409 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:14.054286 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3401cccf-4a8a-4d31-9e61-0e1e43dca749-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-5r4hf\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" Jul 06 16:49:14.054409 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:14.054328 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3401cccf-4a8a-4d31-9e61-0e1e43dca749-cacerts\") pod \"istiod-openshift-gateway-7574459656-5r4hf\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" Jul 06 16:49:14.054409 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:14.054356 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5pgx\" (UniqueName: \"kubernetes.io/projected/3401cccf-4a8a-4d31-9e61-0e1e43dca749-kube-api-access-q5pgx\") pod \"istiod-openshift-gateway-7574459656-5r4hf\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" Jul 06 16:49:14.054409 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:14.054384 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3401cccf-4a8a-4d31-9e61-0e1e43dca749-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-5r4hf\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" Jul 06 16:49:14.054710 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:14.054534 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3401cccf-4a8a-4d31-9e61-0e1e43dca749-istio-token\") pod \"istiod-openshift-gateway-7574459656-5r4hf\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" Jul 06 16:49:14.055171 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:14.055149 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3401cccf-4a8a-4d31-9e61-0e1e43dca749-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-5r4hf\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" Jul 06 16:49:14.056735 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:14.056694 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3401cccf-4a8a-4d31-9e61-0e1e43dca749-cacerts\") pod \"istiod-openshift-gateway-7574459656-5r4hf\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" Jul 06 16:49:14.056735 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:14.056728 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3401cccf-4a8a-4d31-9e61-0e1e43dca749-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-5r4hf\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" Jul 06 16:49:14.056932 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:14.056913 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3401cccf-4a8a-4d31-9e61-0e1e43dca749-local-certs\") pod \"istiod-openshift-gateway-7574459656-5r4hf\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" Jul 06 16:49:14.056989 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:14.056920 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3401cccf-4a8a-4d31-9e61-0e1e43dca749-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-5r4hf\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" Jul 06 16:49:14.065194 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:14.065173 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5pgx\" (UniqueName: \"kubernetes.io/projected/3401cccf-4a8a-4d31-9e61-0e1e43dca749-kube-api-access-q5pgx\") pod \"istiod-openshift-gateway-7574459656-5r4hf\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" Jul 06 16:49:14.065742 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:14.065723 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3401cccf-4a8a-4d31-9e61-0e1e43dca749-istio-token\") pod \"istiod-openshift-gateway-7574459656-5r4hf\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" Jul 06 16:49:14.203597 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:14.203562 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" Jul 06 16:49:14.381607 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:14.381563 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf"] Jul 06 16:49:14.385708 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:49:14.385676 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3401cccf_4a8a_4d31_9e61_0e1e43dca749.slice/crio-81206a522170f12c4a5b70c388062a7db185d59a52e29965bd1f13735e709d81 WatchSource:0}: Error finding container 81206a522170f12c4a5b70c388062a7db185d59a52e29965bd1f13735e709d81: Status 404 returned error can't find the container with id 81206a522170f12c4a5b70c388062a7db185d59a52e29965bd1f13735e709d81 Jul 06 16:49:15.347132 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:15.347082 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" event={"ID":"3401cccf-4a8a-4d31-9e61-0e1e43dca749","Type":"ContainerStarted","Data":"81206a522170f12c4a5b70c388062a7db185d59a52e29965bd1f13735e709d81"} Jul 06 16:49:16.358917 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:16.358885 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5548f7d4db-d8r2s" event={"ID":"d74d37ae-0e47-45c8-9520-b490f54badf9","Type":"ContainerStarted","Data":"bdc26e669261df60b170e57cfadf9368ce3303e1126c126c430c951126a9ebb2"} Jul 06 16:49:16.383567 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:16.383492 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5548f7d4db-d8r2s" podStartSLOduration=1.6424233209999999 podStartE2EDuration="5.383473694s" totalImagesPullingTime="3.741050373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:49:16.382633265 +0000 UTC m=+649.902124393" watchObservedRunningTime="2026-07-06 16:49:16.383473694 +0000 UTC m=+649.902964811" Jul 06 16:49:17.502109 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:17.502052 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 16:49:17.502381 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:17.502146 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 16:49:18.368435 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:18.368402 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" event={"ID":"3401cccf-4a8a-4d31-9e61-0e1e43dca749","Type":"ContainerStarted","Data":"70bb6cd7b8cf7eaf45a86684c9233e42867c75c020253ceb2c5c89a39ab6680b"} Jul 06 16:49:18.368657 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:18.368571 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" Jul 06 16:49:18.370098 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:18.370071 2574 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-5r4hf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 16:49:18.370208 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:18.370118 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" podUID="3401cccf-4a8a-4d31-9e61-0e1e43dca749" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 16:49:18.397960 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:18.397905 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" podStartSLOduration=2.284264204 podStartE2EDuration="5.397888906s" totalImagesPullingTime="3.113624702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:49:18.396709894 +0000 UTC m=+651.916201013" watchObservedRunningTime="2026-07-06 16:49:18.397888906 +0000 UTC m=+651.917380025" Jul 06 16:49:19.373469 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:19.373443 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" Jul 06 16:49:23.875747 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:23.875708 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zkvgc"] Jul 06 16:49:23.878251 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:23.878230 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zkvgc" Jul 06 16:49:23.881423 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:23.881401 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-rw9dm\"" Jul 06 16:49:23.881692 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:23.881663 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 06 16:49:23.893738 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:23.893716 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-xzhs8"] Jul 06 16:49:23.896333 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:23.896314 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xzhs8" Jul 06 16:49:23.897147 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:23.897125 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zkvgc"] Jul 06 16:49:23.899499 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:23.899482 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-rp7p4\"" Jul 06 16:49:23.899842 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:23.899828 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 06 16:49:23.917390 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:23.917367 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-xzhs8"] Jul 06 16:49:24.039350 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:24.039317 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b7e6bbe-d837-4656-b585-d2f9b254830e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xzhs8\" (UID: \"3b7e6bbe-d837-4656-b585-d2f9b254830e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xzhs8" Jul 06 16:49:24.039566 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:24.039366 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ksp7\" (UniqueName: \"kubernetes.io/projected/3b7e6bbe-d837-4656-b585-d2f9b254830e-kube-api-access-9ksp7\") pod \"llmisvc-controller-manager-777c6bd7d5-xzhs8\" (UID: \"3b7e6bbe-d837-4656-b585-d2f9b254830e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xzhs8" Jul 06 16:49:24.039566 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:24.039469 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qkkd\" (UniqueName: \"kubernetes.io/projected/6a166ea3-4b8f-4785-90e7-d807751e6567-kube-api-access-8qkkd\") pod \"kserve-controller-manager-d6f4b65b8-zkvgc\" (UID: \"6a166ea3-4b8f-4785-90e7-d807751e6567\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zkvgc" Jul 06 16:49:24.039566 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:24.039503 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a166ea3-4b8f-4785-90e7-d807751e6567-cert\") pod \"kserve-controller-manager-d6f4b65b8-zkvgc\" (UID: \"6a166ea3-4b8f-4785-90e7-d807751e6567\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zkvgc" Jul 06 16:49:24.140604 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:24.140507 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b7e6bbe-d837-4656-b585-d2f9b254830e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xzhs8\" (UID: \"3b7e6bbe-d837-4656-b585-d2f9b254830e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xzhs8" Jul 06 16:49:24.140604 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:24.140563 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ksp7\" (UniqueName: \"kubernetes.io/projected/3b7e6bbe-d837-4656-b585-d2f9b254830e-kube-api-access-9ksp7\") pod \"llmisvc-controller-manager-777c6bd7d5-xzhs8\" (UID: \"3b7e6bbe-d837-4656-b585-d2f9b254830e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xzhs8" Jul 06 16:49:24.140604 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:24.140605 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qkkd\" (UniqueName: \"kubernetes.io/projected/6a166ea3-4b8f-4785-90e7-d807751e6567-kube-api-access-8qkkd\") pod \"kserve-controller-manager-d6f4b65b8-zkvgc\" (UID: \"6a166ea3-4b8f-4785-90e7-d807751e6567\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zkvgc" Jul 06 16:49:24.140819 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:24.140625 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a166ea3-4b8f-4785-90e7-d807751e6567-cert\") pod \"kserve-controller-manager-d6f4b65b8-zkvgc\" (UID: \"6a166ea3-4b8f-4785-90e7-d807751e6567\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zkvgc" Jul 06 16:49:24.142990 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:24.142960 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a166ea3-4b8f-4785-90e7-d807751e6567-cert\") pod \"kserve-controller-manager-d6f4b65b8-zkvgc\" (UID: \"6a166ea3-4b8f-4785-90e7-d807751e6567\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zkvgc" Jul 06 16:49:24.143095 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:24.143040 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b7e6bbe-d837-4656-b585-d2f9b254830e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-xzhs8\" (UID: \"3b7e6bbe-d837-4656-b585-d2f9b254830e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xzhs8" Jul 06 16:49:24.160684 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:24.160661 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qkkd\" (UniqueName: \"kubernetes.io/projected/6a166ea3-4b8f-4785-90e7-d807751e6567-kube-api-access-8qkkd\") pod \"kserve-controller-manager-d6f4b65b8-zkvgc\" (UID: \"6a166ea3-4b8f-4785-90e7-d807751e6567\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zkvgc" Jul 06 16:49:24.168854 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:24.168832 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ksp7\" (UniqueName: \"kubernetes.io/projected/3b7e6bbe-d837-4656-b585-d2f9b254830e-kube-api-access-9ksp7\") pod \"llmisvc-controller-manager-777c6bd7d5-xzhs8\" (UID: \"3b7e6bbe-d837-4656-b585-d2f9b254830e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xzhs8" Jul 06 16:49:24.189721 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:24.189696 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zkvgc" Jul 06 16:49:24.207604 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:24.207574 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xzhs8" Jul 06 16:49:24.365680 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:24.365648 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zkvgc"] Jul 06 16:49:24.369752 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:49:24.369724 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a166ea3_4b8f_4785_90e7_d807751e6567.slice/crio-6c75d571e7aeee30c19646c4e049fa9bfc436d20c37978e8c2ee1affdaefe20f WatchSource:0}: Error finding container 6c75d571e7aeee30c19646c4e049fa9bfc436d20c37978e8c2ee1affdaefe20f: Status 404 returned error can't find the container with id 6c75d571e7aeee30c19646c4e049fa9bfc436d20c37978e8c2ee1affdaefe20f Jul 06 16:49:24.386863 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:24.386840 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-xzhs8"] Jul 06 16:49:24.388994 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:49:24.388968 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3b7e6bbe_d837_4656_b585_d2f9b254830e.slice/crio-96f2b5ef67b623fb1d94d04474393989b6b599f6b925bf76deed386a44074f07 WatchSource:0}: Error finding container 96f2b5ef67b623fb1d94d04474393989b6b599f6b925bf76deed386a44074f07: Status 404 returned error can't find the container with id 96f2b5ef67b623fb1d94d04474393989b6b599f6b925bf76deed386a44074f07 Jul 06 16:49:24.392937 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:24.392886 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zkvgc" event={"ID":"6a166ea3-4b8f-4785-90e7-d807751e6567","Type":"ContainerStarted","Data":"6c75d571e7aeee30c19646c4e049fa9bfc436d20c37978e8c2ee1affdaefe20f"} Jul 06 16:49:25.398306 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:25.398274 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xzhs8" event={"ID":"3b7e6bbe-d837-4656-b585-d2f9b254830e","Type":"ContainerStarted","Data":"96f2b5ef67b623fb1d94d04474393989b6b599f6b925bf76deed386a44074f07"} Jul 06 16:49:27.388938 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:27.388901 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf"] Jul 06 16:49:27.389416 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:27.389104 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" podUID="3401cccf-4a8a-4d31-9e61-0e1e43dca749" containerName="discovery" containerID="cri-o://70bb6cd7b8cf7eaf45a86684c9233e42867c75c020253ceb2c5c89a39ab6680b" gracePeriod=30 Jul 06 16:49:28.314889 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.314864 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" Jul 06 16:49:28.413667 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.413624 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zkvgc" event={"ID":"6a166ea3-4b8f-4785-90e7-d807751e6567","Type":"ContainerStarted","Data":"d4f2a6fc45d6562e0ca86c64cb514ab0548be95da0e8e1f086e121053a789f0e"} Jul 06 16:49:28.414027 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.413743 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zkvgc" Jul 06 16:49:28.414955 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.414932 2574 generic.go:356] "Generic (PLEG): container finished" podID="3401cccf-4a8a-4d31-9e61-0e1e43dca749" containerID="70bb6cd7b8cf7eaf45a86684c9233e42867c75c020253ceb2c5c89a39ab6680b" exitCode=0 Jul 06 16:49:28.415061 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.414983 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" Jul 06 16:49:28.415061 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.414995 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" event={"ID":"3401cccf-4a8a-4d31-9e61-0e1e43dca749","Type":"ContainerDied","Data":"70bb6cd7b8cf7eaf45a86684c9233e42867c75c020253ceb2c5c89a39ab6680b"} Jul 06 16:49:28.415061 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.415026 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf" event={"ID":"3401cccf-4a8a-4d31-9e61-0e1e43dca749","Type":"ContainerDied","Data":"81206a522170f12c4a5b70c388062a7db185d59a52e29965bd1f13735e709d81"} Jul 06 16:49:28.415061 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.415046 2574 scope.go:117] "RemoveContainer" containerID="70bb6cd7b8cf7eaf45a86684c9233e42867c75c020253ceb2c5c89a39ab6680b" Jul 06 16:49:28.422576 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.422559 2574 scope.go:117] "RemoveContainer" containerID="70bb6cd7b8cf7eaf45a86684c9233e42867c75c020253ceb2c5c89a39ab6680b" Jul 06 16:49:28.422837 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:49:28.422820 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70bb6cd7b8cf7eaf45a86684c9233e42867c75c020253ceb2c5c89a39ab6680b\": container with ID starting with 70bb6cd7b8cf7eaf45a86684c9233e42867c75c020253ceb2c5c89a39ab6680b not found: ID does not exist" containerID="70bb6cd7b8cf7eaf45a86684c9233e42867c75c020253ceb2c5c89a39ab6680b" Jul 06 16:49:28.422895 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.422847 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70bb6cd7b8cf7eaf45a86684c9233e42867c75c020253ceb2c5c89a39ab6680b"} err="failed to get container status \"70bb6cd7b8cf7eaf45a86684c9233e42867c75c020253ceb2c5c89a39ab6680b\": rpc error: code = NotFound desc = could not find container \"70bb6cd7b8cf7eaf45a86684c9233e42867c75c020253ceb2c5c89a39ab6680b\": container with ID starting with 70bb6cd7b8cf7eaf45a86684c9233e42867c75c020253ceb2c5c89a39ab6680b not found: ID does not exist" Jul 06 16:49:28.450387 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.450335 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zkvgc" podStartSLOduration=1.501450256 podStartE2EDuration="5.450317151s" totalImagesPullingTime="3.948866895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:49:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:49:28.449819636 +0000 UTC m=+661.969310753" watchObservedRunningTime="2026-07-06 16:49:28.450317151 +0000 UTC m=+661.969808268" Jul 06 16:49:28.481328 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.481305 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3401cccf-4a8a-4d31-9e61-0e1e43dca749-istio-kubeconfig\") pod \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " Jul 06 16:49:28.481434 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.481354 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5pgx\" (UniqueName: \"kubernetes.io/projected/3401cccf-4a8a-4d31-9e61-0e1e43dca749-kube-api-access-q5pgx\") pod \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " Jul 06 16:49:28.481434 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.481396 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3401cccf-4a8a-4d31-9e61-0e1e43dca749-istio-token\") pod \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " Jul 06 16:49:28.481566 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.481458 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3401cccf-4a8a-4d31-9e61-0e1e43dca749-cacerts\") pod \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " Jul 06 16:49:28.481566 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.481498 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3401cccf-4a8a-4d31-9e61-0e1e43dca749-local-certs\") pod \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " Jul 06 16:49:28.481566 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.481552 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3401cccf-4a8a-4d31-9e61-0e1e43dca749-istio-csr-ca-configmap\") pod \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " Jul 06 16:49:28.481713 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.481667 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3401cccf-4a8a-4d31-9e61-0e1e43dca749-istio-csr-dns-cert\") pod \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\" (UID: \"3401cccf-4a8a-4d31-9e61-0e1e43dca749\") " Jul 06 16:49:28.481924 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.481886 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3401cccf-4a8a-4d31-9e61-0e1e43dca749-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "3401cccf-4a8a-4d31-9e61-0e1e43dca749" (UID: "3401cccf-4a8a-4d31-9e61-0e1e43dca749"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:49:28.482044 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.481972 2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3401cccf-4a8a-4d31-9e61-0e1e43dca749-istio-csr-ca-configmap\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:49:28.483621 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.483595 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3401cccf-4a8a-4d31-9e61-0e1e43dca749-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "3401cccf-4a8a-4d31-9e61-0e1e43dca749" (UID: "3401cccf-4a8a-4d31-9e61-0e1e43dca749"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:49:28.483726 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.483640 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3401cccf-4a8a-4d31-9e61-0e1e43dca749-cacerts" (OuterVolumeSpecName: "cacerts") pod "3401cccf-4a8a-4d31-9e61-0e1e43dca749" (UID: "3401cccf-4a8a-4d31-9e61-0e1e43dca749"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:49:28.483950 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.483934 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3401cccf-4a8a-4d31-9e61-0e1e43dca749-local-certs" (OuterVolumeSpecName: "local-certs") pod "3401cccf-4a8a-4d31-9e61-0e1e43dca749" (UID: "3401cccf-4a8a-4d31-9e61-0e1e43dca749"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:49:28.484008 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.483972 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3401cccf-4a8a-4d31-9e61-0e1e43dca749-kube-api-access-q5pgx" (OuterVolumeSpecName: "kube-api-access-q5pgx") pod "3401cccf-4a8a-4d31-9e61-0e1e43dca749" (UID: "3401cccf-4a8a-4d31-9e61-0e1e43dca749"). InnerVolumeSpecName "kube-api-access-q5pgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:49:28.484062 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.484021 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3401cccf-4a8a-4d31-9e61-0e1e43dca749-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "3401cccf-4a8a-4d31-9e61-0e1e43dca749" (UID: "3401cccf-4a8a-4d31-9e61-0e1e43dca749"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:49:28.484062 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.484048 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3401cccf-4a8a-4d31-9e61-0e1e43dca749-istio-token" (OuterVolumeSpecName: "istio-token") pod "3401cccf-4a8a-4d31-9e61-0e1e43dca749" (UID: "3401cccf-4a8a-4d31-9e61-0e1e43dca749"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:49:28.582682 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.582598 2574 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3401cccf-4a8a-4d31-9e61-0e1e43dca749-cacerts\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:49:28.582682 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.582632 2574 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3401cccf-4a8a-4d31-9e61-0e1e43dca749-local-certs\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:49:28.582682 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.582647 2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3401cccf-4a8a-4d31-9e61-0e1e43dca749-istio-csr-dns-cert\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:49:28.582682 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.582669 2574 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3401cccf-4a8a-4d31-9e61-0e1e43dca749-istio-kubeconfig\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:49:28.582682 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.582683 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5pgx\" (UniqueName: \"kubernetes.io/projected/3401cccf-4a8a-4d31-9e61-0e1e43dca749-kube-api-access-q5pgx\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:49:28.583041 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.582697 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3401cccf-4a8a-4d31-9e61-0e1e43dca749-istio-token\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:49:28.745578 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.745542 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf"] Jul 06 16:49:28.752737 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:28.752708 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-5r4hf"] Jul 06 16:49:29.080187 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:29.080156 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3401cccf-4a8a-4d31-9e61-0e1e43dca749" path="/var/lib/kubelet/pods/3401cccf-4a8a-4d31-9e61-0e1e43dca749/volumes" Jul 06 16:49:29.420854 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:29.420754 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xzhs8" event={"ID":"3b7e6bbe-d837-4656-b585-d2f9b254830e","Type":"ContainerStarted","Data":"b3ee170df7646f4a9bcca322ee323a22b1c14de43877fbd0d180cf77b2e6593d"} Jul 06 16:49:29.421270 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:29.420893 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xzhs8" Jul 06 16:49:29.444894 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:29.444837 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xzhs8" podStartSLOduration=2.517057394 podStartE2EDuration="6.444821509s" totalImagesPullingTime="3.927764115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:49:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:49:29.444502084 +0000 UTC m=+662.963993200" watchObservedRunningTime="2026-07-06 16:49:29.444821509 +0000 UTC m=+662.964312639" Jul 06 16:49:59.427332 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:49:59.427298 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zkvgc" Jul 06 16:50:00.427939 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:00.427911 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-xzhs8" Jul 06 16:50:12.656669 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:12.656634 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zvzzw"] Jul 06 16:50:12.657043 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:12.656902 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3401cccf-4a8a-4d31-9e61-0e1e43dca749" containerName="discovery" Jul 06 16:50:12.657043 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:12.656912 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3401cccf-4a8a-4d31-9e61-0e1e43dca749" containerName="discovery" Jul 06 16:50:12.657043 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:12.656972 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3401cccf-4a8a-4d31-9e61-0e1e43dca749" containerName="discovery" Jul 06 16:50:12.663361 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:12.663332 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zvzzw" Jul 06 16:50:12.667773 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:12.667742 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 16:50:12.667924 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:12.667755 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 16:50:12.668587 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:12.668563 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-q8mw8\"" Jul 06 16:50:12.673475 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:12.673442 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zvzzw"] Jul 06 16:50:12.702906 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:12.702867 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbzpq\" (UniqueName: \"kubernetes.io/projected/80d87899-936b-47a4-b56c-0301f46e4070-kube-api-access-pbzpq\") pod \"kuadrant-operator-catalog-zvzzw\" (UID: \"80d87899-936b-47a4-b56c-0301f46e4070\") " pod="kuadrant-system/kuadrant-operator-catalog-zvzzw" Jul 06 16:50:12.804031 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:12.803999 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbzpq\" (UniqueName: \"kubernetes.io/projected/80d87899-936b-47a4-b56c-0301f46e4070-kube-api-access-pbzpq\") pod \"kuadrant-operator-catalog-zvzzw\" (UID: \"80d87899-936b-47a4-b56c-0301f46e4070\") " pod="kuadrant-system/kuadrant-operator-catalog-zvzzw" Jul 06 16:50:12.813180 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:12.813150 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbzpq\" (UniqueName: \"kubernetes.io/projected/80d87899-936b-47a4-b56c-0301f46e4070-kube-api-access-pbzpq\") pod \"kuadrant-operator-catalog-zvzzw\" (UID: \"80d87899-936b-47a4-b56c-0301f46e4070\") " pod="kuadrant-system/kuadrant-operator-catalog-zvzzw" Jul 06 16:50:12.975144 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:12.975098 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zvzzw" Jul 06 16:50:13.029917 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:13.029875 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zvzzw"] Jul 06 16:50:13.131850 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:13.131811 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zvzzw"] Jul 06 16:50:13.135733 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:50:13.135704 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d87899_936b_47a4_b56c_0301f46e4070.slice/crio-673d2f8c1ff55e3282b8defb8efeee5f8a0cf792e59c7cec6547e4744471e2ca WatchSource:0}: Error finding container 673d2f8c1ff55e3282b8defb8efeee5f8a0cf792e59c7cec6547e4744471e2ca: Status 404 returned error can't find the container with id 673d2f8c1ff55e3282b8defb8efeee5f8a0cf792e59c7cec6547e4744471e2ca Jul 06 16:50:13.230777 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:13.230692 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nr28z"] Jul 06 16:50:13.233351 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:13.233334 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-nr28z" Jul 06 16:50:13.246872 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:13.246845 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nr28z"] Jul 06 16:50:13.308839 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:13.308798 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd7vs\" (UniqueName: \"kubernetes.io/projected/ea5197b4-ea14-4d28-b4c9-16fcd3f6bf69-kube-api-access-gd7vs\") pod \"kuadrant-operator-catalog-nr28z\" (UID: \"ea5197b4-ea14-4d28-b4c9-16fcd3f6bf69\") " pod="kuadrant-system/kuadrant-operator-catalog-nr28z" Jul 06 16:50:13.409649 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:13.409605 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd7vs\" (UniqueName: \"kubernetes.io/projected/ea5197b4-ea14-4d28-b4c9-16fcd3f6bf69-kube-api-access-gd7vs\") pod \"kuadrant-operator-catalog-nr28z\" (UID: \"ea5197b4-ea14-4d28-b4c9-16fcd3f6bf69\") " pod="kuadrant-system/kuadrant-operator-catalog-nr28z" Jul 06 16:50:13.418447 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:13.418417 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd7vs\" (UniqueName: \"kubernetes.io/projected/ea5197b4-ea14-4d28-b4c9-16fcd3f6bf69-kube-api-access-gd7vs\") pod \"kuadrant-operator-catalog-nr28z\" (UID: \"ea5197b4-ea14-4d28-b4c9-16fcd3f6bf69\") " pod="kuadrant-system/kuadrant-operator-catalog-nr28z" Jul 06 16:50:13.542506 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:13.542413 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-nr28z" Jul 06 16:50:13.598270 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:13.598210 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zvzzw" event={"ID":"80d87899-936b-47a4-b56c-0301f46e4070","Type":"ContainerStarted","Data":"673d2f8c1ff55e3282b8defb8efeee5f8a0cf792e59c7cec6547e4744471e2ca"} Jul 06 16:50:13.674810 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:13.674742 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nr28z"] Jul 06 16:50:13.677228 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:50:13.677202 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5197b4_ea14_4d28_b4c9_16fcd3f6bf69.slice/crio-1ddf37ebc06e919dbf3d7cc568c46acbe6fa0d876c6ac0c8e751e0d63ffdc8c6 WatchSource:0}: Error finding container 1ddf37ebc06e919dbf3d7cc568c46acbe6fa0d876c6ac0c8e751e0d63ffdc8c6: Status 404 returned error can't find the container with id 1ddf37ebc06e919dbf3d7cc568c46acbe6fa0d876c6ac0c8e751e0d63ffdc8c6 Jul 06 16:50:14.605017 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:14.604947 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-nr28z" event={"ID":"ea5197b4-ea14-4d28-b4c9-16fcd3f6bf69","Type":"ContainerStarted","Data":"1ddf37ebc06e919dbf3d7cc568c46acbe6fa0d876c6ac0c8e751e0d63ffdc8c6"} Jul 06 16:50:15.611458 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:15.611344 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zvzzw" event={"ID":"80d87899-936b-47a4-b56c-0301f46e4070","Type":"ContainerStarted","Data":"e3da75210ee54bc5cb1032db129dc1518b9638251ffa3c7b84b0dce4d0bd5535"} Jul 06 16:50:15.611458 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:15.611372 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-zvzzw" podUID="80d87899-936b-47a4-b56c-0301f46e4070" containerName="registry-server" containerID="cri-o://e3da75210ee54bc5cb1032db129dc1518b9638251ffa3c7b84b0dce4d0bd5535" gracePeriod=2 Jul 06 16:50:15.613067 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:15.613034 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-nr28z" event={"ID":"ea5197b4-ea14-4d28-b4c9-16fcd3f6bf69","Type":"ContainerStarted","Data":"18f6b3b8543d895c447108d502fd84a0c6215250be27359bdb3601c16a325c3e"} Jul 06 16:50:15.638065 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:15.638016 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-zvzzw" podStartSLOduration=1.323760584 podStartE2EDuration="3.637999872s" totalImagesPullingTime="2.314239288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:50:15.636289412 +0000 UTC m=+709.155780530" watchObservedRunningTime="2026-07-06 16:50:15.637999872 +0000 UTC m=+709.157490989" Jul 06 16:50:15.656973 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:15.656915 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-nr28z" podStartSLOduration=0.883428822 podStartE2EDuration="2.656893106s" totalImagesPullingTime="1.773464284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:50:15.656407146 +0000 UTC m=+709.175898276" watchObservedRunningTime="2026-07-06 16:50:15.656893106 +0000 UTC m=+709.176384223" Jul 06 16:50:15.849684 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:15.849659 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zvzzw" Jul 06 16:50:15.929464 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:15.929429 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbzpq\" (UniqueName: \"kubernetes.io/projected/80d87899-936b-47a4-b56c-0301f46e4070-kube-api-access-pbzpq\") pod \"80d87899-936b-47a4-b56c-0301f46e4070\" (UID: \"80d87899-936b-47a4-b56c-0301f46e4070\") " Jul 06 16:50:15.931616 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:15.931590 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80d87899-936b-47a4-b56c-0301f46e4070-kube-api-access-pbzpq" (OuterVolumeSpecName: "kube-api-access-pbzpq") pod "80d87899-936b-47a4-b56c-0301f46e4070" (UID: "80d87899-936b-47a4-b56c-0301f46e4070"). InnerVolumeSpecName "kube-api-access-pbzpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:50:16.030139 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:16.030068 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbzpq\" (UniqueName: \"kubernetes.io/projected/80d87899-936b-47a4-b56c-0301f46e4070-kube-api-access-pbzpq\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:50:16.618635 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:16.618602 2574 generic.go:356] "Generic (PLEG): container finished" podID="80d87899-936b-47a4-b56c-0301f46e4070" containerID="e3da75210ee54bc5cb1032db129dc1518b9638251ffa3c7b84b0dce4d0bd5535" exitCode=0 Jul 06 16:50:16.619037 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:16.618647 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zvzzw" Jul 06 16:50:16.619037 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:16.618684 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zvzzw" event={"ID":"80d87899-936b-47a4-b56c-0301f46e4070","Type":"ContainerDied","Data":"e3da75210ee54bc5cb1032db129dc1518b9638251ffa3c7b84b0dce4d0bd5535"} Jul 06 16:50:16.619037 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:16.618731 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zvzzw" event={"ID":"80d87899-936b-47a4-b56c-0301f46e4070","Type":"ContainerDied","Data":"673d2f8c1ff55e3282b8defb8efeee5f8a0cf792e59c7cec6547e4744471e2ca"} Jul 06 16:50:16.619037 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:16.618751 2574 scope.go:117] "RemoveContainer" containerID="e3da75210ee54bc5cb1032db129dc1518b9638251ffa3c7b84b0dce4d0bd5535" Jul 06 16:50:16.627736 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:16.627720 2574 scope.go:117] "RemoveContainer" containerID="e3da75210ee54bc5cb1032db129dc1518b9638251ffa3c7b84b0dce4d0bd5535" Jul 06 16:50:16.627987 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:50:16.627968 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3da75210ee54bc5cb1032db129dc1518b9638251ffa3c7b84b0dce4d0bd5535\": container with ID starting with e3da75210ee54bc5cb1032db129dc1518b9638251ffa3c7b84b0dce4d0bd5535 not found: ID does not exist" containerID="e3da75210ee54bc5cb1032db129dc1518b9638251ffa3c7b84b0dce4d0bd5535" Jul 06 16:50:16.628050 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:16.627997 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3da75210ee54bc5cb1032db129dc1518b9638251ffa3c7b84b0dce4d0bd5535"} err="failed to get container status \"e3da75210ee54bc5cb1032db129dc1518b9638251ffa3c7b84b0dce4d0bd5535\": rpc error: code = NotFound desc = could not find container \"e3da75210ee54bc5cb1032db129dc1518b9638251ffa3c7b84b0dce4d0bd5535\": container with ID starting with e3da75210ee54bc5cb1032db129dc1518b9638251ffa3c7b84b0dce4d0bd5535 not found: ID does not exist" Jul 06 16:50:16.640278 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:16.640254 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zvzzw"] Jul 06 16:50:16.643468 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:16.643448 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zvzzw"] Jul 06 16:50:17.075191 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:17.075156 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80d87899-936b-47a4-b56c-0301f46e4070" path="/var/lib/kubelet/pods/80d87899-936b-47a4-b56c-0301f46e4070/volumes" Jul 06 16:50:21.351786 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.351697 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9dd6795c7-td257"] Jul 06 16:50:21.352236 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.352068 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80d87899-936b-47a4-b56c-0301f46e4070" containerName="registry-server" Jul 06 16:50:21.352236 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.352088 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="80d87899-936b-47a4-b56c-0301f46e4070" containerName="registry-server" Jul 06 16:50:21.352236 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.352164 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="80d87899-936b-47a4-b56c-0301f46e4070" containerName="registry-server" Jul 06 16:50:21.355009 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.354984 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9dd6795c7-td257"] Jul 06 16:50:21.355163 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.355083 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9dd6795c7-td257" Jul 06 16:50:21.473054 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.473020 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7-console-config\") pod \"console-9dd6795c7-td257\" (UID: \"43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7\") " pod="openshift-console/console-9dd6795c7-td257" Jul 06 16:50:21.473054 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.473062 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7-console-oauth-config\") pod \"console-9dd6795c7-td257\" (UID: \"43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7\") " pod="openshift-console/console-9dd6795c7-td257" Jul 06 16:50:21.473284 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.473081 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7-trusted-ca-bundle\") pod \"console-9dd6795c7-td257\" (UID: \"43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7\") " pod="openshift-console/console-9dd6795c7-td257" Jul 06 16:50:21.473284 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.473155 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7-console-serving-cert\") pod \"console-9dd6795c7-td257\" (UID: \"43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7\") " pod="openshift-console/console-9dd6795c7-td257" Jul 06 16:50:21.473284 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.473192 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7-oauth-serving-cert\") pod \"console-9dd6795c7-td257\" (UID: \"43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7\") " pod="openshift-console/console-9dd6795c7-td257" Jul 06 16:50:21.473284 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.473222 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhjlm\" (UniqueName: \"kubernetes.io/projected/43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7-kube-api-access-rhjlm\") pod \"console-9dd6795c7-td257\" (UID: \"43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7\") " pod="openshift-console/console-9dd6795c7-td257" Jul 06 16:50:21.473284 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.473260 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7-service-ca\") pod \"console-9dd6795c7-td257\" (UID: \"43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7\") " pod="openshift-console/console-9dd6795c7-td257" Jul 06 16:50:21.573859 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.573811 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7-console-oauth-config\") pod \"console-9dd6795c7-td257\" (UID: \"43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7\") " pod="openshift-console/console-9dd6795c7-td257" Jul 06 16:50:21.573859 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.573862 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7-trusted-ca-bundle\") pod \"console-9dd6795c7-td257\" (UID: \"43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7\") " pod="openshift-console/console-9dd6795c7-td257" Jul 06 16:50:21.574074 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.573883 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7-console-serving-cert\") pod \"console-9dd6795c7-td257\" (UID: \"43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7\") " pod="openshift-console/console-9dd6795c7-td257" Jul 06 16:50:21.574074 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.573914 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7-oauth-serving-cert\") pod \"console-9dd6795c7-td257\" (UID: \"43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7\") " pod="openshift-console/console-9dd6795c7-td257" Jul 06 16:50:21.574074 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.573941 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhjlm\" (UniqueName: \"kubernetes.io/projected/43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7-kube-api-access-rhjlm\") pod \"console-9dd6795c7-td257\" (UID: \"43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7\") " pod="openshift-console/console-9dd6795c7-td257" Jul 06 16:50:21.574199 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.574096 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7-service-ca\") pod \"console-9dd6795c7-td257\" (UID: \"43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7\") " pod="openshift-console/console-9dd6795c7-td257" Jul 06 16:50:21.574254 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.574205 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7-console-config\") pod \"console-9dd6795c7-td257\" (UID: \"43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7\") " pod="openshift-console/console-9dd6795c7-td257" Jul 06 16:50:21.574809 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.574784 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7-trusted-ca-bundle\") pod \"console-9dd6795c7-td257\" (UID: \"43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7\") " pod="openshift-console/console-9dd6795c7-td257" Jul 06 16:50:21.574809 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.574805 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7-oauth-serving-cert\") pod \"console-9dd6795c7-td257\" (UID: \"43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7\") " pod="openshift-console/console-9dd6795c7-td257" Jul 06 16:50:21.575009 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.574902 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7-console-config\") pod \"console-9dd6795c7-td257\" (UID: \"43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7\") " pod="openshift-console/console-9dd6795c7-td257" Jul 06 16:50:21.575075 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.575008 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7-service-ca\") pod \"console-9dd6795c7-td257\" (UID: \"43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7\") " pod="openshift-console/console-9dd6795c7-td257" Jul 06 16:50:21.576550 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.576506 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7-console-oauth-config\") pod \"console-9dd6795c7-td257\" (UID: \"43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7\") " pod="openshift-console/console-9dd6795c7-td257" Jul 06 16:50:21.576647 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.576588 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7-console-serving-cert\") pod \"console-9dd6795c7-td257\" (UID: \"43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7\") " pod="openshift-console/console-9dd6795c7-td257" Jul 06 16:50:21.582703 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.582675 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhjlm\" (UniqueName: \"kubernetes.io/projected/43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7-kube-api-access-rhjlm\") pod \"console-9dd6795c7-td257\" (UID: \"43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7\") " pod="openshift-console/console-9dd6795c7-td257" Jul 06 16:50:21.666460 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.666374 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9dd6795c7-td257" Jul 06 16:50:21.795651 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:21.795613 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9dd6795c7-td257"] Jul 06 16:50:21.798383 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:50:21.798351 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43edfcba_7c81_4ce5_bcbd_22ba27f2e0e7.slice/crio-637f2d8cc0af5310d9ce4a264e6fed2776db3391c1b2f434520c3ca0a0d9a387 WatchSource:0}: Error finding container 637f2d8cc0af5310d9ce4a264e6fed2776db3391c1b2f434520c3ca0a0d9a387: Status 404 returned error can't find the container with id 637f2d8cc0af5310d9ce4a264e6fed2776db3391c1b2f434520c3ca0a0d9a387 Jul 06 16:50:22.644161 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:22.644119 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9dd6795c7-td257" event={"ID":"43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7","Type":"ContainerStarted","Data":"23e43f80d507dcb0e5c0a529b6ff9362c3f2c1df1bc21f61392a095a2afc6f8a"} Jul 06 16:50:22.644584 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:22.644187 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9dd6795c7-td257" event={"ID":"43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7","Type":"ContainerStarted","Data":"637f2d8cc0af5310d9ce4a264e6fed2776db3391c1b2f434520c3ca0a0d9a387"} Jul 06 16:50:22.665309 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:22.665253 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9dd6795c7-td257" podStartSLOduration=1.6652349370000001 podStartE2EDuration="1.665234937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:50:22.663802773 +0000 UTC m=+716.183293904" watchObservedRunningTime="2026-07-06 16:50:22.665234937 +0000 UTC m=+716.184726116" Jul 06 16:50:23.543050 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:23.543010 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-nr28z" Jul 06 16:50:23.543217 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:23.543096 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-nr28z" Jul 06 16:50:23.565180 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:23.565153 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-nr28z" Jul 06 16:50:23.669028 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:23.669006 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-nr28z" Jul 06 16:50:31.667150 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:31.667111 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9dd6795c7-td257" Jul 06 16:50:31.667635 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:31.667166 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9dd6795c7-td257" Jul 06 16:50:31.672036 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:31.672013 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9dd6795c7-td257" Jul 06 16:50:31.683584 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:31.683560 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9dd6795c7-td257" Jul 06 16:50:31.743824 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:31.743794 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-566f7ddb96-l989k"] Jul 06 16:50:45.859993 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:45.859955 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg"] Jul 06 16:50:45.864593 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:45.864572 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" Jul 06 16:50:45.868126 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:45.868104 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xjwsf\"" Jul 06 16:50:45.882153 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:45.882132 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg"] Jul 06 16:50:45.976211 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:45.976173 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b37bf77c-5dba-4d93-9b53-04a8cb0f7421-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4pbtg\" (UID: \"b37bf77c-5dba-4d93-9b53-04a8cb0f7421\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" Jul 06 16:50:45.976392 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:45.976223 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgdtm\" (UniqueName: \"kubernetes.io/projected/b37bf77c-5dba-4d93-9b53-04a8cb0f7421-kube-api-access-cgdtm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4pbtg\" (UID: \"b37bf77c-5dba-4d93-9b53-04a8cb0f7421\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" Jul 06 16:50:46.076724 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:46.076686 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b37bf77c-5dba-4d93-9b53-04a8cb0f7421-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4pbtg\" (UID: \"b37bf77c-5dba-4d93-9b53-04a8cb0f7421\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" Jul 06 16:50:46.076884 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:46.076732 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgdtm\" (UniqueName: \"kubernetes.io/projected/b37bf77c-5dba-4d93-9b53-04a8cb0f7421-kube-api-access-cgdtm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4pbtg\" (UID: \"b37bf77c-5dba-4d93-9b53-04a8cb0f7421\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" Jul 06 16:50:46.077124 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:46.077104 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b37bf77c-5dba-4d93-9b53-04a8cb0f7421-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4pbtg\" (UID: \"b37bf77c-5dba-4d93-9b53-04a8cb0f7421\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" Jul 06 16:50:46.093717 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:46.093695 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgdtm\" (UniqueName: \"kubernetes.io/projected/b37bf77c-5dba-4d93-9b53-04a8cb0f7421-kube-api-access-cgdtm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-4pbtg\" (UID: \"b37bf77c-5dba-4d93-9b53-04a8cb0f7421\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" Jul 06 16:50:46.174722 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:46.174633 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" Jul 06 16:50:46.309879 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:46.309845 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg"] Jul 06 16:50:46.314193 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:50:46.314163 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37bf77c_5dba_4d93_9b53_04a8cb0f7421.slice/crio-aeea89bd6d088a38d6ec5a355d29a6a3ec5770a566543ede3419fb85b7f01ff4 WatchSource:0}: Error finding container aeea89bd6d088a38d6ec5a355d29a6a3ec5770a566543ede3419fb85b7f01ff4: Status 404 returned error can't find the container with id aeea89bd6d088a38d6ec5a355d29a6a3ec5770a566543ede3419fb85b7f01ff4 Jul 06 16:50:46.738880 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:46.738847 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" event={"ID":"b37bf77c-5dba-4d93-9b53-04a8cb0f7421","Type":"ContainerStarted","Data":"aeea89bd6d088a38d6ec5a355d29a6a3ec5770a566543ede3419fb85b7f01ff4"} Jul 06 16:50:47.848770 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:47.848736 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds"] Jul 06 16:50:47.851945 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:47.851924 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds" Jul 06 16:50:47.854477 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:47.854459 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-mzdrb\"" Jul 06 16:50:47.867579 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:47.867558 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds"] Jul 06 16:50:47.994776 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:47.994738 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvdcm\" (UniqueName: \"kubernetes.io/projected/dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a-kube-api-access-gvdcm\") pod \"limitador-operator-controller-manager-859d7b78bd-925ds\" (UID: \"dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds" Jul 06 16:50:48.095811 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:48.095782 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvdcm\" (UniqueName: \"kubernetes.io/projected/dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a-kube-api-access-gvdcm\") pod \"limitador-operator-controller-manager-859d7b78bd-925ds\" (UID: \"dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds" Jul 06 16:50:48.107138 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:48.107065 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvdcm\" (UniqueName: \"kubernetes.io/projected/dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a-kube-api-access-gvdcm\") pod \"limitador-operator-controller-manager-859d7b78bd-925ds\" (UID: \"dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds" Jul 06 16:50:48.161998 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:48.161968 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds" Jul 06 16:50:48.296982 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:48.296950 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds"] Jul 06 16:50:48.300232 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:50:48.300198 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbab875d_7e8a_4a9d_94fc_5de4ffcbd30a.slice/crio-2c5d9da6397810c7e15a364c4c3d37bfd2edd8560e1eb8ea7e1ed03284e30106 WatchSource:0}: Error finding container 2c5d9da6397810c7e15a364c4c3d37bfd2edd8560e1eb8ea7e1ed03284e30106: Status 404 returned error can't find the container with id 2c5d9da6397810c7e15a364c4c3d37bfd2edd8560e1eb8ea7e1ed03284e30106 Jul 06 16:50:48.748035 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:48.747926 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds" event={"ID":"dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a","Type":"ContainerStarted","Data":"2c5d9da6397810c7e15a364c4c3d37bfd2edd8560e1eb8ea7e1ed03284e30106"} Jul 06 16:50:50.298435 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:50.298397 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-bwhj8"] Jul 06 16:50:50.300896 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:50.300821 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bwhj8" Jul 06 16:50:50.303625 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:50.303601 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-tp52q\"" Jul 06 16:50:50.311066 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:50.311038 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-bwhj8"] Jul 06 16:50:50.413192 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:50.413150 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl8jq\" (UniqueName: \"kubernetes.io/projected/6ad1e1c6-a52e-419e-9bc3-5fbd83b79ec9-kube-api-access-pl8jq\") pod \"authorino-operator-75cc8d5cb7-bwhj8\" (UID: \"6ad1e1c6-a52e-419e-9bc3-5fbd83b79ec9\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-bwhj8" Jul 06 16:50:50.514226 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:50.514197 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl8jq\" (UniqueName: \"kubernetes.io/projected/6ad1e1c6-a52e-419e-9bc3-5fbd83b79ec9-kube-api-access-pl8jq\") pod \"authorino-operator-75cc8d5cb7-bwhj8\" (UID: \"6ad1e1c6-a52e-419e-9bc3-5fbd83b79ec9\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-bwhj8" Jul 06 16:50:50.523919 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:50.523898 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl8jq\" (UniqueName: \"kubernetes.io/projected/6ad1e1c6-a52e-419e-9bc3-5fbd83b79ec9-kube-api-access-pl8jq\") pod \"authorino-operator-75cc8d5cb7-bwhj8\" (UID: \"6ad1e1c6-a52e-419e-9bc3-5fbd83b79ec9\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-bwhj8" Jul 06 16:50:50.613725 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:50.613618 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bwhj8" Jul 06 16:50:50.744460 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:50.744413 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-bwhj8"] Jul 06 16:50:50.747827 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:50:50.747799 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad1e1c6_a52e_419e_9bc3_5fbd83b79ec9.slice/crio-32646953c246425d5ee4221396b0b92ecdc10b3e6cd66b096bf0bd9a65b447eb WatchSource:0}: Error finding container 32646953c246425d5ee4221396b0b92ecdc10b3e6cd66b096bf0bd9a65b447eb: Status 404 returned error can't find the container with id 32646953c246425d5ee4221396b0b92ecdc10b3e6cd66b096bf0bd9a65b447eb Jul 06 16:50:50.759040 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:50.759009 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds" event={"ID":"dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a","Type":"ContainerStarted","Data":"85d59d74b3a62d3286540da01c74638b5f13aed540f4c8b9c4472839262ef239"} Jul 06 16:50:50.759153 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:50.759115 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds" Jul 06 16:50:50.760352 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:50.760324 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bwhj8" event={"ID":"6ad1e1c6-a52e-419e-9bc3-5fbd83b79ec9","Type":"ContainerStarted","Data":"32646953c246425d5ee4221396b0b92ecdc10b3e6cd66b096bf0bd9a65b447eb"} Jul 06 16:50:50.786904 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:50.786860 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds" podStartSLOduration=2.037527395 podStartE2EDuration="3.786844734s" totalImagesPullingTime="1.749317339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:50:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:50:50.78580732 +0000 UTC m=+744.305298449" watchObservedRunningTime="2026-07-06 16:50:50.786844734 +0000 UTC m=+744.306335860" Jul 06 16:50:53.729614 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:53.729591 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bhhj5"] Jul 06 16:50:53.732335 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:53.732311 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bhhj5" Jul 06 16:50:53.735067 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:53.735049 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-9lttc\"" Jul 06 16:50:53.735132 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:53.735064 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 06 16:50:53.743625 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:53.743606 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bhhj5"] Jul 06 16:50:53.782708 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:53.782680 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" event={"ID":"b37bf77c-5dba-4d93-9b53-04a8cb0f7421","Type":"ContainerStarted","Data":"360e7893322c9224fee0f077d13da9a9f2f7e36c6d8be0662479fc6bd0ac2d47"} Jul 06 16:50:53.843893 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:53.843854 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnwgj\" (UniqueName: \"kubernetes.io/projected/fb360155-5817-4af0-a955-2f55fb125800-kube-api-access-jnwgj\") pod \"dns-operator-controller-manager-864dd78c4c-bhhj5\" (UID: \"fb360155-5817-4af0-a955-2f55fb125800\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bhhj5" Jul 06 16:50:53.945335 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:53.945301 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnwgj\" (UniqueName: \"kubernetes.io/projected/fb360155-5817-4af0-a955-2f55fb125800-kube-api-access-jnwgj\") pod \"dns-operator-controller-manager-864dd78c4c-bhhj5\" (UID: \"fb360155-5817-4af0-a955-2f55fb125800\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bhhj5" Jul 06 16:50:53.954075 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:53.954047 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnwgj\" (UniqueName: \"kubernetes.io/projected/fb360155-5817-4af0-a955-2f55fb125800-kube-api-access-jnwgj\") pod \"dns-operator-controller-manager-864dd78c4c-bhhj5\" (UID: \"fb360155-5817-4af0-a955-2f55fb125800\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bhhj5" Jul 06 16:50:54.068590 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:54.068557 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bhhj5" Jul 06 16:50:54.203462 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:54.203395 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bhhj5"] Jul 06 16:50:54.205743 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:50:54.205711 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb360155_5817_4af0_a955_2f55fb125800.slice/crio-debcc15a256a7c420d3d6a45d5326fcd0b1cccdfa1efbe146a0c581133aea564 WatchSource:0}: Error finding container debcc15a256a7c420d3d6a45d5326fcd0b1cccdfa1efbe146a0c581133aea564: Status 404 returned error can't find the container with id debcc15a256a7c420d3d6a45d5326fcd0b1cccdfa1efbe146a0c581133aea564 Jul 06 16:50:54.787942 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:54.787904 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bwhj8" event={"ID":"6ad1e1c6-a52e-419e-9bc3-5fbd83b79ec9","Type":"ContainerStarted","Data":"60bc82f539e51556e6b9618c1029feb83ff566ff31462189097d230a04af7f98"} Jul 06 16:50:54.788412 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:54.787979 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bwhj8" Jul 06 16:50:54.789391 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:54.789367 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bhhj5" event={"ID":"fb360155-5817-4af0-a955-2f55fb125800","Type":"ContainerStarted","Data":"debcc15a256a7c420d3d6a45d5326fcd0b1cccdfa1efbe146a0c581133aea564"} Jul 06 16:50:54.789502 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:54.789450 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" Jul 06 16:50:54.806427 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:54.806383 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bwhj8" podStartSLOduration=1.8493245360000001 podStartE2EDuration="4.806370561s" totalImagesPullingTime="2.957046025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:50:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:50:54.804891623 +0000 UTC m=+748.324382752" watchObservedRunningTime="2026-07-06 16:50:54.806370561 +0000 UTC m=+748.325861743" Jul 06 16:50:54.832003 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:54.831952 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" podStartSLOduration=2.446918616 podStartE2EDuration="9.831937531s" totalImagesPullingTime="7.385018915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:50:54.830969949 +0000 UTC m=+748.350461102" watchObservedRunningTime="2026-07-06 16:50:54.831937531 +0000 UTC m=+748.351428646" Jul 06 16:50:56.538422 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:56.538386 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-8f2qf"] Jul 06 16:50:56.542452 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:56.542429 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f2qf" Jul 06 16:50:56.545317 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:56.545296 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-sfj6x\"" Jul 06 16:50:56.545541 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:56.545504 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 06 16:50:56.545651 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:56.545582 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 06 16:50:56.551710 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:56.551686 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-8f2qf"] Jul 06 16:50:56.568787 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:56.568765 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0d3ad447-6e87-43dd-bd9b-82847c08dd5e-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-8f2qf\" (UID: \"0d3ad447-6e87-43dd-bd9b-82847c08dd5e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f2qf" Jul 06 16:50:56.568896 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:56.568805 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx7vm\" (UniqueName: \"kubernetes.io/projected/0d3ad447-6e87-43dd-bd9b-82847c08dd5e-kube-api-access-nx7vm\") pod \"kuadrant-console-plugin-66b5774985-8f2qf\" (UID: \"0d3ad447-6e87-43dd-bd9b-82847c08dd5e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f2qf" Jul 06 16:50:56.568896 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:56.568876 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d3ad447-6e87-43dd-bd9b-82847c08dd5e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8f2qf\" (UID: \"0d3ad447-6e87-43dd-bd9b-82847c08dd5e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f2qf" Jul 06 16:50:56.670130 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:56.670031 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0d3ad447-6e87-43dd-bd9b-82847c08dd5e-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-8f2qf\" (UID: \"0d3ad447-6e87-43dd-bd9b-82847c08dd5e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f2qf" Jul 06 16:50:56.670130 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:56.670082 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx7vm\" (UniqueName: \"kubernetes.io/projected/0d3ad447-6e87-43dd-bd9b-82847c08dd5e-kube-api-access-nx7vm\") pod \"kuadrant-console-plugin-66b5774985-8f2qf\" (UID: \"0d3ad447-6e87-43dd-bd9b-82847c08dd5e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f2qf" Jul 06 16:50:56.670352 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:56.670135 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d3ad447-6e87-43dd-bd9b-82847c08dd5e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8f2qf\" (UID: \"0d3ad447-6e87-43dd-bd9b-82847c08dd5e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f2qf" Jul 06 16:50:56.670716 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:56.670693 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0d3ad447-6e87-43dd-bd9b-82847c08dd5e-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-8f2qf\" (UID: \"0d3ad447-6e87-43dd-bd9b-82847c08dd5e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f2qf" Jul 06 16:50:56.672628 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:56.672610 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d3ad447-6e87-43dd-bd9b-82847c08dd5e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8f2qf\" (UID: \"0d3ad447-6e87-43dd-bd9b-82847c08dd5e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f2qf" Jul 06 16:50:56.680266 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:56.680241 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx7vm\" (UniqueName: \"kubernetes.io/projected/0d3ad447-6e87-43dd-bd9b-82847c08dd5e-kube-api-access-nx7vm\") pod \"kuadrant-console-plugin-66b5774985-8f2qf\" (UID: \"0d3ad447-6e87-43dd-bd9b-82847c08dd5e\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f2qf" Jul 06 16:50:56.764903 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:56.764866 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-566f7ddb96-l989k" podUID="97dab90b-6dee-49c5-8fc2-ef1a6061931e" containerName="console" containerID="cri-o://4aa73770735319b227346ce44cdcdf104e148eafb85ae1cf0e623c27b86d6d31" gracePeriod=15 Jul 06 16:50:56.799137 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:56.799107 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bhhj5" event={"ID":"fb360155-5817-4af0-a955-2f55fb125800","Type":"ContainerStarted","Data":"38c097d755d7aa83f9843520ba8a53f74e141a0648390554dc6db0bd01adb3e7"} Jul 06 16:50:56.799252 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:56.799182 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bhhj5" Jul 06 16:50:56.821171 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:56.821125 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bhhj5" podStartSLOduration=1.61852706 podStartE2EDuration="3.821109756s" totalImagesPullingTime="2.202582696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:50:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:50:56.819664456 +0000 UTC m=+750.339155584" watchObservedRunningTime="2026-07-06 16:50:56.821109756 +0000 UTC m=+750.340600921" Jul 06 16:50:56.853310 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:56.853270 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f2qf" Jul 06 16:50:57.021404 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.021380 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-8f2qf"] Jul 06 16:50:57.023441 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:50:57.023414 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d3ad447_6e87_43dd_bd9b_82847c08dd5e.slice/crio-7d7e80a0a8aff56b3a06b78b3cb728dc2c12459fe437a69ca0be1e23c58066c4 WatchSource:0}: Error finding container 7d7e80a0a8aff56b3a06b78b3cb728dc2c12459fe437a69ca0be1e23c58066c4: Status 404 returned error can't find the container with id 7d7e80a0a8aff56b3a06b78b3cb728dc2c12459fe437a69ca0be1e23c58066c4 Jul 06 16:50:57.026278 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.026256 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:50:57.072646 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.072613 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97dab90b-6dee-49c5-8fc2-ef1a6061931e-service-ca\") pod \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " Jul 06 16:50:57.072797 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.072658 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97dab90b-6dee-49c5-8fc2-ef1a6061931e-console-config\") pod \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " Jul 06 16:50:57.072797 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.072694 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97dab90b-6dee-49c5-8fc2-ef1a6061931e-trusted-ca-bundle\") pod \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " Jul 06 16:50:57.072797 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.072749 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97dab90b-6dee-49c5-8fc2-ef1a6061931e-console-serving-cert\") pod \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " Jul 06 16:50:57.072797 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.072792 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97dab90b-6dee-49c5-8fc2-ef1a6061931e-console-oauth-config\") pod \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " Jul 06 16:50:57.073025 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.072837 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97dab90b-6dee-49c5-8fc2-ef1a6061931e-oauth-serving-cert\") pod \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " Jul 06 16:50:57.073025 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.072863 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmslq\" (UniqueName: \"kubernetes.io/projected/97dab90b-6dee-49c5-8fc2-ef1a6061931e-kube-api-access-hmslq\") pod \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\" (UID: \"97dab90b-6dee-49c5-8fc2-ef1a6061931e\") " Jul 06 16:50:57.073129 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.073044 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97dab90b-6dee-49c5-8fc2-ef1a6061931e-service-ca" (OuterVolumeSpecName: "service-ca") pod "97dab90b-6dee-49c5-8fc2-ef1a6061931e" (UID: "97dab90b-6dee-49c5-8fc2-ef1a6061931e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:50:57.073129 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.073118 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/97dab90b-6dee-49c5-8fc2-ef1a6061931e-service-ca\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:50:57.073234 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.073140 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97dab90b-6dee-49c5-8fc2-ef1a6061931e-console-config" (OuterVolumeSpecName: "console-config") pod "97dab90b-6dee-49c5-8fc2-ef1a6061931e" (UID: "97dab90b-6dee-49c5-8fc2-ef1a6061931e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:50:57.073432 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.073408 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97dab90b-6dee-49c5-8fc2-ef1a6061931e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "97dab90b-6dee-49c5-8fc2-ef1a6061931e" (UID: "97dab90b-6dee-49c5-8fc2-ef1a6061931e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:50:57.073549 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.073456 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97dab90b-6dee-49c5-8fc2-ef1a6061931e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "97dab90b-6dee-49c5-8fc2-ef1a6061931e" (UID: "97dab90b-6dee-49c5-8fc2-ef1a6061931e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:50:57.075100 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.075069 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97dab90b-6dee-49c5-8fc2-ef1a6061931e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "97dab90b-6dee-49c5-8fc2-ef1a6061931e" (UID: "97dab90b-6dee-49c5-8fc2-ef1a6061931e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:50:57.075194 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.075131 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97dab90b-6dee-49c5-8fc2-ef1a6061931e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "97dab90b-6dee-49c5-8fc2-ef1a6061931e" (UID: "97dab90b-6dee-49c5-8fc2-ef1a6061931e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:50:57.075194 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.075149 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97dab90b-6dee-49c5-8fc2-ef1a6061931e-kube-api-access-hmslq" (OuterVolumeSpecName: "kube-api-access-hmslq") pod "97dab90b-6dee-49c5-8fc2-ef1a6061931e" (UID: "97dab90b-6dee-49c5-8fc2-ef1a6061931e"). InnerVolumeSpecName "kube-api-access-hmslq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:50:57.174221 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.174186 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/97dab90b-6dee-49c5-8fc2-ef1a6061931e-console-oauth-config\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:50:57.174221 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.174214 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/97dab90b-6dee-49c5-8fc2-ef1a6061931e-oauth-serving-cert\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:50:57.174399 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.174229 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmslq\" (UniqueName: \"kubernetes.io/projected/97dab90b-6dee-49c5-8fc2-ef1a6061931e-kube-api-access-hmslq\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:50:57.174399 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.174242 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/97dab90b-6dee-49c5-8fc2-ef1a6061931e-console-config\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:50:57.174399 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.174256 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97dab90b-6dee-49c5-8fc2-ef1a6061931e-trusted-ca-bundle\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:50:57.174399 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.174268 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/97dab90b-6dee-49c5-8fc2-ef1a6061931e-console-serving-cert\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:50:57.804444 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.804409 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f2qf" event={"ID":"0d3ad447-6e87-43dd-bd9b-82847c08dd5e","Type":"ContainerStarted","Data":"7d7e80a0a8aff56b3a06b78b3cb728dc2c12459fe437a69ca0be1e23c58066c4"} Jul 06 16:50:57.805651 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.805629 2574 generic.go:356] "Generic (PLEG): container finished" podID="97dab90b-6dee-49c5-8fc2-ef1a6061931e" containerID="4aa73770735319b227346ce44cdcdf104e148eafb85ae1cf0e623c27b86d6d31" exitCode=0 Jul 06 16:50:57.805730 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.805674 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-566f7ddb96-l989k" Jul 06 16:50:57.805764 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.805721 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-566f7ddb96-l989k" event={"ID":"97dab90b-6dee-49c5-8fc2-ef1a6061931e","Type":"ContainerDied","Data":"4aa73770735319b227346ce44cdcdf104e148eafb85ae1cf0e623c27b86d6d31"} Jul 06 16:50:57.805804 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.805766 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-566f7ddb96-l989k" event={"ID":"97dab90b-6dee-49c5-8fc2-ef1a6061931e","Type":"ContainerDied","Data":"c974d66089d778290db5eee1039a89e386fbac6a76cda9ff51a69ff95641bc06"} Jul 06 16:50:57.805804 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.805791 2574 scope.go:117] "RemoveContainer" containerID="4aa73770735319b227346ce44cdcdf104e148eafb85ae1cf0e623c27b86d6d31" Jul 06 16:50:57.815231 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.815211 2574 scope.go:117] "RemoveContainer" containerID="4aa73770735319b227346ce44cdcdf104e148eafb85ae1cf0e623c27b86d6d31" Jul 06 16:50:57.815497 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:50:57.815473 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aa73770735319b227346ce44cdcdf104e148eafb85ae1cf0e623c27b86d6d31\": container with ID starting with 4aa73770735319b227346ce44cdcdf104e148eafb85ae1cf0e623c27b86d6d31 not found: ID does not exist" containerID="4aa73770735319b227346ce44cdcdf104e148eafb85ae1cf0e623c27b86d6d31" Jul 06 16:50:57.815604 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.815508 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa73770735319b227346ce44cdcdf104e148eafb85ae1cf0e623c27b86d6d31"} err="failed to get container status \"4aa73770735319b227346ce44cdcdf104e148eafb85ae1cf0e623c27b86d6d31\": rpc error: code = NotFound desc = could not find container \"4aa73770735319b227346ce44cdcdf104e148eafb85ae1cf0e623c27b86d6d31\": container with ID starting with 4aa73770735319b227346ce44cdcdf104e148eafb85ae1cf0e623c27b86d6d31 not found: ID does not exist" Jul 06 16:50:57.823297 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.823275 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-566f7ddb96-l989k"] Jul 06 16:50:57.827845 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:57.827824 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-566f7ddb96-l989k"] Jul 06 16:50:59.077110 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:50:59.077075 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97dab90b-6dee-49c5-8fc2-ef1a6061931e" path="/var/lib/kubelet/pods/97dab90b-6dee-49c5-8fc2-ef1a6061931e/volumes" Jul 06 16:51:01.769318 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:01.769282 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds" Jul 06 16:51:05.795872 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:05.795842 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" Jul 06 16:51:05.796412 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:05.795923 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-bwhj8" Jul 06 16:51:07.429613 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.429576 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v7s9b"] Jul 06 16:51:07.430052 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.429919 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97dab90b-6dee-49c5-8fc2-ef1a6061931e" containerName="console" Jul 06 16:51:07.430052 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.429934 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="97dab90b-6dee-49c5-8fc2-ef1a6061931e" containerName="console" Jul 06 16:51:07.430052 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.430008 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="97dab90b-6dee-49c5-8fc2-ef1a6061931e" containerName="console" Jul 06 16:51:07.436299 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.436271 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v7s9b" Jul 06 16:51:07.457123 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.457090 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v7s9b"] Jul 06 16:51:07.463463 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.463433 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2cc2\" (UniqueName: \"kubernetes.io/projected/970f8a3e-26ec-49a7-8f29-442250b15f3a-kube-api-access-h2cc2\") pod \"kuadrant-operator-controller-manager-55467d9f7-v7s9b\" (UID: \"970f8a3e-26ec-49a7-8f29-442250b15f3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v7s9b" Jul 06 16:51:07.463648 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.463625 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/970f8a3e-26ec-49a7-8f29-442250b15f3a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-v7s9b\" (UID: \"970f8a3e-26ec-49a7-8f29-442250b15f3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v7s9b" Jul 06 16:51:07.565027 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.564987 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/970f8a3e-26ec-49a7-8f29-442250b15f3a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-v7s9b\" (UID: \"970f8a3e-26ec-49a7-8f29-442250b15f3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v7s9b" Jul 06 16:51:07.565209 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.565061 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2cc2\" (UniqueName: \"kubernetes.io/projected/970f8a3e-26ec-49a7-8f29-442250b15f3a-kube-api-access-h2cc2\") pod \"kuadrant-operator-controller-manager-55467d9f7-v7s9b\" (UID: \"970f8a3e-26ec-49a7-8f29-442250b15f3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v7s9b" Jul 06 16:51:07.565456 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.565432 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/970f8a3e-26ec-49a7-8f29-442250b15f3a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-v7s9b\" (UID: \"970f8a3e-26ec-49a7-8f29-442250b15f3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v7s9b" Jul 06 16:51:07.581214 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.581183 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2cc2\" (UniqueName: \"kubernetes.io/projected/970f8a3e-26ec-49a7-8f29-442250b15f3a-kube-api-access-h2cc2\") pod \"kuadrant-operator-controller-manager-55467d9f7-v7s9b\" (UID: \"970f8a3e-26ec-49a7-8f29-442250b15f3a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v7s9b" Jul 06 16:51:07.595440 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.595414 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v7s9b"] Jul 06 16:51:07.595738 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.595720 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v7s9b" Jul 06 16:51:07.608913 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.608886 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v7s9b"] Jul 06 16:51:07.618743 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.618569 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg"] Jul 06 16:51:07.619016 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.618791 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" podUID="b37bf77c-5dba-4d93-9b53-04a8cb0f7421" containerName="manager" containerID="cri-o://360e7893322c9224fee0f077d13da9a9f2f7e36c6d8be0662479fc6bd0ac2d47" gracePeriod=2 Jul 06 16:51:07.620805 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.620783 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gmnks"] Jul 06 16:51:07.625616 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.625594 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gmnks" Jul 06 16:51:07.632615 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.632558 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg"] Jul 06 16:51:07.633828 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:07.633792 2574 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"b37bf77c-5dba-4d93-9b53-04a8cb0f7421\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-4pbtg\": pods \"kuadrant-operator-controller-manager-8684c9ff49-4pbtg\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" Jul 06 16:51:07.636488 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:07.636450 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4pbtg\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="b37bf77c-5dba-4d93-9b53-04a8cb0f7421" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" Jul 06 16:51:07.642141 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.642119 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gmnks"] Jul 06 16:51:07.658279 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.658241 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds"] Jul 06 16:51:07.658498 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.658472 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds" podUID="dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a" containerName="manager" containerID="cri-o://85d59d74b3a62d3286540da01c74638b5f13aed540f4c8b9c4472839262ef239" gracePeriod=2 Jul 06 16:51:07.660196 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.660170 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8kc2"] Jul 06 16:51:07.660696 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.660654 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b37bf77c-5dba-4d93-9b53-04a8cb0f7421" containerName="manager" Jul 06 16:51:07.660696 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.660678 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b37bf77c-5dba-4d93-9b53-04a8cb0f7421" containerName="manager" Jul 06 16:51:07.660850 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.660750 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b37bf77c-5dba-4d93-9b53-04a8cb0f7421" containerName="manager" Jul 06 16:51:07.665624 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.665202 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8kc2" Jul 06 16:51:07.665624 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.665434 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5cqn\" (UniqueName: \"kubernetes.io/projected/5bfaeb62-e27b-4c20-925e-7b9c18e78891-kube-api-access-b5cqn\") pod \"kuadrant-operator-controller-manager-55467d9f7-gmnks\" (UID: \"5bfaeb62-e27b-4c20-925e-7b9c18e78891\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gmnks" Jul 06 16:51:07.665624 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.665479 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5bfaeb62-e27b-4c20-925e-7b9c18e78891-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-gmnks\" (UID: \"5bfaeb62-e27b-4c20-925e-7b9c18e78891\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gmnks" Jul 06 16:51:07.666195 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.666172 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gmnks"] Jul 06 16:51:07.673869 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.673758 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sln8p"] Jul 06 16:51:07.677389 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.677366 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sln8p" Jul 06 16:51:07.682429 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.681670 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8kc2"] Jul 06 16:51:07.684007 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.683986 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds"] Jul 06 16:51:07.694218 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.694194 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sln8p"] Jul 06 16:51:07.727121 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:07.727080 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-b5cqn], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gmnks" podUID="5bfaeb62-e27b-4c20-925e-7b9c18e78891" Jul 06 16:51:07.729867 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:07.729832 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4pbtg\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="b37bf77c-5dba-4d93-9b53-04a8cb0f7421" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" Jul 06 16:51:07.732011 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:07.731980 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-925ds\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds" Jul 06 16:51:07.766419 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.766391 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l727p\" (UniqueName: \"kubernetes.io/projected/4f9d7f2b-46bc-43a6-a279-87114bf05246-kube-api-access-l727p\") pod \"kuadrant-operator-controller-manager-8684c9ff49-m8kc2\" (UID: \"4f9d7f2b-46bc-43a6-a279-87114bf05246\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8kc2" Jul 06 16:51:07.766572 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.766434 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5cqn\" (UniqueName: \"kubernetes.io/projected/5bfaeb62-e27b-4c20-925e-7b9c18e78891-kube-api-access-b5cqn\") pod \"kuadrant-operator-controller-manager-55467d9f7-gmnks\" (UID: \"5bfaeb62-e27b-4c20-925e-7b9c18e78891\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gmnks" Jul 06 16:51:07.766572 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.766459 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8tw7\" (UniqueName: \"kubernetes.io/projected/5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5-kube-api-access-s8tw7\") pod \"kuadrant-operator-controller-manager-86878c769f-sln8p\" (UID: \"5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sln8p" Jul 06 16:51:07.766674 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.766577 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5bfaeb62-e27b-4c20-925e-7b9c18e78891-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-gmnks\" (UID: \"5bfaeb62-e27b-4c20-925e-7b9c18e78891\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gmnks" Jul 06 16:51:07.766674 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.766643 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f9d7f2b-46bc-43a6-a279-87114bf05246-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-m8kc2\" (UID: \"4f9d7f2b-46bc-43a6-a279-87114bf05246\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8kc2" Jul 06 16:51:07.766764 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.766674 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-sln8p\" (UID: \"5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sln8p" Jul 06 16:51:07.767035 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.767015 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5bfaeb62-e27b-4c20-925e-7b9c18e78891-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-gmnks\" (UID: \"5bfaeb62-e27b-4c20-925e-7b9c18e78891\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gmnks" Jul 06 16:51:07.775425 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.775387 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5cqn\" (UniqueName: \"kubernetes.io/projected/5bfaeb62-e27b-4c20-925e-7b9c18e78891-kube-api-access-b5cqn\") pod \"kuadrant-operator-controller-manager-55467d9f7-gmnks\" (UID: \"5bfaeb62-e27b-4c20-925e-7b9c18e78891\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gmnks" Jul 06 16:51:07.808907 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.808857 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bhhj5" Jul 06 16:51:07.811261 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:07.811227 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4pbtg\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="b37bf77c-5dba-4d93-9b53-04a8cb0f7421" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" Jul 06 16:51:07.813389 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:07.813354 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-925ds\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds" Jul 06 16:51:07.857819 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.857791 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gmnks" Jul 06 16:51:07.867223 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.867194 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l727p\" (UniqueName: \"kubernetes.io/projected/4f9d7f2b-46bc-43a6-a279-87114bf05246-kube-api-access-l727p\") pod \"kuadrant-operator-controller-manager-8684c9ff49-m8kc2\" (UID: \"4f9d7f2b-46bc-43a6-a279-87114bf05246\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8kc2" Jul 06 16:51:07.867389 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.867238 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8tw7\" (UniqueName: \"kubernetes.io/projected/5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5-kube-api-access-s8tw7\") pod \"kuadrant-operator-controller-manager-86878c769f-sln8p\" (UID: \"5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sln8p" Jul 06 16:51:07.867389 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.867287 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f9d7f2b-46bc-43a6-a279-87114bf05246-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-m8kc2\" (UID: \"4f9d7f2b-46bc-43a6-a279-87114bf05246\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8kc2" Jul 06 16:51:07.867502 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.867460 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-sln8p\" (UID: \"5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sln8p" Jul 06 16:51:07.867644 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.867620 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f9d7f2b-46bc-43a6-a279-87114bf05246-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-m8kc2\" (UID: \"4f9d7f2b-46bc-43a6-a279-87114bf05246\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8kc2" Jul 06 16:51:07.867733 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.867692 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gmnks" Jul 06 16:51:07.867936 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.867915 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-sln8p\" (UID: \"5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sln8p" Jul 06 16:51:07.876396 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.876368 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8tw7\" (UniqueName: \"kubernetes.io/projected/5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5-kube-api-access-s8tw7\") pod \"kuadrant-operator-controller-manager-86878c769f-sln8p\" (UID: \"5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sln8p" Jul 06 16:51:07.876564 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.876489 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l727p\" (UniqueName: \"kubernetes.io/projected/4f9d7f2b-46bc-43a6-a279-87114bf05246-kube-api-access-l727p\") pod \"kuadrant-operator-controller-manager-8684c9ff49-m8kc2\" (UID: \"4f9d7f2b-46bc-43a6-a279-87114bf05246\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8kc2" Jul 06 16:51:07.881591 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:07.881475 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4pbtg\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="b37bf77c-5dba-4d93-9b53-04a8cb0f7421" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" Jul 06 16:51:07.883626 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:07.883588 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-925ds\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds" Jul 06 16:51:07.888771 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:07.888743 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4pbtg\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="b37bf77c-5dba-4d93-9b53-04a8cb0f7421" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" Jul 06 16:51:07.890640 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:07.890612 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-925ds\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds" Jul 06 16:51:07.968834 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.968803 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5bfaeb62-e27b-4c20-925e-7b9c18e78891-extensions-socket-volume\") pod \"5bfaeb62-e27b-4c20-925e-7b9c18e78891\" (UID: \"5bfaeb62-e27b-4c20-925e-7b9c18e78891\") " Jul 06 16:51:07.969059 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.968871 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5cqn\" (UniqueName: \"kubernetes.io/projected/5bfaeb62-e27b-4c20-925e-7b9c18e78891-kube-api-access-b5cqn\") pod \"5bfaeb62-e27b-4c20-925e-7b9c18e78891\" (UID: \"5bfaeb62-e27b-4c20-925e-7b9c18e78891\") " Jul 06 16:51:07.969135 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.969085 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfaeb62-e27b-4c20-925e-7b9c18e78891-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5bfaeb62-e27b-4c20-925e-7b9c18e78891" (UID: "5bfaeb62-e27b-4c20-925e-7b9c18e78891"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:51:07.971192 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:07.971170 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfaeb62-e27b-4c20-925e-7b9c18e78891-kube-api-access-b5cqn" (OuterVolumeSpecName: "kube-api-access-b5cqn") pod "5bfaeb62-e27b-4c20-925e-7b9c18e78891" (UID: "5bfaeb62-e27b-4c20-925e-7b9c18e78891"). InnerVolumeSpecName "kube-api-access-b5cqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:51:08.037006 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:08.036966 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8kc2" Jul 06 16:51:08.043499 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:08.043468 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sln8p" Jul 06 16:51:08.070234 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:08.070205 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5bfaeb62-e27b-4c20-925e-7b9c18e78891-extensions-socket-volume\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:51:08.070344 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:08.070255 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5cqn\" (UniqueName: \"kubernetes.io/projected/5bfaeb62-e27b-4c20-925e-7b9c18e78891-kube-api-access-b5cqn\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:51:08.862967 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:08.862937 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gmnks" Jul 06 16:51:08.872710 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:08.872681 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4pbtg\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="b37bf77c-5dba-4d93-9b53-04a8cb0f7421" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" Jul 06 16:51:08.874701 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:08.874675 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-925ds\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds" Jul 06 16:51:08.877088 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:08.877063 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-925ds\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds" Jul 06 16:51:08.918367 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:08.918317 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gmnks"] Jul 06 16:51:08.922833 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:08.922810 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gmnks"] Jul 06 16:51:08.923147 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:08.923121 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4pbtg\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="b37bf77c-5dba-4d93-9b53-04a8cb0f7421" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" Jul 06 16:51:09.077217 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:09.077177 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bfaeb62-e27b-4c20-925e-7b9c18e78891" path="/var/lib/kubelet/pods/5bfaeb62-e27b-4c20-925e-7b9c18e78891/volumes" Jul 06 16:51:21.202837 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.202763 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sln8p"] Jul 06 16:51:21.205902 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:51:21.205863 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cef6a44_8a2e_4a92_bfef_8597e6d0b9f5.slice/crio-9fc194c49b71ce4f985e4b92779a3ca9da8d23cb1cb23b0051eb5a54ce08f11b WatchSource:0}: Error finding container 9fc194c49b71ce4f985e4b92779a3ca9da8d23cb1cb23b0051eb5a54ce08f11b: Status 404 returned error can't find the container with id 9fc194c49b71ce4f985e4b92779a3ca9da8d23cb1cb23b0051eb5a54ce08f11b Jul 06 16:51:21.207860 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.207839 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" Jul 06 16:51:21.210148 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:21.210120 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4pbtg\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="b37bf77c-5dba-4d93-9b53-04a8cb0f7421" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" Jul 06 16:51:21.211319 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.211298 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds" Jul 06 16:51:21.213413 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:21.213304 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4pbtg\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="b37bf77c-5dba-4d93-9b53-04a8cb0f7421" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" Jul 06 16:51:21.215397 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:21.215365 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-925ds\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds" Jul 06 16:51:21.230629 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.230606 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8kc2"] Jul 06 16:51:21.238956 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:51:21.238928 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9d7f2b_46bc_43a6_a279_87114bf05246.slice/crio-40c8ee83ca3c036cae151d080508615e71d13f35f322cae4cd13710c2e6e0150 WatchSource:0}: Error finding container 40c8ee83ca3c036cae151d080508615e71d13f35f322cae4cd13710c2e6e0150: Status 404 returned error can't find the container with id 40c8ee83ca3c036cae151d080508615e71d13f35f322cae4cd13710c2e6e0150 Jul 06 16:51:21.289760 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.289740 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvdcm\" (UniqueName: \"kubernetes.io/projected/dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a-kube-api-access-gvdcm\") pod \"dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a\" (UID: \"dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a\") " Jul 06 16:51:21.289900 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.289786 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b37bf77c-5dba-4d93-9b53-04a8cb0f7421-extensions-socket-volume\") pod \"b37bf77c-5dba-4d93-9b53-04a8cb0f7421\" (UID: \"b37bf77c-5dba-4d93-9b53-04a8cb0f7421\") " Jul 06 16:51:21.289900 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.289814 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgdtm\" (UniqueName: \"kubernetes.io/projected/b37bf77c-5dba-4d93-9b53-04a8cb0f7421-kube-api-access-cgdtm\") pod \"b37bf77c-5dba-4d93-9b53-04a8cb0f7421\" (UID: \"b37bf77c-5dba-4d93-9b53-04a8cb0f7421\") " Jul 06 16:51:21.290863 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.290767 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b37bf77c-5dba-4d93-9b53-04a8cb0f7421-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b37bf77c-5dba-4d93-9b53-04a8cb0f7421" (UID: "b37bf77c-5dba-4d93-9b53-04a8cb0f7421"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:51:21.292197 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.292169 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a-kube-api-access-gvdcm" (OuterVolumeSpecName: "kube-api-access-gvdcm") pod "dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a" (UID: "dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a"). InnerVolumeSpecName "kube-api-access-gvdcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:51:21.292319 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.292303 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37bf77c-5dba-4d93-9b53-04a8cb0f7421-kube-api-access-cgdtm" (OuterVolumeSpecName: "kube-api-access-cgdtm") pod "b37bf77c-5dba-4d93-9b53-04a8cb0f7421" (UID: "b37bf77c-5dba-4d93-9b53-04a8cb0f7421"). InnerVolumeSpecName "kube-api-access-cgdtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:51:21.390761 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.390724 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvdcm\" (UniqueName: \"kubernetes.io/projected/dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a-kube-api-access-gvdcm\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:51:21.390761 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.390756 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b37bf77c-5dba-4d93-9b53-04a8cb0f7421-extensions-socket-volume\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:51:21.390761 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.390765 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgdtm\" (UniqueName: \"kubernetes.io/projected/b37bf77c-5dba-4d93-9b53-04a8cb0f7421-kube-api-access-cgdtm\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:51:21.927448 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.927410 2574 generic.go:356] "Generic (PLEG): container finished" podID="b37bf77c-5dba-4d93-9b53-04a8cb0f7421" containerID="360e7893322c9224fee0f077d13da9a9f2f7e36c6d8be0662479fc6bd0ac2d47" exitCode=0 Jul 06 16:51:21.927664 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.927460 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" Jul 06 16:51:21.927664 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.927542 2574 scope.go:117] "RemoveContainer" containerID="360e7893322c9224fee0f077d13da9a9f2f7e36c6d8be0662479fc6bd0ac2d47" Jul 06 16:51:21.929093 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.929065 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f2qf" event={"ID":"0d3ad447-6e87-43dd-bd9b-82847c08dd5e","Type":"ContainerStarted","Data":"64e8875b359801dd498b9ea840d269269fb42ed5cf84b3a317530d28fccf8adb"} Jul 06 16:51:21.929955 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:21.929931 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4pbtg\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="b37bf77c-5dba-4d93-9b53-04a8cb0f7421" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" Jul 06 16:51:21.930988 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.930965 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8kc2" event={"ID":"4f9d7f2b-46bc-43a6-a279-87114bf05246","Type":"ContainerStarted","Data":"9d86d31658a58c3ca2eaf38f911f1803f4ba2f48b877351666d5b85a07bf3d8c"} Jul 06 16:51:21.931098 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.931011 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8kc2" event={"ID":"4f9d7f2b-46bc-43a6-a279-87114bf05246","Type":"ContainerStarted","Data":"40c8ee83ca3c036cae151d080508615e71d13f35f322cae4cd13710c2e6e0150"} Jul 06 16:51:21.931098 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.931066 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8kc2" Jul 06 16:51:21.931905 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:21.931881 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-925ds\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds" Jul 06 16:51:21.932375 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.932344 2574 generic.go:356] "Generic (PLEG): container finished" podID="dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a" containerID="85d59d74b3a62d3286540da01c74638b5f13aed540f4c8b9c4472839262ef239" exitCode=0 Jul 06 16:51:21.932473 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.932380 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds" Jul 06 16:51:21.934558 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:21.934129 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4pbtg\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="b37bf77c-5dba-4d93-9b53-04a8cb0f7421" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" Jul 06 16:51:21.935308 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.934734 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sln8p" event={"ID":"5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5","Type":"ContainerStarted","Data":"6b024405f60545fdb11ba14cc3c542116c57610d080f96325c5f6600205f305f"} Jul 06 16:51:21.935308 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.934784 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sln8p" event={"ID":"5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5","Type":"ContainerStarted","Data":"9fc194c49b71ce4f985e4b92779a3ca9da8d23cb1cb23b0051eb5a54ce08f11b"} Jul 06 16:51:21.935308 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.935220 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sln8p" Jul 06 16:51:21.936746 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:21.936589 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-925ds\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds" Jul 06 16:51:21.941393 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.941375 2574 scope.go:117] "RemoveContainer" containerID="360e7893322c9224fee0f077d13da9a9f2f7e36c6d8be0662479fc6bd0ac2d47" Jul 06 16:51:21.941841 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:21.941816 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"360e7893322c9224fee0f077d13da9a9f2f7e36c6d8be0662479fc6bd0ac2d47\": container with ID starting with 360e7893322c9224fee0f077d13da9a9f2f7e36c6d8be0662479fc6bd0ac2d47 not found: ID does not exist" containerID="360e7893322c9224fee0f077d13da9a9f2f7e36c6d8be0662479fc6bd0ac2d47" Jul 06 16:51:21.941920 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.941845 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"360e7893322c9224fee0f077d13da9a9f2f7e36c6d8be0662479fc6bd0ac2d47"} err="failed to get container status \"360e7893322c9224fee0f077d13da9a9f2f7e36c6d8be0662479fc6bd0ac2d47\": rpc error: code = NotFound desc = could not find container \"360e7893322c9224fee0f077d13da9a9f2f7e36c6d8be0662479fc6bd0ac2d47\": container with ID starting with 360e7893322c9224fee0f077d13da9a9f2f7e36c6d8be0662479fc6bd0ac2d47 not found: ID does not exist" Jul 06 16:51:21.941920 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.941866 2574 scope.go:117] "RemoveContainer" containerID="85d59d74b3a62d3286540da01c74638b5f13aed540f4c8b9c4472839262ef239" Jul 06 16:51:21.949975 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.949958 2574 scope.go:117] "RemoveContainer" containerID="85d59d74b3a62d3286540da01c74638b5f13aed540f4c8b9c4472839262ef239" Jul 06 16:51:21.950235 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:21.950217 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85d59d74b3a62d3286540da01c74638b5f13aed540f4c8b9c4472839262ef239\": container with ID starting with 85d59d74b3a62d3286540da01c74638b5f13aed540f4c8b9c4472839262ef239 not found: ID does not exist" containerID="85d59d74b3a62d3286540da01c74638b5f13aed540f4c8b9c4472839262ef239" Jul 06 16:51:21.950282 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.950247 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85d59d74b3a62d3286540da01c74638b5f13aed540f4c8b9c4472839262ef239"} err="failed to get container status \"85d59d74b3a62d3286540da01c74638b5f13aed540f4c8b9c4472839262ef239\": rpc error: code = NotFound desc = could not find container \"85d59d74b3a62d3286540da01c74638b5f13aed540f4c8b9c4472839262ef239\": container with ID starting with 85d59d74b3a62d3286540da01c74638b5f13aed540f4c8b9c4472839262ef239 not found: ID does not exist" Jul 06 16:51:21.952892 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.952856 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8f2qf" podStartSLOduration=1.7405909419999999 podStartE2EDuration="25.952845719s" totalImagesPullingTime="24.212254777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:50:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:51:21.951910685 +0000 UTC m=+775.471401800" watchObservedRunningTime="2026-07-06 16:51:21.952845719 +0000 UTC m=+775.472336834" Jul 06 16:51:21.977166 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:21.977108 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8kc2" podStartSLOduration=14.977087264 podStartE2EDuration="14.977087264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:51:21.975941847 +0000 UTC m=+775.495432962" watchObservedRunningTime="2026-07-06 16:51:21.977087264 +0000 UTC m=+775.496578382" Jul 06 16:51:21.977846 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:21.977819 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-4pbtg\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="b37bf77c-5dba-4d93-9b53-04a8cb0f7421" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-4pbtg" Jul 06 16:51:21.979702 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:21.979673 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-925ds\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-925ds" Jul 06 16:51:22.000441 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:22.000396 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sln8p" podStartSLOduration=15.000379826 podStartE2EDuration="15.000379826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:51:21.999032877 +0000 UTC m=+775.518523995" watchObservedRunningTime="2026-07-06 16:51:22.000379826 +0000 UTC m=+775.519870942" Jul 06 16:51:23.075968 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:23.075935 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b37bf77c-5dba-4d93-9b53-04a8cb0f7421" path="/var/lib/kubelet/pods/b37bf77c-5dba-4d93-9b53-04a8cb0f7421/volumes" Jul 06 16:51:23.076348 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:23.076232 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a" path="/var/lib/kubelet/pods/dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a/volumes" Jul 06 16:51:23.576577 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:23.576496 2574 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-v7s9b_kuadrant-system_970f8a3e-26ec-49a7-8f29-442250b15f3a_0(c86a6ee77126fe8541d9e8e7aaec4e1a9875e4537e28f96e50f01f7b4eda4549): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-v7s9b to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 06 16:51:23.576653 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:23.576603 2574 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-v7s9b_kuadrant-system_970f8a3e-26ec-49a7-8f29-442250b15f3a_0(c86a6ee77126fe8541d9e8e7aaec4e1a9875e4537e28f96e50f01f7b4eda4549): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-v7s9b to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v7s9b" Jul 06 16:51:23.946738 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:23.946702 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v7s9b" Jul 06 16:51:23.949014 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:23.948976 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-v7s9b\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="970f8a3e-26ec-49a7-8f29-442250b15f3a" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v7s9b" Jul 06 16:51:24.016359 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:24.016328 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v7s9b" Jul 06 16:51:24.018787 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:24.018762 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-v7s9b\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="970f8a3e-26ec-49a7-8f29-442250b15f3a" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v7s9b" Jul 06 16:51:24.113372 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:24.113339 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/970f8a3e-26ec-49a7-8f29-442250b15f3a-extensions-socket-volume\") pod \"970f8a3e-26ec-49a7-8f29-442250b15f3a\" (UID: \"970f8a3e-26ec-49a7-8f29-442250b15f3a\") " Jul 06 16:51:24.113818 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:24.113412 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2cc2\" (UniqueName: \"kubernetes.io/projected/970f8a3e-26ec-49a7-8f29-442250b15f3a-kube-api-access-h2cc2\") pod \"970f8a3e-26ec-49a7-8f29-442250b15f3a\" (UID: \"970f8a3e-26ec-49a7-8f29-442250b15f3a\") " Jul 06 16:51:24.113818 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:24.113614 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/970f8a3e-26ec-49a7-8f29-442250b15f3a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "970f8a3e-26ec-49a7-8f29-442250b15f3a" (UID: "970f8a3e-26ec-49a7-8f29-442250b15f3a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:51:24.113818 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:24.113692 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/970f8a3e-26ec-49a7-8f29-442250b15f3a-extensions-socket-volume\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:51:24.115409 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:24.115387 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/970f8a3e-26ec-49a7-8f29-442250b15f3a-kube-api-access-h2cc2" (OuterVolumeSpecName: "kube-api-access-h2cc2") pod "970f8a3e-26ec-49a7-8f29-442250b15f3a" (UID: "970f8a3e-26ec-49a7-8f29-442250b15f3a"). InnerVolumeSpecName "kube-api-access-h2cc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:51:24.214988 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:24.214915 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2cc2\" (UniqueName: \"kubernetes.io/projected/970f8a3e-26ec-49a7-8f29-442250b15f3a-kube-api-access-h2cc2\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:51:24.951240 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:24.951210 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v7s9b" Jul 06 16:51:24.953940 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:24.953912 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-v7s9b\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="970f8a3e-26ec-49a7-8f29-442250b15f3a" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v7s9b" Jul 06 16:51:24.959554 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:24.959506 2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-v7s9b\" is forbidden: User \"system:node:ip-10-0-131-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-71.ec2.internal' and this object" podUID="970f8a3e-26ec-49a7-8f29-442250b15f3a" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-v7s9b" Jul 06 16:51:25.076111 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:25.076079 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="970f8a3e-26ec-49a7-8f29-442250b15f3a" path="/var/lib/kubelet/pods/970f8a3e-26ec-49a7-8f29-442250b15f3a/volumes" Jul 06 16:51:32.944005 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:32.943977 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8kc2" Jul 06 16:51:33.949466 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:33.949433 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sln8p" Jul 06 16:51:34.005366 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:34.005337 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8kc2"] Jul 06 16:51:34.005564 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:34.005538 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8kc2" podUID="4f9d7f2b-46bc-43a6-a279-87114bf05246" containerName="manager" containerID="cri-o://9d86d31658a58c3ca2eaf38f911f1803f4ba2f48b877351666d5b85a07bf3d8c" gracePeriod=10 Jul 06 16:51:34.752571 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:34.752548 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8kc2" Jul 06 16:51:34.899819 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:34.899738 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f9d7f2b-46bc-43a6-a279-87114bf05246-extensions-socket-volume\") pod \"4f9d7f2b-46bc-43a6-a279-87114bf05246\" (UID: \"4f9d7f2b-46bc-43a6-a279-87114bf05246\") " Jul 06 16:51:34.899819 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:34.899788 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l727p\" (UniqueName: \"kubernetes.io/projected/4f9d7f2b-46bc-43a6-a279-87114bf05246-kube-api-access-l727p\") pod \"4f9d7f2b-46bc-43a6-a279-87114bf05246\" (UID: \"4f9d7f2b-46bc-43a6-a279-87114bf05246\") " Jul 06 16:51:34.900174 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:34.900151 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9d7f2b-46bc-43a6-a279-87114bf05246-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4f9d7f2b-46bc-43a6-a279-87114bf05246" (UID: "4f9d7f2b-46bc-43a6-a279-87114bf05246"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:51:34.901860 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:34.901836 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f9d7f2b-46bc-43a6-a279-87114bf05246-kube-api-access-l727p" (OuterVolumeSpecName: "kube-api-access-l727p") pod "4f9d7f2b-46bc-43a6-a279-87114bf05246" (UID: "4f9d7f2b-46bc-43a6-a279-87114bf05246"). InnerVolumeSpecName "kube-api-access-l727p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:51:34.996281 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:34.996249 2574 generic.go:356] "Generic (PLEG): container finished" podID="4f9d7f2b-46bc-43a6-a279-87114bf05246" containerID="9d86d31658a58c3ca2eaf38f911f1803f4ba2f48b877351666d5b85a07bf3d8c" exitCode=0 Jul 06 16:51:34.996716 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:34.996294 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8kc2" Jul 06 16:51:34.996716 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:34.996321 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8kc2" event={"ID":"4f9d7f2b-46bc-43a6-a279-87114bf05246","Type":"ContainerDied","Data":"9d86d31658a58c3ca2eaf38f911f1803f4ba2f48b877351666d5b85a07bf3d8c"} Jul 06 16:51:34.996716 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:34.996390 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8kc2" event={"ID":"4f9d7f2b-46bc-43a6-a279-87114bf05246","Type":"ContainerDied","Data":"40c8ee83ca3c036cae151d080508615e71d13f35f322cae4cd13710c2e6e0150"} Jul 06 16:51:34.996716 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:34.996411 2574 scope.go:117] "RemoveContainer" containerID="9d86d31658a58c3ca2eaf38f911f1803f4ba2f48b877351666d5b85a07bf3d8c" Jul 06 16:51:35.000431 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:35.000405 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f9d7f2b-46bc-43a6-a279-87114bf05246-extensions-socket-volume\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:51:35.000431 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:35.000435 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l727p\" (UniqueName: \"kubernetes.io/projected/4f9d7f2b-46bc-43a6-a279-87114bf05246-kube-api-access-l727p\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:51:35.005318 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:35.005301 2574 scope.go:117] "RemoveContainer" containerID="9d86d31658a58c3ca2eaf38f911f1803f4ba2f48b877351666d5b85a07bf3d8c" Jul 06 16:51:35.005623 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:51:35.005602 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d86d31658a58c3ca2eaf38f911f1803f4ba2f48b877351666d5b85a07bf3d8c\": container with ID starting with 9d86d31658a58c3ca2eaf38f911f1803f4ba2f48b877351666d5b85a07bf3d8c not found: ID does not exist" containerID="9d86d31658a58c3ca2eaf38f911f1803f4ba2f48b877351666d5b85a07bf3d8c" Jul 06 16:51:35.005713 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:35.005634 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d86d31658a58c3ca2eaf38f911f1803f4ba2f48b877351666d5b85a07bf3d8c"} err="failed to get container status \"9d86d31658a58c3ca2eaf38f911f1803f4ba2f48b877351666d5b85a07bf3d8c\": rpc error: code = NotFound desc = could not find container \"9d86d31658a58c3ca2eaf38f911f1803f4ba2f48b877351666d5b85a07bf3d8c\": container with ID starting with 9d86d31658a58c3ca2eaf38f911f1803f4ba2f48b877351666d5b85a07bf3d8c not found: ID does not exist" Jul 06 16:51:35.022685 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:35.022661 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8kc2"] Jul 06 16:51:35.026974 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:35.026952 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-m8kc2"] Jul 06 16:51:35.075718 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:35.075694 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f9d7f2b-46bc-43a6-a279-87114bf05246" path="/var/lib/kubelet/pods/4f9d7f2b-46bc-43a6-a279-87114bf05246/volumes" Jul 06 16:51:50.357334 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.357252 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82"] Jul 06 16:51:50.357732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.357627 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a" containerName="manager" Jul 06 16:51:50.357732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.357641 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a" containerName="manager" Jul 06 16:51:50.357732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.357658 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f9d7f2b-46bc-43a6-a279-87114bf05246" containerName="manager" Jul 06 16:51:50.357732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.357663 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f9d7f2b-46bc-43a6-a279-87114bf05246" containerName="manager" Jul 06 16:51:50.357732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.357733 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbab875d-7e8a-4a9d-94fc-5de4ffcbd30a" containerName="manager" Jul 06 16:51:50.357884 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.357740 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f9d7f2b-46bc-43a6-a279-87114bf05246" containerName="manager" Jul 06 16:51:50.450025 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.449994 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82"] Jul 06 16:51:50.450199 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.450083 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.454016 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.453989 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-7gqrl\"" Jul 06 16:51:50.454254 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.454235 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 16:51:50.627651 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.627546 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccabcba-7d81-48aa-919f-150b01a4959c-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqc82\" (UID: \"9ccabcba-7d81-48aa-919f-150b01a4959c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.627651 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.627587 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccabcba-7d81-48aa-919f-150b01a4959c-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqc82\" (UID: \"9ccabcba-7d81-48aa-919f-150b01a4959c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.627651 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.627609 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ccabcba-7d81-48aa-919f-150b01a4959c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqc82\" (UID: \"9ccabcba-7d81-48aa-919f-150b01a4959c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.627651 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.627635 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ccabcba-7d81-48aa-919f-150b01a4959c-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqc82\" (UID: \"9ccabcba-7d81-48aa-919f-150b01a4959c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.627651 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.627654 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ccabcba-7d81-48aa-919f-150b01a4959c-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqc82\" (UID: \"9ccabcba-7d81-48aa-919f-150b01a4959c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.628063 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.627706 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ccabcba-7d81-48aa-919f-150b01a4959c-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqc82\" (UID: \"9ccabcba-7d81-48aa-919f-150b01a4959c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.628063 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.627802 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r496c\" (UniqueName: \"kubernetes.io/projected/9ccabcba-7d81-48aa-919f-150b01a4959c-kube-api-access-r496c\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqc82\" (UID: \"9ccabcba-7d81-48aa-919f-150b01a4959c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.628063 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.627847 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ccabcba-7d81-48aa-919f-150b01a4959c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqc82\" (UID: \"9ccabcba-7d81-48aa-919f-150b01a4959c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.628063 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.627881 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ccabcba-7d81-48aa-919f-150b01a4959c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqc82\" (UID: \"9ccabcba-7d81-48aa-919f-150b01a4959c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.729027 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.728991 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccabcba-7d81-48aa-919f-150b01a4959c-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqc82\" (UID: \"9ccabcba-7d81-48aa-919f-150b01a4959c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.729027 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.729028 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccabcba-7d81-48aa-919f-150b01a4959c-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqc82\" (UID: \"9ccabcba-7d81-48aa-919f-150b01a4959c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.729281 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.729092 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ccabcba-7d81-48aa-919f-150b01a4959c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqc82\" (UID: \"9ccabcba-7d81-48aa-919f-150b01a4959c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.729281 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.729144 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ccabcba-7d81-48aa-919f-150b01a4959c-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqc82\" (UID: \"9ccabcba-7d81-48aa-919f-150b01a4959c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.729281 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.729177 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ccabcba-7d81-48aa-919f-150b01a4959c-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqc82\" (UID: \"9ccabcba-7d81-48aa-919f-150b01a4959c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.729281 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.729204 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ccabcba-7d81-48aa-919f-150b01a4959c-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqc82\" (UID: \"9ccabcba-7d81-48aa-919f-150b01a4959c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.729281 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.729258 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r496c\" (UniqueName: \"kubernetes.io/projected/9ccabcba-7d81-48aa-919f-150b01a4959c-kube-api-access-r496c\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqc82\" (UID: \"9ccabcba-7d81-48aa-919f-150b01a4959c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.729575 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.729285 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ccabcba-7d81-48aa-919f-150b01a4959c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqc82\" (UID: \"9ccabcba-7d81-48aa-919f-150b01a4959c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.729575 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.729324 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ccabcba-7d81-48aa-919f-150b01a4959c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqc82\" (UID: \"9ccabcba-7d81-48aa-919f-150b01a4959c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.729575 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.729485 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccabcba-7d81-48aa-919f-150b01a4959c-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqc82\" (UID: \"9ccabcba-7d81-48aa-919f-150b01a4959c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.729771 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.729670 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ccabcba-7d81-48aa-919f-150b01a4959c-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqc82\" (UID: \"9ccabcba-7d81-48aa-919f-150b01a4959c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.729836 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.729792 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ccabcba-7d81-48aa-919f-150b01a4959c-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqc82\" (UID: \"9ccabcba-7d81-48aa-919f-150b01a4959c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.729890 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.729836 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ccabcba-7d81-48aa-919f-150b01a4959c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqc82\" (UID: \"9ccabcba-7d81-48aa-919f-150b01a4959c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.729954 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.729934 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ccabcba-7d81-48aa-919f-150b01a4959c-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqc82\" (UID: \"9ccabcba-7d81-48aa-919f-150b01a4959c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.731809 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.731786 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ccabcba-7d81-48aa-919f-150b01a4959c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqc82\" (UID: \"9ccabcba-7d81-48aa-919f-150b01a4959c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.731913 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.731891 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ccabcba-7d81-48aa-919f-150b01a4959c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqc82\" (UID: \"9ccabcba-7d81-48aa-919f-150b01a4959c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.741440 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.741421 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ccabcba-7d81-48aa-919f-150b01a4959c-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqc82\" (UID: \"9ccabcba-7d81-48aa-919f-150b01a4959c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.744439 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.744418 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r496c\" (UniqueName: \"kubernetes.io/projected/9ccabcba-7d81-48aa-919f-150b01a4959c-kube-api-access-r496c\") pod \"maas-default-gateway-openshift-default-5896d4dd49-mqc82\" (UID: \"9ccabcba-7d81-48aa-919f-150b01a4959c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.763344 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.763322 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:50.970286 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:50.970259 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82"] Jul 06 16:51:50.972893 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:51:50.972864 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ccabcba_7d81_48aa_919f_150b01a4959c.slice/crio-bb995d53fde22bd42afc334bc303c94962be76a6390594e387ccd294da9c3ac3 WatchSource:0}: Error finding container bb995d53fde22bd42afc334bc303c94962be76a6390594e387ccd294da9c3ac3: Status 404 returned error can't find the container with id bb995d53fde22bd42afc334bc303c94962be76a6390594e387ccd294da9c3ac3 Jul 06 16:51:51.077046 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:51.077016 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" event={"ID":"9ccabcba-7d81-48aa-919f-150b01a4959c","Type":"ContainerStarted","Data":"bb995d53fde22bd42afc334bc303c94962be76a6390594e387ccd294da9c3ac3"} Jul 06 16:51:54.392295 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:54.392259 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 16:51:54.392706 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:54.392344 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 16:51:54.392706 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:54.392394 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 16:51:55.096721 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:55.096680 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" event={"ID":"9ccabcba-7d81-48aa-919f-150b01a4959c","Type":"ContainerStarted","Data":"9ee6acf2901095753d1ab40b5f7fd5f831ddd1f719ff279def90ab75486a6427"} Jul 06 16:51:55.119964 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:55.119915 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" podStartSLOduration=1.702820525 podStartE2EDuration="5.119898265s" totalImagesPullingTime="3.41707774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:51:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:51:55.117949551 +0000 UTC m=+808.637440667" watchObservedRunningTime="2026-07-06 16:51:55.119898265 +0000 UTC m=+808.639389436" Jul 06 16:51:55.764134 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:55.764101 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:55.768563 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:55.768542 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:56.100920 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:56.100837 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:56.101689 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:56.101671 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:51:56.315653 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:56.315620 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vdv9j"] Jul 06 16:51:56.319105 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:56.319084 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vdv9j" Jul 06 16:51:56.322405 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:56.322383 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 06 16:51:56.330072 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:56.330050 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vdv9j"] Jul 06 16:51:56.377880 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:56.377813 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/84cfadcd-be57-40b4-b9c9-8a12aaabfa24-config-file\") pod \"limitador-limitador-64c94fd64f-vdv9j\" (UID: \"84cfadcd-be57-40b4-b9c9-8a12aaabfa24\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vdv9j" Jul 06 16:51:56.378015 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:56.377900 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtsvr\" (UniqueName: \"kubernetes.io/projected/84cfadcd-be57-40b4-b9c9-8a12aaabfa24-kube-api-access-wtsvr\") pod \"limitador-limitador-64c94fd64f-vdv9j\" (UID: \"84cfadcd-be57-40b4-b9c9-8a12aaabfa24\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vdv9j" Jul 06 16:51:56.478506 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:56.478477 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/84cfadcd-be57-40b4-b9c9-8a12aaabfa24-config-file\") pod \"limitador-limitador-64c94fd64f-vdv9j\" (UID: \"84cfadcd-be57-40b4-b9c9-8a12aaabfa24\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vdv9j" Jul 06 16:51:56.478694 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:56.478563 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtsvr\" (UniqueName: \"kubernetes.io/projected/84cfadcd-be57-40b4-b9c9-8a12aaabfa24-kube-api-access-wtsvr\") pod \"limitador-limitador-64c94fd64f-vdv9j\" (UID: \"84cfadcd-be57-40b4-b9c9-8a12aaabfa24\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vdv9j" Jul 06 16:51:56.479132 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:56.479108 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/84cfadcd-be57-40b4-b9c9-8a12aaabfa24-config-file\") pod \"limitador-limitador-64c94fd64f-vdv9j\" (UID: \"84cfadcd-be57-40b4-b9c9-8a12aaabfa24\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vdv9j" Jul 06 16:51:56.501143 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:56.501115 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vdv9j"] Jul 06 16:51:56.502875 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:56.502854 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtsvr\" (UniqueName: \"kubernetes.io/projected/84cfadcd-be57-40b4-b9c9-8a12aaabfa24-kube-api-access-wtsvr\") pod \"limitador-limitador-64c94fd64f-vdv9j\" (UID: \"84cfadcd-be57-40b4-b9c9-8a12aaabfa24\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vdv9j" Jul 06 16:51:56.629899 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:56.629800 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vdv9j" Jul 06 16:51:56.773140 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:56.773102 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vdv9j"] Jul 06 16:51:56.776463 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:51:56.776431 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84cfadcd_be57_40b4_b9c9_8a12aaabfa24.slice/crio-89639c345990c110c9f2393847e59f53f4d582807f0d0e209d89f0763639e228 WatchSource:0}: Error finding container 89639c345990c110c9f2393847e59f53f4d582807f0d0e209d89f0763639e228: Status 404 returned error can't find the container with id 89639c345990c110c9f2393847e59f53f4d582807f0d0e209d89f0763639e228 Jul 06 16:51:56.918136 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:56.918060 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 16:51:56.924831 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:56.924799 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-q294l" Jul 06 16:51:56.934447 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:56.934419 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 16:51:56.954904 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:56.954871 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 16:51:56.983886 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:56.983852 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9e689a68-5912-40d5-a37f-b9d8d37ccd47-config-file\") pod \"limitador-limitador-54d6d988d5-q294l\" (UID: \"9e689a68-5912-40d5-a37f-b9d8d37ccd47\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q294l" Jul 06 16:51:56.984019 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:56.983895 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vq5g\" (UniqueName: \"kubernetes.io/projected/9e689a68-5912-40d5-a37f-b9d8d37ccd47-kube-api-access-7vq5g\") pod \"limitador-limitador-54d6d988d5-q294l\" (UID: \"9e689a68-5912-40d5-a37f-b9d8d37ccd47\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q294l" Jul 06 16:51:57.084799 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:57.084768 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9e689a68-5912-40d5-a37f-b9d8d37ccd47-config-file\") pod \"limitador-limitador-54d6d988d5-q294l\" (UID: \"9e689a68-5912-40d5-a37f-b9d8d37ccd47\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q294l" Jul 06 16:51:57.084957 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:57.084815 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vq5g\" (UniqueName: \"kubernetes.io/projected/9e689a68-5912-40d5-a37f-b9d8d37ccd47-kube-api-access-7vq5g\") pod \"limitador-limitador-54d6d988d5-q294l\" (UID: \"9e689a68-5912-40d5-a37f-b9d8d37ccd47\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q294l" Jul 06 16:51:57.085386 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:57.085364 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9e689a68-5912-40d5-a37f-b9d8d37ccd47-config-file\") pod \"limitador-limitador-54d6d988d5-q294l\" (UID: \"9e689a68-5912-40d5-a37f-b9d8d37ccd47\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q294l" Jul 06 16:51:57.094807 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:57.094779 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vq5g\" (UniqueName: \"kubernetes.io/projected/9e689a68-5912-40d5-a37f-b9d8d37ccd47-kube-api-access-7vq5g\") pod \"limitador-limitador-54d6d988d5-q294l\" (UID: \"9e689a68-5912-40d5-a37f-b9d8d37ccd47\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-q294l" Jul 06 16:51:57.106368 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:57.106343 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vdv9j" event={"ID":"84cfadcd-be57-40b4-b9c9-8a12aaabfa24","Type":"ContainerStarted","Data":"89639c345990c110c9f2393847e59f53f4d582807f0d0e209d89f0763639e228"} Jul 06 16:51:57.235621 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:57.235584 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-q294l" Jul 06 16:51:57.371437 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:57.371373 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 16:51:57.373698 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:51:57.373675 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e689a68_5912_40d5_a37f_b9d8d37ccd47.slice/crio-eafe2d0803a745b753f0567e2d60403acbf0e577648a554cfbc51804e3c9446f WatchSource:0}: Error finding container eafe2d0803a745b753f0567e2d60403acbf0e577648a554cfbc51804e3c9446f: Status 404 returned error can't find the container with id eafe2d0803a745b753f0567e2d60403acbf0e577648a554cfbc51804e3c9446f Jul 06 16:51:58.118140 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:51:58.118088 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-q294l" event={"ID":"9e689a68-5912-40d5-a37f-b9d8d37ccd47","Type":"ContainerStarted","Data":"eafe2d0803a745b753f0567e2d60403acbf0e577648a554cfbc51804e3c9446f"} Jul 06 16:52:00.129957 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:00.129864 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vdv9j" event={"ID":"84cfadcd-be57-40b4-b9c9-8a12aaabfa24","Type":"ContainerStarted","Data":"3fb104eeee59a01a91e8ab3b805833db510399c83a411b52a9790b7e9502d274"} Jul 06 16:52:00.129957 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:00.129918 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-vdv9j" Jul 06 16:52:00.137109 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:00.137069 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-q294l" event={"ID":"9e689a68-5912-40d5-a37f-b9d8d37ccd47","Type":"ContainerStarted","Data":"a3a807bcea3335527d97c51a9c31cc40995a01740229c10e021baff1d4432ed7"} Jul 06 16:52:00.137244 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:00.137162 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-q294l" Jul 06 16:52:00.150331 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:00.150284 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-vdv9j" podStartSLOduration=1.195820615 podStartE2EDuration="4.150268912s" totalImagesPullingTime="2.954448297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:51:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:52:00.149542835 +0000 UTC m=+813.669033942" watchObservedRunningTime="2026-07-06 16:52:00.150268912 +0000 UTC m=+813.669760053" Jul 06 16:52:00.171288 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:00.171233 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-q294l" podStartSLOduration=1.8030350080000002 podStartE2EDuration="4.171215467s" totalImagesPullingTime="2.368180459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:51:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:52:00.170117991 +0000 UTC m=+813.689609107" watchObservedRunningTime="2026-07-06 16:52:00.171215467 +0000 UTC m=+813.690706584" Jul 06 16:52:11.142570 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:11.142541 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-vdv9j" Jul 06 16:52:11.142982 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:11.142608 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-q294l" Jul 06 16:52:11.271834 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:11.271804 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vdv9j"] Jul 06 16:52:11.272026 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:11.272005 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-vdv9j" podUID="84cfadcd-be57-40b4-b9c9-8a12aaabfa24" containerName="limitador" containerID="cri-o://3fb104eeee59a01a91e8ab3b805833db510399c83a411b52a9790b7e9502d274" gracePeriod=30 Jul 06 16:52:11.826503 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:11.826478 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vdv9j" Jul 06 16:52:11.913854 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:11.913766 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtsvr\" (UniqueName: \"kubernetes.io/projected/84cfadcd-be57-40b4-b9c9-8a12aaabfa24-kube-api-access-wtsvr\") pod \"84cfadcd-be57-40b4-b9c9-8a12aaabfa24\" (UID: \"84cfadcd-be57-40b4-b9c9-8a12aaabfa24\") " Jul 06 16:52:11.913854 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:11.913806 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/84cfadcd-be57-40b4-b9c9-8a12aaabfa24-config-file\") pod \"84cfadcd-be57-40b4-b9c9-8a12aaabfa24\" (UID: \"84cfadcd-be57-40b4-b9c9-8a12aaabfa24\") " Jul 06 16:52:11.914172 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:11.914150 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84cfadcd-be57-40b4-b9c9-8a12aaabfa24-config-file" (OuterVolumeSpecName: "config-file") pod "84cfadcd-be57-40b4-b9c9-8a12aaabfa24" (UID: "84cfadcd-be57-40b4-b9c9-8a12aaabfa24"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:52:11.915871 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:11.915853 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84cfadcd-be57-40b4-b9c9-8a12aaabfa24-kube-api-access-wtsvr" (OuterVolumeSpecName: "kube-api-access-wtsvr") pod "84cfadcd-be57-40b4-b9c9-8a12aaabfa24" (UID: "84cfadcd-be57-40b4-b9c9-8a12aaabfa24"). InnerVolumeSpecName "kube-api-access-wtsvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:52:12.014828 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:12.014792 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtsvr\" (UniqueName: \"kubernetes.io/projected/84cfadcd-be57-40b4-b9c9-8a12aaabfa24-kube-api-access-wtsvr\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:52:12.014828 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:12.014825 2574 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/84cfadcd-be57-40b4-b9c9-8a12aaabfa24-config-file\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 16:52:12.189157 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:12.189071 2574 generic.go:356] "Generic (PLEG): container finished" podID="84cfadcd-be57-40b4-b9c9-8a12aaabfa24" containerID="3fb104eeee59a01a91e8ab3b805833db510399c83a411b52a9790b7e9502d274" exitCode=0 Jul 06 16:52:12.189157 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:12.189110 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vdv9j" Jul 06 16:52:12.189157 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:12.189136 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vdv9j" event={"ID":"84cfadcd-be57-40b4-b9c9-8a12aaabfa24","Type":"ContainerDied","Data":"3fb104eeee59a01a91e8ab3b805833db510399c83a411b52a9790b7e9502d274"} Jul 06 16:52:12.189623 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:12.189183 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vdv9j" event={"ID":"84cfadcd-be57-40b4-b9c9-8a12aaabfa24","Type":"ContainerDied","Data":"89639c345990c110c9f2393847e59f53f4d582807f0d0e209d89f0763639e228"} Jul 06 16:52:12.189623 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:12.189206 2574 scope.go:117] "RemoveContainer" containerID="3fb104eeee59a01a91e8ab3b805833db510399c83a411b52a9790b7e9502d274" Jul 06 16:52:12.197424 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:12.197405 2574 scope.go:117] "RemoveContainer" containerID="3fb104eeee59a01a91e8ab3b805833db510399c83a411b52a9790b7e9502d274" Jul 06 16:52:12.197723 ip-10-0-131-71 kubenswrapper[2574]: E0706 16:52:12.197705 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fb104eeee59a01a91e8ab3b805833db510399c83a411b52a9790b7e9502d274\": container with ID starting with 3fb104eeee59a01a91e8ab3b805833db510399c83a411b52a9790b7e9502d274 not found: ID does not exist" containerID="3fb104eeee59a01a91e8ab3b805833db510399c83a411b52a9790b7e9502d274" Jul 06 16:52:12.197796 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:12.197733 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fb104eeee59a01a91e8ab3b805833db510399c83a411b52a9790b7e9502d274"} err="failed to get container status \"3fb104eeee59a01a91e8ab3b805833db510399c83a411b52a9790b7e9502d274\": rpc error: code = NotFound desc = could not find container \"3fb104eeee59a01a91e8ab3b805833db510399c83a411b52a9790b7e9502d274\": container with ID starting with 3fb104eeee59a01a91e8ab3b805833db510399c83a411b52a9790b7e9502d274 not found: ID does not exist" Jul 06 16:52:12.218407 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:12.218380 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vdv9j"] Jul 06 16:52:12.227443 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:12.227420 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vdv9j"] Jul 06 16:52:13.075975 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:13.075941 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84cfadcd-be57-40b4-b9c9-8a12aaabfa24" path="/var/lib/kubelet/pods/84cfadcd-be57-40b4-b9c9-8a12aaabfa24/volumes" Jul 06 16:52:17.751612 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:17.751577 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-ndsgq"] Jul 06 16:52:17.752077 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:17.751935 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84cfadcd-be57-40b4-b9c9-8a12aaabfa24" containerName="limitador" Jul 06 16:52:17.752077 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:17.751948 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="84cfadcd-be57-40b4-b9c9-8a12aaabfa24" containerName="limitador" Jul 06 16:52:17.752077 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:17.752017 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="84cfadcd-be57-40b4-b9c9-8a12aaabfa24" containerName="limitador" Jul 06 16:52:17.756146 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:17.756122 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-ndsgq" Jul 06 16:52:17.758886 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:17.758867 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 06 16:52:17.759008 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:17.758918 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zdcr2\"" Jul 06 16:52:17.765452 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:17.765432 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-ndsgq"] Jul 06 16:52:17.866499 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:17.866472 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zktqj\" (UniqueName: \"kubernetes.io/projected/a6fcd064-9328-429a-8356-84b50309e4fd-kube-api-access-zktqj\") pod \"postgres-69fb4594cf-ndsgq\" (UID: \"a6fcd064-9328-429a-8356-84b50309e4fd\") " pod="opendatahub/postgres-69fb4594cf-ndsgq" Jul 06 16:52:17.866660 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:17.866547 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a6fcd064-9328-429a-8356-84b50309e4fd-data\") pod \"postgres-69fb4594cf-ndsgq\" (UID: \"a6fcd064-9328-429a-8356-84b50309e4fd\") " pod="opendatahub/postgres-69fb4594cf-ndsgq" Jul 06 16:52:17.967332 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:17.967296 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a6fcd064-9328-429a-8356-84b50309e4fd-data\") pod \"postgres-69fb4594cf-ndsgq\" (UID: \"a6fcd064-9328-429a-8356-84b50309e4fd\") " pod="opendatahub/postgres-69fb4594cf-ndsgq" Jul 06 16:52:17.967506 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:17.967400 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zktqj\" (UniqueName: \"kubernetes.io/projected/a6fcd064-9328-429a-8356-84b50309e4fd-kube-api-access-zktqj\") pod \"postgres-69fb4594cf-ndsgq\" (UID: \"a6fcd064-9328-429a-8356-84b50309e4fd\") " pod="opendatahub/postgres-69fb4594cf-ndsgq" Jul 06 16:52:17.967814 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:17.967795 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a6fcd064-9328-429a-8356-84b50309e4fd-data\") pod \"postgres-69fb4594cf-ndsgq\" (UID: \"a6fcd064-9328-429a-8356-84b50309e4fd\") " pod="opendatahub/postgres-69fb4594cf-ndsgq" Jul 06 16:52:17.976927 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:17.976895 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zktqj\" (UniqueName: \"kubernetes.io/projected/a6fcd064-9328-429a-8356-84b50309e4fd-kube-api-access-zktqj\") pod \"postgres-69fb4594cf-ndsgq\" (UID: \"a6fcd064-9328-429a-8356-84b50309e4fd\") " pod="opendatahub/postgres-69fb4594cf-ndsgq" Jul 06 16:52:18.066925 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:18.066836 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-ndsgq" Jul 06 16:52:18.198894 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:18.198869 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-ndsgq"] Jul 06 16:52:18.201539 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:52:18.201493 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6fcd064_9328_429a_8356_84b50309e4fd.slice/crio-fd03f781ca621dc05b12ad26eaab52436fb7fef6eaeaea0ef001bcd2368d2464 WatchSource:0}: Error finding container fd03f781ca621dc05b12ad26eaab52436fb7fef6eaeaea0ef001bcd2368d2464: Status 404 returned error can't find the container with id fd03f781ca621dc05b12ad26eaab52436fb7fef6eaeaea0ef001bcd2368d2464 Jul 06 16:52:18.217790 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:18.217766 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-ndsgq" event={"ID":"a6fcd064-9328-429a-8356-84b50309e4fd","Type":"ContainerStarted","Data":"fd03f781ca621dc05b12ad26eaab52436fb7fef6eaeaea0ef001bcd2368d2464"} Jul 06 16:52:24.253069 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:24.252977 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-ndsgq" event={"ID":"a6fcd064-9328-429a-8356-84b50309e4fd","Type":"ContainerStarted","Data":"a8f2bd2e68e7fe174db7c628b33a0aae89896136e390a3afdc5f720815cbac8b"} Jul 06 16:52:24.253069 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:24.253049 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-ndsgq" Jul 06 16:52:24.273119 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:24.273071 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-ndsgq" podStartSLOduration=1.482844546 podStartE2EDuration="7.27305626s" totalImagesPullingTime="5.790211714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:52:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:52:24.271860553 +0000 UTC m=+837.791351706" watchObservedRunningTime="2026-07-06 16:52:24.27305626 +0000 UTC m=+837.792547373" Jul 06 16:52:30.287001 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:52:30.286972 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-ndsgq" Jul 06 16:53:14.412021 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:14.411982 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-cm8wv"] Jul 06 16:53:14.427833 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:14.427799 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-cm8wv" Jul 06 16:53:14.430693 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:14.430635 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-cm8wv"] Jul 06 16:53:14.430941 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:14.430912 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 06 16:53:14.431092 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:14.431073 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-9djzt\"" Jul 06 16:53:14.494960 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:14.494934 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 16:53:14.551362 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:14.551335 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfn8v\" (UniqueName: \"kubernetes.io/projected/ef7d126c-7733-4313-9644-7b8de3b45a93-kube-api-access-zfn8v\") pod \"payload-processing-58f86c7798-cm8wv\" (UID: \"ef7d126c-7733-4313-9644-7b8de3b45a93\") " pod="openshift-ingress/payload-processing-58f86c7798-cm8wv" Jul 06 16:53:14.551502 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:14.551407 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef7d126c-7733-4313-9644-7b8de3b45a93-plugins-config-volume\") pod \"payload-processing-58f86c7798-cm8wv\" (UID: \"ef7d126c-7733-4313-9644-7b8de3b45a93\") " pod="openshift-ingress/payload-processing-58f86c7798-cm8wv" Jul 06 16:53:14.652391 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:14.652356 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef7d126c-7733-4313-9644-7b8de3b45a93-plugins-config-volume\") pod \"payload-processing-58f86c7798-cm8wv\" (UID: \"ef7d126c-7733-4313-9644-7b8de3b45a93\") " pod="openshift-ingress/payload-processing-58f86c7798-cm8wv" Jul 06 16:53:14.652600 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:14.652457 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfn8v\" (UniqueName: \"kubernetes.io/projected/ef7d126c-7733-4313-9644-7b8de3b45a93-kube-api-access-zfn8v\") pod \"payload-processing-58f86c7798-cm8wv\" (UID: \"ef7d126c-7733-4313-9644-7b8de3b45a93\") " pod="openshift-ingress/payload-processing-58f86c7798-cm8wv" Jul 06 16:53:14.653241 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:14.653215 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef7d126c-7733-4313-9644-7b8de3b45a93-plugins-config-volume\") pod \"payload-processing-58f86c7798-cm8wv\" (UID: \"ef7d126c-7733-4313-9644-7b8de3b45a93\") " pod="openshift-ingress/payload-processing-58f86c7798-cm8wv" Jul 06 16:53:14.666953 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:14.666903 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfn8v\" (UniqueName: \"kubernetes.io/projected/ef7d126c-7733-4313-9644-7b8de3b45a93-kube-api-access-zfn8v\") pod \"payload-processing-58f86c7798-cm8wv\" (UID: \"ef7d126c-7733-4313-9644-7b8de3b45a93\") " pod="openshift-ingress/payload-processing-58f86c7798-cm8wv" Jul 06 16:53:14.762001 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:14.761964 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-cm8wv" Jul 06 16:53:14.899978 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:14.899948 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-cm8wv"] Jul 06 16:53:14.903330 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:53:14.903280 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7d126c_7733_4313_9644_7b8de3b45a93.slice/crio-b2d60706d3f1765513bc0ed838efafc96362f87ca6e8149eae8d02cf0c0cb0ed WatchSource:0}: Error finding container b2d60706d3f1765513bc0ed838efafc96362f87ca6e8149eae8d02cf0c0cb0ed: Status 404 returned error can't find the container with id b2d60706d3f1765513bc0ed838efafc96362f87ca6e8149eae8d02cf0c0cb0ed Jul 06 16:53:15.477390 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:15.477361 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-cm8wv" event={"ID":"ef7d126c-7733-4313-9644-7b8de3b45a93","Type":"ContainerStarted","Data":"b2d60706d3f1765513bc0ed838efafc96362f87ca6e8149eae8d02cf0c0cb0ed"} Jul 06 16:53:17.488894 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:17.488861 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-cm8wv" event={"ID":"ef7d126c-7733-4313-9644-7b8de3b45a93","Type":"ContainerStarted","Data":"fd0d3dd2b4b1a471f3beac69dfca5d5a07e4e7e4576dc56aa83cb643f8dbaf3a"} Jul 06 16:53:17.489248 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:17.488939 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-cm8wv" Jul 06 16:53:17.508717 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:17.508669 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-cm8wv" podStartSLOduration=1.81218636 podStartE2EDuration="3.508653411s" totalImagesPullingTime="1.696467051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:53:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:53:17.506764967 +0000 UTC m=+891.026256084" watchObservedRunningTime="2026-07-06 16:53:17.508653411 +0000 UTC m=+891.028144558" Jul 06 16:53:22.513732 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:22.513681 2574 generic.go:356] "Generic (PLEG): container finished" podID="9ccabcba-7d81-48aa-919f-150b01a4959c" containerID="9ee6acf2901095753d1ab40b5f7fd5f831ddd1f719ff279def90ab75486a6427" exitCode=0 Jul 06 16:53:22.514258 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:22.513760 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" event={"ID":"9ccabcba-7d81-48aa-919f-150b01a4959c","Type":"ContainerDied","Data":"9ee6acf2901095753d1ab40b5f7fd5f831ddd1f719ff279def90ab75486a6427"} Jul 06 16:53:22.514258 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:22.514249 2574 scope.go:117] "RemoveContainer" containerID="9ee6acf2901095753d1ab40b5f7fd5f831ddd1f719ff279def90ab75486a6427" Jul 06 16:53:22.515310 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:22.515279 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 16:53:22.515389 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:22.515364 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 16:53:22.515424 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:22.515409 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 16:53:23.519879 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:23.519842 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" event={"ID":"9ccabcba-7d81-48aa-919f-150b01a4959c","Type":"ContainerStarted","Data":"0226113f3ed6e70812da414ff4cceb90077cf2b7a6c05926f7e19d564c9b26ad"} Jul 06 16:53:23.763830 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:23.763781 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:53:23.765081 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:23.765053 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-mqc82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 06 16:53:23.765209 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:23.765116 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" podUID="9ccabcba-7d81-48aa-919f-150b01a4959c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 06 16:53:24.764612 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:24.764577 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-mqc82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 06 16:53:24.765086 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:24.764639 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" podUID="9ccabcba-7d81-48aa-919f-150b01a4959c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 06 16:53:25.764057 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:25.764017 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-mqc82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 06 16:53:25.764235 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:25.764073 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" podUID="9ccabcba-7d81-48aa-919f-150b01a4959c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 06 16:53:26.101291 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:26.101197 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:53:26.764561 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:26.764498 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-mqc82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 06 16:53:26.764743 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:26.764576 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" podUID="9ccabcba-7d81-48aa-919f-150b01a4959c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 06 16:53:27.767704 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:27.767673 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:53:27.768659 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:27.768643 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-mqc82" Jul 06 16:53:28.496009 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:28.495979 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-cm8wv" Jul 06 16:53:33.876118 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:33.876080 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 16:53:37.474342 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.474309 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk"] Jul 06 16:53:37.477462 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.477440 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk" Jul 06 16:53:37.481170 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.481144 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 16:53:37.481306 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.481224 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-s9bmb\"" Jul 06 16:53:37.481306 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.481225 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 16:53:37.481425 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.481225 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 06 16:53:37.488730 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.488707 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk"] Jul 06 16:53:37.562094 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.562059 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88361605-3306-42ad-b7bd-ffab7e1f3313-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jfbsk\" (UID: \"88361605-3306-42ad-b7bd-ffab7e1f3313\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk" Jul 06 16:53:37.562270 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.562102 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88361605-3306-42ad-b7bd-ffab7e1f3313-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jfbsk\" (UID: \"88361605-3306-42ad-b7bd-ffab7e1f3313\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk" Jul 06 16:53:37.562270 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.562130 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9fpd\" (UniqueName: \"kubernetes.io/projected/88361605-3306-42ad-b7bd-ffab7e1f3313-kube-api-access-p9fpd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jfbsk\" (UID: \"88361605-3306-42ad-b7bd-ffab7e1f3313\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk" Jul 06 16:53:37.562270 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.562187 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88361605-3306-42ad-b7bd-ffab7e1f3313-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jfbsk\" (UID: \"88361605-3306-42ad-b7bd-ffab7e1f3313\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk" Jul 06 16:53:37.562270 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.562209 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88361605-3306-42ad-b7bd-ffab7e1f3313-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jfbsk\" (UID: \"88361605-3306-42ad-b7bd-ffab7e1f3313\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk" Jul 06 16:53:37.562418 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.562286 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88361605-3306-42ad-b7bd-ffab7e1f3313-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jfbsk\" (UID: \"88361605-3306-42ad-b7bd-ffab7e1f3313\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk" Jul 06 16:53:37.662940 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.662904 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88361605-3306-42ad-b7bd-ffab7e1f3313-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jfbsk\" (UID: \"88361605-3306-42ad-b7bd-ffab7e1f3313\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk" Jul 06 16:53:37.662940 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.662940 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9fpd\" (UniqueName: \"kubernetes.io/projected/88361605-3306-42ad-b7bd-ffab7e1f3313-kube-api-access-p9fpd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jfbsk\" (UID: \"88361605-3306-42ad-b7bd-ffab7e1f3313\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk" Jul 06 16:53:37.663188 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.662999 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88361605-3306-42ad-b7bd-ffab7e1f3313-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jfbsk\" (UID: \"88361605-3306-42ad-b7bd-ffab7e1f3313\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk" Jul 06 16:53:37.663188 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.663014 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88361605-3306-42ad-b7bd-ffab7e1f3313-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jfbsk\" (UID: \"88361605-3306-42ad-b7bd-ffab7e1f3313\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk" Jul 06 16:53:37.663188 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.663041 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88361605-3306-42ad-b7bd-ffab7e1f3313-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jfbsk\" (UID: \"88361605-3306-42ad-b7bd-ffab7e1f3313\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk" Jul 06 16:53:37.663188 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.663069 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88361605-3306-42ad-b7bd-ffab7e1f3313-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jfbsk\" (UID: \"88361605-3306-42ad-b7bd-ffab7e1f3313\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk" Jul 06 16:53:37.663462 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.663440 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88361605-3306-42ad-b7bd-ffab7e1f3313-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jfbsk\" (UID: \"88361605-3306-42ad-b7bd-ffab7e1f3313\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk" Jul 06 16:53:37.663547 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.663489 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88361605-3306-42ad-b7bd-ffab7e1f3313-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jfbsk\" (UID: \"88361605-3306-42ad-b7bd-ffab7e1f3313\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk" Jul 06 16:53:37.663547 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.663540 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88361605-3306-42ad-b7bd-ffab7e1f3313-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jfbsk\" (UID: \"88361605-3306-42ad-b7bd-ffab7e1f3313\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk" Jul 06 16:53:37.665308 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.665281 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88361605-3306-42ad-b7bd-ffab7e1f3313-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jfbsk\" (UID: \"88361605-3306-42ad-b7bd-ffab7e1f3313\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk" Jul 06 16:53:37.665564 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.665550 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88361605-3306-42ad-b7bd-ffab7e1f3313-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jfbsk\" (UID: \"88361605-3306-42ad-b7bd-ffab7e1f3313\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk" Jul 06 16:53:37.671936 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.671915 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9fpd\" (UniqueName: \"kubernetes.io/projected/88361605-3306-42ad-b7bd-ffab7e1f3313-kube-api-access-p9fpd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-jfbsk\" (UID: \"88361605-3306-42ad-b7bd-ffab7e1f3313\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk" Jul 06 16:53:37.789905 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.789809 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk" Jul 06 16:53:37.925506 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.925482 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk"] Jul 06 16:53:37.927691 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:53:37.927661 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88361605_3306_42ad_b7bd_ffab7e1f3313.slice/crio-69ff10a7acb98e9e15c5d811f3ff853bbb105b71af6ce6ecef1187dac02a2662 WatchSource:0}: Error finding container 69ff10a7acb98e9e15c5d811f3ff853bbb105b71af6ce6ecef1187dac02a2662: Status 404 returned error can't find the container with id 69ff10a7acb98e9e15c5d811f3ff853bbb105b71af6ce6ecef1187dac02a2662 Jul 06 16:53:37.929116 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:37.929097 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 16:53:38.090602 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:38.090501 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 16:53:38.592022 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:38.591984 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk" event={"ID":"88361605-3306-42ad-b7bd-ffab7e1f3313","Type":"ContainerStarted","Data":"69ff10a7acb98e9e15c5d811f3ff853bbb105b71af6ce6ecef1187dac02a2662"} Jul 06 16:53:39.597650 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:39.597612 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk" event={"ID":"88361605-3306-42ad-b7bd-ffab7e1f3313","Type":"ContainerStarted","Data":"1c85ad3cfc917d2a234f115c5e806c8762d41ceedf9c21a381c36af6ba131a52"} Jul 06 16:53:39.620340 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:39.620288 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk" podStartSLOduration=1.458206807 podStartE2EDuration="2.620273412s" totalImagesPullingTime="1.162066605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:53:39.61886273 +0000 UTC m=+913.138353871" watchObservedRunningTime="2026-07-06 16:53:39.620273412 +0000 UTC m=+913.139764525" Jul 06 16:53:47.790620 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:47.790578 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk" Jul 06 16:53:47.791188 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:47.790883 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk" Jul 06 16:53:47.802644 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:47.802623 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk" Jul 06 16:53:48.652771 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:48.652746 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-jfbsk" Jul 06 16:53:59.093396 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:59.093365 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s"] Jul 06 16:53:59.096781 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:59.096759 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s" Jul 06 16:53:59.099252 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:59.099230 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 06 16:53:59.108668 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:59.108644 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s"] Jul 06 16:53:59.242357 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:59.242323 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4486b3ed-42d1-4b23-ac20-08c5247ff5f5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mn95s\" (UID: \"4486b3ed-42d1-4b23-ac20-08c5247ff5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s" Jul 06 16:53:59.242357 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:59.242362 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4486b3ed-42d1-4b23-ac20-08c5247ff5f5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mn95s\" (UID: \"4486b3ed-42d1-4b23-ac20-08c5247ff5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s" Jul 06 16:53:59.242603 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:59.242384 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcd7d\" (UniqueName: \"kubernetes.io/projected/4486b3ed-42d1-4b23-ac20-08c5247ff5f5-kube-api-access-lcd7d\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mn95s\" (UID: \"4486b3ed-42d1-4b23-ac20-08c5247ff5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s" Jul 06 16:53:59.242603 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:59.242404 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4486b3ed-42d1-4b23-ac20-08c5247ff5f5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mn95s\" (UID: \"4486b3ed-42d1-4b23-ac20-08c5247ff5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s" Jul 06 16:53:59.242603 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:59.242479 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4486b3ed-42d1-4b23-ac20-08c5247ff5f5-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mn95s\" (UID: \"4486b3ed-42d1-4b23-ac20-08c5247ff5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s" Jul 06 16:53:59.242603 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:59.242510 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4486b3ed-42d1-4b23-ac20-08c5247ff5f5-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mn95s\" (UID: \"4486b3ed-42d1-4b23-ac20-08c5247ff5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s" Jul 06 16:53:59.344059 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:59.343964 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4486b3ed-42d1-4b23-ac20-08c5247ff5f5-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mn95s\" (UID: \"4486b3ed-42d1-4b23-ac20-08c5247ff5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s" Jul 06 16:53:59.344210 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:59.344076 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4486b3ed-42d1-4b23-ac20-08c5247ff5f5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mn95s\" (UID: \"4486b3ed-42d1-4b23-ac20-08c5247ff5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s" Jul 06 16:53:59.344210 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:59.344105 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4486b3ed-42d1-4b23-ac20-08c5247ff5f5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mn95s\" (UID: \"4486b3ed-42d1-4b23-ac20-08c5247ff5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s" Jul 06 16:53:59.344210 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:59.344141 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcd7d\" (UniqueName: \"kubernetes.io/projected/4486b3ed-42d1-4b23-ac20-08c5247ff5f5-kube-api-access-lcd7d\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mn95s\" (UID: \"4486b3ed-42d1-4b23-ac20-08c5247ff5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s" Jul 06 16:53:59.344210 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:59.344168 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4486b3ed-42d1-4b23-ac20-08c5247ff5f5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mn95s\" (UID: \"4486b3ed-42d1-4b23-ac20-08c5247ff5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s" Jul 06 16:53:59.344400 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:59.344215 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4486b3ed-42d1-4b23-ac20-08c5247ff5f5-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mn95s\" (UID: \"4486b3ed-42d1-4b23-ac20-08c5247ff5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s" Jul 06 16:53:59.344502 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:59.344478 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4486b3ed-42d1-4b23-ac20-08c5247ff5f5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mn95s\" (UID: \"4486b3ed-42d1-4b23-ac20-08c5247ff5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s" Jul 06 16:53:59.344615 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:59.344595 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4486b3ed-42d1-4b23-ac20-08c5247ff5f5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mn95s\" (UID: \"4486b3ed-42d1-4b23-ac20-08c5247ff5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s" Jul 06 16:53:59.344714 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:59.344698 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4486b3ed-42d1-4b23-ac20-08c5247ff5f5-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mn95s\" (UID: \"4486b3ed-42d1-4b23-ac20-08c5247ff5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s" Jul 06 16:53:59.346278 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:59.346260 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4486b3ed-42d1-4b23-ac20-08c5247ff5f5-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mn95s\" (UID: \"4486b3ed-42d1-4b23-ac20-08c5247ff5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s" Jul 06 16:53:59.346683 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:59.346663 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4486b3ed-42d1-4b23-ac20-08c5247ff5f5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mn95s\" (UID: \"4486b3ed-42d1-4b23-ac20-08c5247ff5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s" Jul 06 16:53:59.352863 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:59.352842 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcd7d\" (UniqueName: \"kubernetes.io/projected/4486b3ed-42d1-4b23-ac20-08c5247ff5f5-kube-api-access-lcd7d\") pod \"e2e-trlp-test-simulated-kserve-598466548d-mn95s\" (UID: \"4486b3ed-42d1-4b23-ac20-08c5247ff5f5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s" Jul 06 16:53:59.406637 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:59.406605 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s" Jul 06 16:53:59.539430 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:59.539407 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s"] Jul 06 16:53:59.541551 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:53:59.541494 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4486b3ed_42d1_4b23_ac20_08c5247ff5f5.slice/crio-566eb7d13242cf847fd9b5ffaf79b67200a3dd269d55048fbcddb1f628d121dc WatchSource:0}: Error finding container 566eb7d13242cf847fd9b5ffaf79b67200a3dd269d55048fbcddb1f628d121dc: Status 404 returned error can't find the container with id 566eb7d13242cf847fd9b5ffaf79b67200a3dd269d55048fbcddb1f628d121dc Jul 06 16:53:59.691950 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:53:59.691869 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s" event={"ID":"4486b3ed-42d1-4b23-ac20-08c5247ff5f5","Type":"ContainerStarted","Data":"566eb7d13242cf847fd9b5ffaf79b67200a3dd269d55048fbcddb1f628d121dc"} Jul 06 16:54:00.197544 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:00.197497 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 16:54:00.697654 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:00.697620 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s" event={"ID":"4486b3ed-42d1-4b23-ac20-08c5247ff5f5","Type":"ContainerStarted","Data":"504d9509d4999c377c5b36895bdbf80b0d97c87f240c6a952f9c799b89cb1a0d"} Jul 06 16:54:00.723557 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:00.723472 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s" podStartSLOduration=1.535771014 podStartE2EDuration="1.723448957s" totalImagesPullingTime="187.677943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:54:00.722467652 +0000 UTC m=+934.241958770" watchObservedRunningTime="2026-07-06 16:54:00.723448957 +0000 UTC m=+934.242940075" Jul 06 16:54:09.407568 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:09.407533 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s" Jul 06 16:54:09.407954 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:09.407595 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s" Jul 06 16:54:09.419848 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:09.419826 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s" Jul 06 16:54:09.749257 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:09.749230 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-mn95s" Jul 06 16:54:10.201106 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:10.201064 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 16:54:22.915451 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:22.915414 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 16:54:31.757289 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:31.757226 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg"] Jul 06 16:54:31.760776 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:31.760759 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg" Jul 06 16:54:31.767243 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:31.766805 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 06 16:54:31.801897 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:31.801862 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg"] Jul 06 16:54:31.830761 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:31.830726 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92e4ce7c-9cc9-446a-ba9a-b4505caddd7f-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg\" (UID: \"92e4ce7c-9cc9-446a-ba9a-b4505caddd7f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg" Jul 06 16:54:31.830946 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:31.830847 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92e4ce7c-9cc9-446a-ba9a-b4505caddd7f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg\" (UID: \"92e4ce7c-9cc9-446a-ba9a-b4505caddd7f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg" Jul 06 16:54:31.830946 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:31.830887 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92e4ce7c-9cc9-446a-ba9a-b4505caddd7f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg\" (UID: \"92e4ce7c-9cc9-446a-ba9a-b4505caddd7f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg" Jul 06 16:54:31.830946 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:31.830920 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkp8s\" (UniqueName: \"kubernetes.io/projected/92e4ce7c-9cc9-446a-ba9a-b4505caddd7f-kube-api-access-jkp8s\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg\" (UID: \"92e4ce7c-9cc9-446a-ba9a-b4505caddd7f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg" Jul 06 16:54:31.831093 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:31.830971 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92e4ce7c-9cc9-446a-ba9a-b4505caddd7f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg\" (UID: \"92e4ce7c-9cc9-446a-ba9a-b4505caddd7f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg" Jul 06 16:54:31.831093 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:31.830997 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92e4ce7c-9cc9-446a-ba9a-b4505caddd7f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg\" (UID: \"92e4ce7c-9cc9-446a-ba9a-b4505caddd7f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg" Jul 06 16:54:31.931448 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:31.931415 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92e4ce7c-9cc9-446a-ba9a-b4505caddd7f-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg\" (UID: \"92e4ce7c-9cc9-446a-ba9a-b4505caddd7f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg" Jul 06 16:54:31.931448 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:31.931463 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92e4ce7c-9cc9-446a-ba9a-b4505caddd7f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg\" (UID: \"92e4ce7c-9cc9-446a-ba9a-b4505caddd7f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg" Jul 06 16:54:31.931731 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:31.931479 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92e4ce7c-9cc9-446a-ba9a-b4505caddd7f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg\" (UID: \"92e4ce7c-9cc9-446a-ba9a-b4505caddd7f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg" Jul 06 16:54:31.931731 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:31.931503 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkp8s\" (UniqueName: \"kubernetes.io/projected/92e4ce7c-9cc9-446a-ba9a-b4505caddd7f-kube-api-access-jkp8s\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg\" (UID: \"92e4ce7c-9cc9-446a-ba9a-b4505caddd7f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg" Jul 06 16:54:31.931731 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:31.931565 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92e4ce7c-9cc9-446a-ba9a-b4505caddd7f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg\" (UID: \"92e4ce7c-9cc9-446a-ba9a-b4505caddd7f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg" Jul 06 16:54:31.931731 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:31.931595 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92e4ce7c-9cc9-446a-ba9a-b4505caddd7f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg\" (UID: \"92e4ce7c-9cc9-446a-ba9a-b4505caddd7f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg" Jul 06 16:54:31.931924 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:31.931901 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92e4ce7c-9cc9-446a-ba9a-b4505caddd7f-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg\" (UID: \"92e4ce7c-9cc9-446a-ba9a-b4505caddd7f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg" Jul 06 16:54:31.931983 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:31.931943 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92e4ce7c-9cc9-446a-ba9a-b4505caddd7f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg\" (UID: \"92e4ce7c-9cc9-446a-ba9a-b4505caddd7f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg" Jul 06 16:54:31.932057 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:31.932037 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92e4ce7c-9cc9-446a-ba9a-b4505caddd7f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg\" (UID: \"92e4ce7c-9cc9-446a-ba9a-b4505caddd7f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg" Jul 06 16:54:31.933939 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:31.933914 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92e4ce7c-9cc9-446a-ba9a-b4505caddd7f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg\" (UID: \"92e4ce7c-9cc9-446a-ba9a-b4505caddd7f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg" Jul 06 16:54:31.934574 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:31.934551 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92e4ce7c-9cc9-446a-ba9a-b4505caddd7f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg\" (UID: \"92e4ce7c-9cc9-446a-ba9a-b4505caddd7f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg" Jul 06 16:54:31.955015 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:31.954982 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkp8s\" (UniqueName: \"kubernetes.io/projected/92e4ce7c-9cc9-446a-ba9a-b4505caddd7f-kube-api-access-jkp8s\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg\" (UID: \"92e4ce7c-9cc9-446a-ba9a-b4505caddd7f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg" Jul 06 16:54:32.069967 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:32.069879 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg" Jul 06 16:54:32.208635 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:32.208607 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg"] Jul 06 16:54:32.211235 ip-10-0-131-71 kubenswrapper[2574]: W0706 16:54:32.211211 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod92e4ce7c_9cc9_446a_ba9a_b4505caddd7f.slice/crio-ae13d5021966b5752a8e77ec80dd73ccbe95edc6956e39c3de4d6d16f6455062 WatchSource:0}: Error finding container ae13d5021966b5752a8e77ec80dd73ccbe95edc6956e39c3de4d6d16f6455062: Status 404 returned error can't find the container with id ae13d5021966b5752a8e77ec80dd73ccbe95edc6956e39c3de4d6d16f6455062 Jul 06 16:54:32.850757 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:32.850718 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg" event={"ID":"92e4ce7c-9cc9-446a-ba9a-b4505caddd7f","Type":"ContainerStarted","Data":"2ab44ab53d47cc2b962baf0c25a067ce6d8172fafff7ccaf8704b9cb83e2d385"} Jul 06 16:54:32.850757 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:32.850761 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg" event={"ID":"92e4ce7c-9cc9-446a-ba9a-b4505caddd7f","Type":"ContainerStarted","Data":"ae13d5021966b5752a8e77ec80dd73ccbe95edc6956e39c3de4d6d16f6455062"} Jul 06 16:54:32.876631 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:32.876566 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg" podStartSLOduration=1.6727171379999999 podStartE2EDuration="1.876547627s" totalImagesPullingTime="203.830489ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:54:32.875273398 +0000 UTC m=+966.394764538" watchObservedRunningTime="2026-07-06 16:54:32.876547627 +0000 UTC m=+966.396038740" Jul 06 16:54:34.676183 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:34.676153 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 16:54:42.070277 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:42.070239 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg" Jul 06 16:54:42.070277 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:42.070283 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg" Jul 06 16:54:42.083033 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:42.083007 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg" Jul 06 16:54:42.908337 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:54:42.908308 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg" Jul 06 16:56:33.707067 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:56:33.707030 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 16:56:43.298684 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:56:43.298641 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 16:56:55.188920 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:56:55.188881 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 16:57:04.136908 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:57:04.136866 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 16:57:14.493351 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:57:14.493318 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 16:57:41.601080 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:57:41.601043 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 16:57:52.310424 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:57:52.310388 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 16:58:28.208232 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:58:28.208199 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 16:58:37.803706 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:58:37.803672 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 16:58:46.009548 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:58:46.009478 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 16:58:54.810283 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:58:54.810249 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 16:59:06.358772 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:59:06.358726 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 16:59:15.218085 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:59:15.218056 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 16:59:26.898654 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:59:26.898561 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 16:59:35.797107 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:59:35.797068 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 16:59:59.502318 ip-10-0-131-71 kubenswrapper[2574]: I0706 16:59:59.502264 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:00:08.099961 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:00:08.099925 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:00:32.904971 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:00:32.904934 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:00:43.293410 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:00:43.293371 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:00:57.405363 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:00:57.405325 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:01:06.697957 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:01:06.697921 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:01:17.406574 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:01:17.406536 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:01:26.909549 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:01:26.909494 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:01:36.108131 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:01:36.108091 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:02:01.923007 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:02:01.922970 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sln8p"] Jul 06 17:02:01.923637 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:02:01.923165 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sln8p" podUID="5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5" containerName="manager" containerID="cri-o://6b024405f60545fdb11ba14cc3c542116c57610d080f96325c5f6600205f305f" gracePeriod=10 Jul 06 17:02:04.983822 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:02:04.983738 2574 generic.go:356] "Generic (PLEG): container finished" podID="5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5" containerID="6b024405f60545fdb11ba14cc3c542116c57610d080f96325c5f6600205f305f" exitCode=0 Jul 06 17:02:04.983822 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:02:04.983795 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sln8p" event={"ID":"5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5","Type":"ContainerDied","Data":"6b024405f60545fdb11ba14cc3c542116c57610d080f96325c5f6600205f305f"} Jul 06 17:02:05.375430 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:02:05.375402 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sln8p" Jul 06 17:02:05.463453 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:02:05.463423 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5-extensions-socket-volume\") pod \"5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5\" (UID: \"5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5\") " Jul 06 17:02:05.463662 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:02:05.463581 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8tw7\" (UniqueName: \"kubernetes.io/projected/5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5-kube-api-access-s8tw7\") pod \"5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5\" (UID: \"5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5\") " Jul 06 17:02:05.463807 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:02:05.463783 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5" (UID: "5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:02:05.465475 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:02:05.465455 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5-kube-api-access-s8tw7" (OuterVolumeSpecName: "kube-api-access-s8tw7") pod "5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5" (UID: "5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5"). InnerVolumeSpecName "kube-api-access-s8tw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:02:05.564440 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:02:05.564362 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8tw7\" (UniqueName: \"kubernetes.io/projected/5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5-kube-api-access-s8tw7\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:02:05.564440 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:02:05.564390 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5-extensions-socket-volume\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:02:05.989978 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:02:05.989949 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sln8p" Jul 06 17:02:05.990371 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:02:05.989986 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sln8p" event={"ID":"5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5","Type":"ContainerDied","Data":"9fc194c49b71ce4f985e4b92779a3ca9da8d23cb1cb23b0051eb5a54ce08f11b"} Jul 06 17:02:05.990371 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:02:05.990048 2574 scope.go:117] "RemoveContainer" containerID="6b024405f60545fdb11ba14cc3c542116c57610d080f96325c5f6600205f305f" Jul 06 17:02:06.012900 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:02:06.012873 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sln8p"] Jul 06 17:02:06.017648 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:02:06.017626 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-sln8p"] Jul 06 17:02:07.076112 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:02:07.076081 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5" path="/var/lib/kubelet/pods/5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5/volumes" Jul 06 17:03:03.081406 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:03:03.081368 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pbk8v"] Jul 06 17:03:03.082036 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:03:03.081884 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5" containerName="manager" Jul 06 17:03:03.082036 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:03:03.081906 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5" containerName="manager" Jul 06 17:03:03.082036 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:03:03.082020 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cef6a44-8a2e-4a92-bfef-8597e6d0b9f5" containerName="manager" Jul 06 17:03:03.084802 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:03:03.084781 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pbk8v" Jul 06 17:03:03.087727 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:03:03.087710 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xjwsf\"" Jul 06 17:03:03.102096 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:03:03.102071 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pbk8v"] Jul 06 17:03:03.175896 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:03:03.175863 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6305877e-6719-41f7-9d9f-3ec5534c8cdd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-pbk8v\" (UID: \"6305877e-6719-41f7-9d9f-3ec5534c8cdd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pbk8v" Jul 06 17:03:03.176177 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:03:03.176150 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtqhl\" (UniqueName: \"kubernetes.io/projected/6305877e-6719-41f7-9d9f-3ec5534c8cdd-kube-api-access-jtqhl\") pod \"kuadrant-operator-controller-manager-86878c769f-pbk8v\" (UID: \"6305877e-6719-41f7-9d9f-3ec5534c8cdd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pbk8v" Jul 06 17:03:03.277141 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:03:03.277107 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6305877e-6719-41f7-9d9f-3ec5534c8cdd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-pbk8v\" (UID: \"6305877e-6719-41f7-9d9f-3ec5534c8cdd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pbk8v" Jul 06 17:03:03.277289 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:03:03.277164 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtqhl\" (UniqueName: \"kubernetes.io/projected/6305877e-6719-41f7-9d9f-3ec5534c8cdd-kube-api-access-jtqhl\") pod \"kuadrant-operator-controller-manager-86878c769f-pbk8v\" (UID: \"6305877e-6719-41f7-9d9f-3ec5534c8cdd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pbk8v" Jul 06 17:03:03.277478 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:03:03.277460 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6305877e-6719-41f7-9d9f-3ec5534c8cdd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-pbk8v\" (UID: \"6305877e-6719-41f7-9d9f-3ec5534c8cdd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pbk8v" Jul 06 17:03:03.287223 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:03:03.287190 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtqhl\" (UniqueName: \"kubernetes.io/projected/6305877e-6719-41f7-9d9f-3ec5534c8cdd-kube-api-access-jtqhl\") pod \"kuadrant-operator-controller-manager-86878c769f-pbk8v\" (UID: \"6305877e-6719-41f7-9d9f-3ec5534c8cdd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pbk8v" Jul 06 17:03:03.394842 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:03:03.394761 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pbk8v" Jul 06 17:03:03.542533 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:03:03.542480 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pbk8v"] Jul 06 17:03:03.544477 ip-10-0-131-71 kubenswrapper[2574]: W0706 17:03:03.544438 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6305877e_6719_41f7_9d9f_3ec5534c8cdd.slice/crio-03c05fc54dc7e619df9dac4e7f1f3f1e6249cd6b4c4f1718907a6c06ad5a7052 WatchSource:0}: Error finding container 03c05fc54dc7e619df9dac4e7f1f3f1e6249cd6b4c4f1718907a6c06ad5a7052: Status 404 returned error can't find the container with id 03c05fc54dc7e619df9dac4e7f1f3f1e6249cd6b4c4f1718907a6c06ad5a7052 Jul 06 17:03:04.261485 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:03:04.261445 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pbk8v" event={"ID":"6305877e-6719-41f7-9d9f-3ec5534c8cdd","Type":"ContainerStarted","Data":"5605fdee0ccbff4481f92a643f0ca578e6d19f1035471bd8156709411901ce17"} Jul 06 17:03:04.261910 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:03:04.261505 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pbk8v" event={"ID":"6305877e-6719-41f7-9d9f-3ec5534c8cdd","Type":"ContainerStarted","Data":"03c05fc54dc7e619df9dac4e7f1f3f1e6249cd6b4c4f1718907a6c06ad5a7052"} Jul 06 17:03:04.261910 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:03:04.261569 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pbk8v" Jul 06 17:03:04.285945 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:03:04.285901 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pbk8v" podStartSLOduration=1.28588776 podStartE2EDuration="1.28588776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:03:04.282865383 +0000 UTC m=+1477.802356499" watchObservedRunningTime="2026-07-06 17:03:04.28588776 +0000 UTC m=+1477.805378875" Jul 06 17:03:15.268223 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:03:15.268195 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-pbk8v" Jul 06 17:03:22.907392 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:03:22.907362 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:03:33.836391 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:03:33.836354 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:03:45.125385 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:03:45.125351 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:03:55.418182 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:03:55.418150 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:04:05.748162 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:04:05.748125 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:04:17.144990 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:04:17.144954 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:04:27.024711 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:04:27.024677 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:04:38.101858 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:04:38.101822 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:04:48.749552 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:04:48.749498 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:04:57.613838 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:04:57.613803 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:05:07.622548 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:05:07.622501 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:05:32.298482 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:05:32.298398 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:05:39.717324 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:05:39.717289 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:05:49.326614 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:05:49.326579 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:06:01.713901 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:06:01.713864 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:06:10.412206 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:06:10.412171 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:06:18.673452 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:06:18.673414 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:06:28.214208 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:06:28.214172 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:06:37.201951 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:06:37.201917 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:06:48.736394 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:06:48.736300 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:06:57.840325 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:06:57.840288 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:07:07.009010 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:07:07.008976 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:07:19.902721 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:07:19.902686 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:07:28.314684 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:07:28.314647 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:07:38.990236 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:07:38.990203 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:07:50.922965 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:07:50.922931 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:08:01.399486 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:08:01.399452 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:08:10.992704 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:08:10.992663 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:08:21.692744 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:08:21.692662 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:08:33.614153 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:08:33.614118 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:08:45.196598 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:08:45.196566 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:08:57.600427 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:08:57.600385 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:09:09.799870 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:09.799832 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:09:16.058332 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.058299 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck"] Jul 06 17:09:16.061785 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.061769 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.065134 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.065103 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-38d19e47-openshift-default-dockercfg-cq7wd\"" Jul 06 17:09:16.083689 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.083665 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck"] Jul 06 17:09:16.159371 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.159339 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6504264f-9022-4136-94e5-787d001652bb-istiod-ca-cert\") pod \"e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.159371 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.159382 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-workload-socket\") pod \"e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.159661 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.159404 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-workload-certs\") pod \"e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.159661 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.159489 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6504264f-9022-4136-94e5-787d001652bb-istio-token\") pod \"e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.159661 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.159573 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-istio-envoy\") pod \"e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.159661 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.159591 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6504264f-9022-4136-94e5-787d001652bb-istio-podinfo\") pod \"e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.159661 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.159614 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-istio-data\") pod \"e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.159856 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.159675 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzzq9\" (UniqueName: \"kubernetes.io/projected/6504264f-9022-4136-94e5-787d001652bb-kube-api-access-xzzq9\") pod \"e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.159856 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.159714 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-credential-socket\") pod \"e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.261013 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.260971 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-credential-socket\") pod \"e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.261218 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.261069 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6504264f-9022-4136-94e5-787d001652bb-istiod-ca-cert\") pod \"e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.261218 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.261112 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-workload-socket\") pod \"e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.261218 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.261140 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-workload-certs\") pod \"e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.261218 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.261169 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6504264f-9022-4136-94e5-787d001652bb-istio-token\") pod \"e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.261218 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.261211 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-istio-envoy\") pod \"e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.261478 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.261235 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6504264f-9022-4136-94e5-787d001652bb-istio-podinfo\") pod \"e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.261478 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.261274 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-istio-data\") pod \"e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.261478 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.261313 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzzq9\" (UniqueName: \"kubernetes.io/projected/6504264f-9022-4136-94e5-787d001652bb-kube-api-access-xzzq9\") pod \"e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.261478 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.261443 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-credential-socket\") pod \"e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.261710 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.261583 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-workload-certs\") pod \"e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.261810 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.261790 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-istio-data\") pod \"e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.262022 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.261999 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6504264f-9022-4136-94e5-787d001652bb-istiod-ca-cert\") pod \"e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.262124 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.262101 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-workload-socket\") pod \"e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.263787 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.263764 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-istio-envoy\") pod \"e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.264113 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.264094 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6504264f-9022-4136-94e5-787d001652bb-istio-podinfo\") pod \"e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.269202 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.269179 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6504264f-9022-4136-94e5-787d001652bb-istio-token\") pod \"e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.269291 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.269211 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzzq9\" (UniqueName: \"kubernetes.io/projected/6504264f-9022-4136-94e5-787d001652bb-kube-api-access-xzzq9\") pod \"e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.372720 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.372623 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:16.512255 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.512233 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck"] Jul 06 17:09:16.514370 ip-10-0-131-71 kubenswrapper[2574]: W0706 17:09:16.514342 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6504264f_9022_4136_94e5_787d001652bb.slice/crio-b496696a14b0bebcdd3e1dfccd73ae011d6ec0c6ca61b148aca16444b1fc1d0f WatchSource:0}: Error finding container b496696a14b0bebcdd3e1dfccd73ae011d6ec0c6ca61b148aca16444b1fc1d0f: Status 404 returned error can't find the container with id b496696a14b0bebcdd3e1dfccd73ae011d6ec0c6ca61b148aca16444b1fc1d0f Jul 06 17:09:16.516540 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.516494 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 17:09:16.516642 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.516579 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 17:09:16.516642 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:16.516605 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 17:09:17.021171 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:17.021131 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" event={"ID":"6504264f-9022-4136-94e5-787d001652bb","Type":"ContainerStarted","Data":"5601e63daa32b403629c7ac312fc40e6469c34d3f46404b977308b974895bfa4"} Jul 06 17:09:17.021580 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:17.021200 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" event={"ID":"6504264f-9022-4136-94e5-787d001652bb","Type":"ContainerStarted","Data":"b496696a14b0bebcdd3e1dfccd73ae011d6ec0c6ca61b148aca16444b1fc1d0f"} Jul 06 17:09:17.051201 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:17.051141 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" podStartSLOduration=1.051120241 podStartE2EDuration="1.051120241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:09:17.049894511 +0000 UTC m=+1850.569385630" watchObservedRunningTime="2026-07-06 17:09:17.051120241 +0000 UTC m=+1850.570611356" Jul 06 17:09:17.373092 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:17.373007 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:17.377812 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:17.377787 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:18.025910 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:18.025874 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:18.026885 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:18.026868 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:22.595734 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:22.595696 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck"] Jul 06 17:09:22.596163 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:22.595878 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" podUID="6504264f-9022-4136-94e5-787d001652bb" containerName="istio-proxy" containerID="cri-o://5601e63daa32b403629c7ac312fc40e6469c34d3f46404b977308b974895bfa4" gracePeriod=30 Jul 06 17:09:27.836751 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:27.836728 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:27.973453 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:27.973421 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzzq9\" (UniqueName: \"kubernetes.io/projected/6504264f-9022-4136-94e5-787d001652bb-kube-api-access-xzzq9\") pod \"6504264f-9022-4136-94e5-787d001652bb\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " Jul 06 17:09:27.973667 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:27.973466 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6504264f-9022-4136-94e5-787d001652bb-istio-podinfo\") pod \"6504264f-9022-4136-94e5-787d001652bb\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " Jul 06 17:09:27.973667 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:27.973510 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-credential-socket\") pod \"6504264f-9022-4136-94e5-787d001652bb\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " Jul 06 17:09:27.973667 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:27.973550 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-istio-envoy\") pod \"6504264f-9022-4136-94e5-787d001652bb\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " Jul 06 17:09:27.973667 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:27.973570 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-workload-certs\") pod \"6504264f-9022-4136-94e5-787d001652bb\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " Jul 06 17:09:27.973667 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:27.973623 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6504264f-9022-4136-94e5-787d001652bb-istio-token\") pod \"6504264f-9022-4136-94e5-787d001652bb\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " Jul 06 17:09:27.973667 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:27.973645 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6504264f-9022-4136-94e5-787d001652bb-istiod-ca-cert\") pod \"6504264f-9022-4136-94e5-787d001652bb\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " Jul 06 17:09:27.974108 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:27.973680 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-istio-data\") pod \"6504264f-9022-4136-94e5-787d001652bb\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " Jul 06 17:09:27.974108 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:27.973730 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-workload-socket\") pod \"6504264f-9022-4136-94e5-787d001652bb\" (UID: \"6504264f-9022-4136-94e5-787d001652bb\") " Jul 06 17:09:27.974108 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:27.973879 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6504264f-9022-4136-94e5-787d001652bb" (UID: "6504264f-9022-4136-94e5-787d001652bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:09:27.974108 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:27.974032 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-credential-socket\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:09:27.974311 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:27.974158 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6504264f-9022-4136-94e5-787d001652bb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6504264f-9022-4136-94e5-787d001652bb" (UID: "6504264f-9022-4136-94e5-787d001652bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:09:27.974311 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:27.974243 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-istio-data" (OuterVolumeSpecName: "istio-data") pod "6504264f-9022-4136-94e5-787d001652bb" (UID: "6504264f-9022-4136-94e5-787d001652bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:09:27.974311 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:27.974234 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6504264f-9022-4136-94e5-787d001652bb" (UID: "6504264f-9022-4136-94e5-787d001652bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:09:27.974311 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:27.974294 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6504264f-9022-4136-94e5-787d001652bb" (UID: "6504264f-9022-4136-94e5-787d001652bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:09:27.975897 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:27.975866 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6504264f-9022-4136-94e5-787d001652bb-istio-token" (OuterVolumeSpecName: "istio-token") pod "6504264f-9022-4136-94e5-787d001652bb" (UID: "6504264f-9022-4136-94e5-787d001652bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:09:27.975986 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:27.975894 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6504264f-9022-4136-94e5-787d001652bb-kube-api-access-xzzq9" (OuterVolumeSpecName: "kube-api-access-xzzq9") pod "6504264f-9022-4136-94e5-787d001652bb" (UID: "6504264f-9022-4136-94e5-787d001652bb"). InnerVolumeSpecName "kube-api-access-xzzq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:09:27.976151 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:27.976130 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6504264f-9022-4136-94e5-787d001652bb" (UID: "6504264f-9022-4136-94e5-787d001652bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:09:27.976204 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:27.976142 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6504264f-9022-4136-94e5-787d001652bb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6504264f-9022-4136-94e5-787d001652bb" (UID: "6504264f-9022-4136-94e5-787d001652bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 17:09:28.075279 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:28.075250 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-istio-envoy\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:09:28.075279 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:28.075272 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-workload-certs\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:09:28.075279 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:28.075281 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6504264f-9022-4136-94e5-787d001652bb-istio-token\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:09:28.075279 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:28.075289 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6504264f-9022-4136-94e5-787d001652bb-istiod-ca-cert\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:09:28.075558 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:28.075297 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-istio-data\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:09:28.075558 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:28.075305 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6504264f-9022-4136-94e5-787d001652bb-workload-socket\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:09:28.075558 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:28.075313 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzzq9\" (UniqueName: \"kubernetes.io/projected/6504264f-9022-4136-94e5-787d001652bb-kube-api-access-xzzq9\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:09:28.075558 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:28.075322 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6504264f-9022-4136-94e5-787d001652bb-istio-podinfo\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:09:28.080203 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:28.080171 2574 generic.go:356] "Generic (PLEG): container finished" podID="6504264f-9022-4136-94e5-787d001652bb" containerID="5601e63daa32b403629c7ac312fc40e6469c34d3f46404b977308b974895bfa4" exitCode=0 Jul 06 17:09:28.080327 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:28.080216 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" Jul 06 17:09:28.080327 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:28.080250 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" event={"ID":"6504264f-9022-4136-94e5-787d001652bb","Type":"ContainerDied","Data":"5601e63daa32b403629c7ac312fc40e6469c34d3f46404b977308b974895bfa4"} Jul 06 17:09:28.080327 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:28.080307 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck" event={"ID":"6504264f-9022-4136-94e5-787d001652bb","Type":"ContainerDied","Data":"b496696a14b0bebcdd3e1dfccd73ae011d6ec0c6ca61b148aca16444b1fc1d0f"} Jul 06 17:09:28.080435 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:28.080338 2574 scope.go:117] "RemoveContainer" containerID="5601e63daa32b403629c7ac312fc40e6469c34d3f46404b977308b974895bfa4" Jul 06 17:09:28.090406 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:28.090386 2574 scope.go:117] "RemoveContainer" containerID="5601e63daa32b403629c7ac312fc40e6469c34d3f46404b977308b974895bfa4" Jul 06 17:09:28.090744 ip-10-0-131-71 kubenswrapper[2574]: E0706 17:09:28.090727 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5601e63daa32b403629c7ac312fc40e6469c34d3f46404b977308b974895bfa4\": container with ID starting with 5601e63daa32b403629c7ac312fc40e6469c34d3f46404b977308b974895bfa4 not found: ID does not exist" containerID="5601e63daa32b403629c7ac312fc40e6469c34d3f46404b977308b974895bfa4" Jul 06 17:09:28.090811 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:28.090753 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5601e63daa32b403629c7ac312fc40e6469c34d3f46404b977308b974895bfa4"} err="failed to get container status \"5601e63daa32b403629c7ac312fc40e6469c34d3f46404b977308b974895bfa4\": rpc error: code = NotFound desc = could not find container \"5601e63daa32b403629c7ac312fc40e6469c34d3f46404b977308b974895bfa4\": container with ID starting with 5601e63daa32b403629c7ac312fc40e6469c34d3f46404b977308b974895bfa4 not found: ID does not exist" Jul 06 17:09:28.104727 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:28.104697 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck"] Jul 06 17:09:28.108170 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:28.108147 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-38d19e47-openshift-default-8654df49cc-8tbck"] Jul 06 17:09:28.114243 ip-10-0-131-71 kubenswrapper[2574]: E0706 17:09:28.114220 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6504264f_9022_4136_94e5_787d001652bb.slice\": RecentStats: unable to find data in memory cache]" Jul 06 17:09:29.075841 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:29.075811 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6504264f-9022-4136-94e5-787d001652bb" path="/var/lib/kubelet/pods/6504264f-9022-4136-94e5-787d001652bb/volumes" Jul 06 17:09:33.108245 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:33.108212 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:09:43.099781 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:43.099739 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:09:48.819017 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.818928 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz"] Jul 06 17:09:48.819505 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.819485 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6504264f-9022-4136-94e5-787d001652bb" containerName="istio-proxy" Jul 06 17:09:48.819610 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.819509 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6504264f-9022-4136-94e5-787d001652bb" containerName="istio-proxy" Jul 06 17:09:48.819673 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.819665 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6504264f-9022-4136-94e5-787d001652bb" containerName="istio-proxy" Jul 06 17:09:48.822648 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.822629 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:48.826247 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.826226 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-8b33da08-openshift-default-dockercfg-lhs47\"" Jul 06 17:09:48.847950 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.847922 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz"] Jul 06 17:09:48.849208 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.849185 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-workload-socket\") pod \"e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:48.849324 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.849222 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26209a16-5b6c-463c-b205-5c27f0366fc1-istio-token\") pod \"e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:48.849324 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.849275 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xznvh\" (UniqueName: \"kubernetes.io/projected/26209a16-5b6c-463c-b205-5c27f0366fc1-kube-api-access-xznvh\") pod \"e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:48.849441 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.849348 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26209a16-5b6c-463c-b205-5c27f0366fc1-istiod-ca-cert\") pod \"e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:48.849441 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.849415 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-workload-certs\") pod \"e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:48.849551 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.849473 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-credential-socket\") pod \"e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:48.849551 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.849510 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26209a16-5b6c-463c-b205-5c27f0366fc1-istio-podinfo\") pod \"e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:48.849646 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.849634 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-istio-envoy\") pod \"e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:48.849700 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.849656 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-istio-data\") pod \"e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:48.950674 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.950637 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-credential-socket\") pod \"e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:48.950674 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.950675 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26209a16-5b6c-463c-b205-5c27f0366fc1-istio-podinfo\") pod \"e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:48.950916 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.950707 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-istio-envoy\") pod \"e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:48.950916 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.950820 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-istio-data\") pod \"e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:48.950916 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.950884 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-workload-socket\") pod \"e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:48.950916 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.950911 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26209a16-5b6c-463c-b205-5c27f0366fc1-istio-token\") pod \"e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:48.951135 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.950964 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xznvh\" (UniqueName: \"kubernetes.io/projected/26209a16-5b6c-463c-b205-5c27f0366fc1-kube-api-access-xznvh\") pod \"e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:48.951135 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.950993 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26209a16-5b6c-463c-b205-5c27f0366fc1-istiod-ca-cert\") pod \"e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:48.951135 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.951028 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-workload-certs\") pod \"e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:48.951299 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.951180 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-credential-socket\") pod \"e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:48.951299 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.951220 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-istio-data\") pod \"e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:48.951299 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.951273 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-workload-socket\") pod \"e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:48.951456 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.951395 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-workload-certs\") pod \"e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:48.951808 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.951787 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26209a16-5b6c-463c-b205-5c27f0366fc1-istiod-ca-cert\") pod \"e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:48.953063 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.953042 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-istio-envoy\") pod \"e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:48.953150 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.953137 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26209a16-5b6c-463c-b205-5c27f0366fc1-istio-podinfo\") pod \"e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:48.963792 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.963767 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xznvh\" (UniqueName: \"kubernetes.io/projected/26209a16-5b6c-463c-b205-5c27f0366fc1-kube-api-access-xznvh\") pod \"e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:48.963891 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:48.963815 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26209a16-5b6c-463c-b205-5c27f0366fc1-istio-token\") pod \"e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:49.132768 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:49.132666 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:49.277046 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:49.277024 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz"] Jul 06 17:09:49.279229 ip-10-0-131-71 kubenswrapper[2574]: W0706 17:09:49.279199 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26209a16_5b6c_463c_b205_5c27f0366fc1.slice/crio-431c9be9d4b4937d6af31340fc6654f8efd3e4b84a9c71afa2256c8dc55fe649 WatchSource:0}: Error finding container 431c9be9d4b4937d6af31340fc6654f8efd3e4b84a9c71afa2256c8dc55fe649: Status 404 returned error can't find the container with id 431c9be9d4b4937d6af31340fc6654f8efd3e4b84a9c71afa2256c8dc55fe649 Jul 06 17:09:49.281373 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:49.281345 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 17:09:49.281440 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:49.281411 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 17:09:49.281477 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:49.281438 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 06 17:09:50.198070 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:50.198029 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" event={"ID":"26209a16-5b6c-463c-b205-5c27f0366fc1","Type":"ContainerStarted","Data":"65146e556302947f06138516940b29152888197ba3106e065d52a2c7b9078127"} Jul 06 17:09:50.198456 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:50.198124 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" event={"ID":"26209a16-5b6c-463c-b205-5c27f0366fc1","Type":"ContainerStarted","Data":"431c9be9d4b4937d6af31340fc6654f8efd3e4b84a9c71afa2256c8dc55fe649"} Jul 06 17:09:50.227479 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:50.227423 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" podStartSLOduration=2.227403594 podStartE2EDuration="2.227403594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:09:50.225831832 +0000 UTC m=+1883.745322960" watchObservedRunningTime="2026-07-06 17:09:50.227403594 +0000 UTC m=+1883.746894711" Jul 06 17:09:51.133537 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:51.133486 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:51.139093 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:51.139068 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:51.202878 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:51.202843 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:51.203858 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:51.203842 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:09:53.097166 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:53.097138 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:09:54.817647 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:54.817610 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz"] Jul 06 17:09:54.818071 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:09:54.817834 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" podUID="26209a16-5b6c-463c-b205-5c27f0366fc1" containerName="istio-proxy" containerID="cri-o://65146e556302947f06138516940b29152888197ba3106e065d52a2c7b9078127" gracePeriod=30 Jul 06 17:10:00.868771 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:00.868744 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:10:00.968812 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:00.968780 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-istio-envoy\") pod \"26209a16-5b6c-463c-b205-5c27f0366fc1\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " Jul 06 17:10:00.968983 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:00.968865 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xznvh\" (UniqueName: \"kubernetes.io/projected/26209a16-5b6c-463c-b205-5c27f0366fc1-kube-api-access-xznvh\") pod \"26209a16-5b6c-463c-b205-5c27f0366fc1\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " Jul 06 17:10:00.968983 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:00.968891 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-workload-certs\") pod \"26209a16-5b6c-463c-b205-5c27f0366fc1\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " Jul 06 17:10:00.968983 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:00.968909 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26209a16-5b6c-463c-b205-5c27f0366fc1-istio-token\") pod \"26209a16-5b6c-463c-b205-5c27f0366fc1\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " Jul 06 17:10:00.968983 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:00.968934 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26209a16-5b6c-463c-b205-5c27f0366fc1-istiod-ca-cert\") pod \"26209a16-5b6c-463c-b205-5c27f0366fc1\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " Jul 06 17:10:00.968983 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:00.968953 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-credential-socket\") pod \"26209a16-5b6c-463c-b205-5c27f0366fc1\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " Jul 06 17:10:00.968983 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:00.968979 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-istio-data\") pod \"26209a16-5b6c-463c-b205-5c27f0366fc1\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " Jul 06 17:10:00.969280 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:00.969016 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26209a16-5b6c-463c-b205-5c27f0366fc1-istio-podinfo\") pod \"26209a16-5b6c-463c-b205-5c27f0366fc1\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " Jul 06 17:10:00.969280 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:00.969086 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-workload-socket\") pod \"26209a16-5b6c-463c-b205-5c27f0366fc1\" (UID: \"26209a16-5b6c-463c-b205-5c27f0366fc1\") " Jul 06 17:10:00.969392 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:00.969263 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "26209a16-5b6c-463c-b205-5c27f0366fc1" (UID: "26209a16-5b6c-463c-b205-5c27f0366fc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:10:00.969392 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:00.969362 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-workload-certs\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:10:00.969392 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:00.969369 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "26209a16-5b6c-463c-b205-5c27f0366fc1" (UID: "26209a16-5b6c-463c-b205-5c27f0366fc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:10:00.969593 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:00.969496 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26209a16-5b6c-463c-b205-5c27f0366fc1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "26209a16-5b6c-463c-b205-5c27f0366fc1" (UID: "26209a16-5b6c-463c-b205-5c27f0366fc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:10:00.969593 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:00.969580 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "26209a16-5b6c-463c-b205-5c27f0366fc1" (UID: "26209a16-5b6c-463c-b205-5c27f0366fc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:10:00.970203 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:00.970172 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-istio-data" (OuterVolumeSpecName: "istio-data") pod "26209a16-5b6c-463c-b205-5c27f0366fc1" (UID: "26209a16-5b6c-463c-b205-5c27f0366fc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:10:00.971179 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:00.971144 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26209a16-5b6c-463c-b205-5c27f0366fc1-kube-api-access-xznvh" (OuterVolumeSpecName: "kube-api-access-xznvh") pod "26209a16-5b6c-463c-b205-5c27f0366fc1" (UID: "26209a16-5b6c-463c-b205-5c27f0366fc1"). InnerVolumeSpecName "kube-api-access-xznvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:10:00.971301 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:00.971176 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26209a16-5b6c-463c-b205-5c27f0366fc1-istio-token" (OuterVolumeSpecName: "istio-token") pod "26209a16-5b6c-463c-b205-5c27f0366fc1" (UID: "26209a16-5b6c-463c-b205-5c27f0366fc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:10:00.971365 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:00.971342 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "26209a16-5b6c-463c-b205-5c27f0366fc1" (UID: "26209a16-5b6c-463c-b205-5c27f0366fc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:10:00.971449 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:00.971433 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26209a16-5b6c-463c-b205-5c27f0366fc1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "26209a16-5b6c-463c-b205-5c27f0366fc1" (UID: "26209a16-5b6c-463c-b205-5c27f0366fc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 17:10:01.070314 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:01.070275 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-workload-socket\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:10:01.070470 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:01.070338 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-istio-envoy\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:10:01.070470 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:01.070375 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xznvh\" (UniqueName: \"kubernetes.io/projected/26209a16-5b6c-463c-b205-5c27f0366fc1-kube-api-access-xznvh\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:10:01.070567 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:01.070476 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26209a16-5b6c-463c-b205-5c27f0366fc1-istio-token\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:10:01.070567 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:01.070551 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26209a16-5b6c-463c-b205-5c27f0366fc1-istiod-ca-cert\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:10:01.070679 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:01.070665 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-credential-socket\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:10:01.070748 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:01.070735 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26209a16-5b6c-463c-b205-5c27f0366fc1-istio-data\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:10:01.070786 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:01.070754 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26209a16-5b6c-463c-b205-5c27f0366fc1-istio-podinfo\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:10:01.253718 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:01.253621 2574 generic.go:356] "Generic (PLEG): container finished" podID="26209a16-5b6c-463c-b205-5c27f0366fc1" containerID="65146e556302947f06138516940b29152888197ba3106e065d52a2c7b9078127" exitCode=0 Jul 06 17:10:01.253718 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:01.253684 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" Jul 06 17:10:01.253718 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:01.253704 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" event={"ID":"26209a16-5b6c-463c-b205-5c27f0366fc1","Type":"ContainerDied","Data":"65146e556302947f06138516940b29152888197ba3106e065d52a2c7b9078127"} Jul 06 17:10:01.253944 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:01.253769 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz" event={"ID":"26209a16-5b6c-463c-b205-5c27f0366fc1","Type":"ContainerDied","Data":"431c9be9d4b4937d6af31340fc6654f8efd3e4b84a9c71afa2256c8dc55fe649"} Jul 06 17:10:01.253944 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:01.253798 2574 scope.go:117] "RemoveContainer" containerID="65146e556302947f06138516940b29152888197ba3106e065d52a2c7b9078127" Jul 06 17:10:01.262478 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:01.262452 2574 scope.go:117] "RemoveContainer" containerID="65146e556302947f06138516940b29152888197ba3106e065d52a2c7b9078127" Jul 06 17:10:01.262810 ip-10-0-131-71 kubenswrapper[2574]: E0706 17:10:01.262789 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65146e556302947f06138516940b29152888197ba3106e065d52a2c7b9078127\": container with ID starting with 65146e556302947f06138516940b29152888197ba3106e065d52a2c7b9078127 not found: ID does not exist" containerID="65146e556302947f06138516940b29152888197ba3106e065d52a2c7b9078127" Jul 06 17:10:01.262909 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:01.262817 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65146e556302947f06138516940b29152888197ba3106e065d52a2c7b9078127"} err="failed to get container status \"65146e556302947f06138516940b29152888197ba3106e065d52a2c7b9078127\": rpc error: code = NotFound desc = could not find container \"65146e556302947f06138516940b29152888197ba3106e065d52a2c7b9078127\": container with ID starting with 65146e556302947f06138516940b29152888197ba3106e065d52a2c7b9078127 not found: ID does not exist" Jul 06 17:10:01.274267 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:01.274243 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz"] Jul 06 17:10:01.278445 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:01.278423 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-8b33da08-openshift-default-6f7876d755-xhwlz"] Jul 06 17:10:03.076251 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:03.076217 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26209a16-5b6c-463c-b205-5c27f0366fc1" path="/var/lib/kubelet/pods/26209a16-5b6c-463c-b205-5c27f0366fc1/volumes" Jul 06 17:10:04.597871 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:04.597831 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:10:15.395243 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:10:15.395205 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:11:25.416182 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:11:25.416105 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:11:32.682629 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:11:32.682596 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:11:40.836044 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:11:40.836003 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:11:50.490996 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:11:50.490960 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:12:01.409867 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:01.409831 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:12:32.092695 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:32.092665 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:12:46.509634 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.509596 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8"] Jul 06 17:12:46.510184 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.510162 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26209a16-5b6c-463c-b205-5c27f0366fc1" containerName="istio-proxy" Jul 06 17:12:46.510240 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.510188 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="26209a16-5b6c-463c-b205-5c27f0366fc1" containerName="istio-proxy" Jul 06 17:12:46.510294 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.510282 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="26209a16-5b6c-463c-b205-5c27f0366fc1" containerName="istio-proxy" Jul 06 17:12:46.513446 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.513430 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" Jul 06 17:12:46.517960 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.517940 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8bfb9318\"/\"openshift-service-ca.crt\"" Jul 06 17:12:46.518160 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.518142 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8bfb9318\"/\"default-dockercfg-q6l7f\"" Jul 06 17:12:46.518236 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.518220 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-8bfb9318\"/\"kube-root-ca.crt\"" Jul 06 17:12:46.518342 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.518330 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-8bfb9318\"/\"e2e-lifecycle-model-8bfb9318-kserve-self-signed-certs\"" Jul 06 17:12:46.521860 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.521833 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8"] Jul 06 17:12:46.558261 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.558229 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7810a030-f72f-4901-9d9d-aa23af9251c5-model-cache\") pod \"e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8\" (UID: \"7810a030-f72f-4901-9d9d-aa23af9251c5\") " pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" Jul 06 17:12:46.558261 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.558263 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7810a030-f72f-4901-9d9d-aa23af9251c5-home\") pod \"e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8\" (UID: \"7810a030-f72f-4901-9d9d-aa23af9251c5\") " pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" Jul 06 17:12:46.558489 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.558290 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7810a030-f72f-4901-9d9d-aa23af9251c5-dshm\") pod \"e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8\" (UID: \"7810a030-f72f-4901-9d9d-aa23af9251c5\") " pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" Jul 06 17:12:46.558489 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.558408 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7810a030-f72f-4901-9d9d-aa23af9251c5-tmp-dir\") pod \"e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8\" (UID: \"7810a030-f72f-4901-9d9d-aa23af9251c5\") " pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" Jul 06 17:12:46.558489 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.558434 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7810a030-f72f-4901-9d9d-aa23af9251c5-tls-certs\") pod \"e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8\" (UID: \"7810a030-f72f-4901-9d9d-aa23af9251c5\") " pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" Jul 06 17:12:46.558489 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.558454 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7k9q\" (UniqueName: \"kubernetes.io/projected/7810a030-f72f-4901-9d9d-aa23af9251c5-kube-api-access-j7k9q\") pod \"e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8\" (UID: \"7810a030-f72f-4901-9d9d-aa23af9251c5\") " pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" Jul 06 17:12:46.659134 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.659096 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7810a030-f72f-4901-9d9d-aa23af9251c5-dshm\") pod \"e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8\" (UID: \"7810a030-f72f-4901-9d9d-aa23af9251c5\") " pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" Jul 06 17:12:46.659340 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.659252 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7810a030-f72f-4901-9d9d-aa23af9251c5-tmp-dir\") pod \"e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8\" (UID: \"7810a030-f72f-4901-9d9d-aa23af9251c5\") " pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" Jul 06 17:12:46.659340 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.659272 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7810a030-f72f-4901-9d9d-aa23af9251c5-tls-certs\") pod \"e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8\" (UID: \"7810a030-f72f-4901-9d9d-aa23af9251c5\") " pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" Jul 06 17:12:46.659340 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.659296 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7k9q\" (UniqueName: \"kubernetes.io/projected/7810a030-f72f-4901-9d9d-aa23af9251c5-kube-api-access-j7k9q\") pod \"e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8\" (UID: \"7810a030-f72f-4901-9d9d-aa23af9251c5\") " pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" Jul 06 17:12:46.659510 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.659359 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7810a030-f72f-4901-9d9d-aa23af9251c5-model-cache\") pod \"e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8\" (UID: \"7810a030-f72f-4901-9d9d-aa23af9251c5\") " pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" Jul 06 17:12:46.659510 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.659390 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7810a030-f72f-4901-9d9d-aa23af9251c5-home\") pod \"e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8\" (UID: \"7810a030-f72f-4901-9d9d-aa23af9251c5\") " pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" Jul 06 17:12:46.659849 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.659693 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7810a030-f72f-4901-9d9d-aa23af9251c5-tmp-dir\") pod \"e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8\" (UID: \"7810a030-f72f-4901-9d9d-aa23af9251c5\") " pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" Jul 06 17:12:46.659849 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.659727 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7810a030-f72f-4901-9d9d-aa23af9251c5-home\") pod \"e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8\" (UID: \"7810a030-f72f-4901-9d9d-aa23af9251c5\") " pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" Jul 06 17:12:46.659849 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.659801 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7810a030-f72f-4901-9d9d-aa23af9251c5-model-cache\") pod \"e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8\" (UID: \"7810a030-f72f-4901-9d9d-aa23af9251c5\") " pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" Jul 06 17:12:46.661629 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.661600 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7810a030-f72f-4901-9d9d-aa23af9251c5-dshm\") pod \"e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8\" (UID: \"7810a030-f72f-4901-9d9d-aa23af9251c5\") " pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" Jul 06 17:12:46.661952 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.661931 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7810a030-f72f-4901-9d9d-aa23af9251c5-tls-certs\") pod \"e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8\" (UID: \"7810a030-f72f-4901-9d9d-aa23af9251c5\") " pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" Jul 06 17:12:46.668567 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.668545 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7k9q\" (UniqueName: \"kubernetes.io/projected/7810a030-f72f-4901-9d9d-aa23af9251c5-kube-api-access-j7k9q\") pod \"e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8\" (UID: \"7810a030-f72f-4901-9d9d-aa23af9251c5\") " pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" Jul 06 17:12:46.824112 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.824001 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" Jul 06 17:12:46.968655 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:46.968628 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8"] Jul 06 17:12:46.973192 ip-10-0-131-71 kubenswrapper[2574]: W0706 17:12:46.973160 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7810a030_f72f_4901_9d9d_aa23af9251c5.slice/crio-5c0b77acefee0cf1b9ed998cde6316c8af703f8c9e60c3e74c3c5a751f7a5cf7 WatchSource:0}: Error finding container 5c0b77acefee0cf1b9ed998cde6316c8af703f8c9e60c3e74c3c5a751f7a5cf7: Status 404 returned error can't find the container with id 5c0b77acefee0cf1b9ed998cde6316c8af703f8c9e60c3e74c3c5a751f7a5cf7 Jul 06 17:12:47.023745 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:47.023712 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" event={"ID":"7810a030-f72f-4901-9d9d-aa23af9251c5","Type":"ContainerStarted","Data":"5c0b77acefee0cf1b9ed998cde6316c8af703f8c9e60c3e74c3c5a751f7a5cf7"} Jul 06 17:12:48.029278 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:48.029243 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" event={"ID":"7810a030-f72f-4901-9d9d-aa23af9251c5","Type":"ContainerStarted","Data":"af9a13497dbbde8578660e0fac410dacf1d7de1e93f919b66cd22e9c1804b438"} Jul 06 17:12:48.050752 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:48.050701 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" podStartSLOduration=2.050685565 podStartE2EDuration="2.050685565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:12:48.048583842 +0000 UTC m=+2061.568074960" watchObservedRunningTime="2026-07-06 17:12:48.050685565 +0000 UTC m=+2061.570176680" Jul 06 17:12:49.897830 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:49.897790 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:12:56.824624 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:56.824567 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" Jul 06 17:12:56.824624 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:56.824640 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" Jul 06 17:12:56.837348 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:56.837318 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" Jul 06 17:12:57.085236 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:12:57.085151 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" Jul 06 17:13:07.999709 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:07.999670 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:13:17.697583 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:17.697547 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:13:25.015451 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:25.015412 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8"] Jul 06 17:13:29.141266 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:29.141230 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:13:40.035822 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:40.035777 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" podUID="7810a030-f72f-4901-9d9d-aa23af9251c5" containerName="main" containerID="cri-o://af9a13497dbbde8578660e0fac410dacf1d7de1e93f919b66cd22e9c1804b438" gracePeriod=45 Jul 06 17:13:40.272755 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:40.272719 2574 generic.go:356] "Generic (PLEG): container finished" podID="7810a030-f72f-4901-9d9d-aa23af9251c5" containerID="af9a13497dbbde8578660e0fac410dacf1d7de1e93f919b66cd22e9c1804b438" exitCode=0 Jul 06 17:13:40.272911 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:40.272796 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" event={"ID":"7810a030-f72f-4901-9d9d-aa23af9251c5","Type":"ContainerDied","Data":"af9a13497dbbde8578660e0fac410dacf1d7de1e93f919b66cd22e9c1804b438"} Jul 06 17:13:40.293482 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:40.293421 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" Jul 06 17:13:40.344667 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:40.344629 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7810a030-f72f-4901-9d9d-aa23af9251c5-dshm\") pod \"7810a030-f72f-4901-9d9d-aa23af9251c5\" (UID: \"7810a030-f72f-4901-9d9d-aa23af9251c5\") " Jul 06 17:13:40.344667 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:40.344663 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7810a030-f72f-4901-9d9d-aa23af9251c5-tls-certs\") pod \"7810a030-f72f-4901-9d9d-aa23af9251c5\" (UID: \"7810a030-f72f-4901-9d9d-aa23af9251c5\") " Jul 06 17:13:40.344917 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:40.344691 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7k9q\" (UniqueName: \"kubernetes.io/projected/7810a030-f72f-4901-9d9d-aa23af9251c5-kube-api-access-j7k9q\") pod \"7810a030-f72f-4901-9d9d-aa23af9251c5\" (UID: \"7810a030-f72f-4901-9d9d-aa23af9251c5\") " Jul 06 17:13:40.344917 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:40.344729 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7810a030-f72f-4901-9d9d-aa23af9251c5-home\") pod \"7810a030-f72f-4901-9d9d-aa23af9251c5\" (UID: \"7810a030-f72f-4901-9d9d-aa23af9251c5\") " Jul 06 17:13:40.344917 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:40.344845 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7810a030-f72f-4901-9d9d-aa23af9251c5-tmp-dir\") pod \"7810a030-f72f-4901-9d9d-aa23af9251c5\" (UID: \"7810a030-f72f-4901-9d9d-aa23af9251c5\") " Jul 06 17:13:40.345081 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:40.344938 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7810a030-f72f-4901-9d9d-aa23af9251c5-model-cache\") pod \"7810a030-f72f-4901-9d9d-aa23af9251c5\" (UID: \"7810a030-f72f-4901-9d9d-aa23af9251c5\") " Jul 06 17:13:40.345081 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:40.344981 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7810a030-f72f-4901-9d9d-aa23af9251c5-home" (OuterVolumeSpecName: "home") pod "7810a030-f72f-4901-9d9d-aa23af9251c5" (UID: "7810a030-f72f-4901-9d9d-aa23af9251c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:13:40.345210 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:40.345188 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7810a030-f72f-4901-9d9d-aa23af9251c5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7810a030-f72f-4901-9d9d-aa23af9251c5" (UID: "7810a030-f72f-4901-9d9d-aa23af9251c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:13:40.345272 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:40.345246 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7810a030-f72f-4901-9d9d-aa23af9251c5-home\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:13:40.345272 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:40.345263 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7810a030-f72f-4901-9d9d-aa23af9251c5-tmp-dir\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:13:40.345378 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:40.345284 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7810a030-f72f-4901-9d9d-aa23af9251c5-model-cache" (OuterVolumeSpecName: "model-cache") pod "7810a030-f72f-4901-9d9d-aa23af9251c5" (UID: "7810a030-f72f-4901-9d9d-aa23af9251c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:13:40.347018 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:40.346988 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7810a030-f72f-4901-9d9d-aa23af9251c5-dshm" (OuterVolumeSpecName: "dshm") pod "7810a030-f72f-4901-9d9d-aa23af9251c5" (UID: "7810a030-f72f-4901-9d9d-aa23af9251c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:13:40.347320 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:40.347297 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7810a030-f72f-4901-9d9d-aa23af9251c5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7810a030-f72f-4901-9d9d-aa23af9251c5" (UID: "7810a030-f72f-4901-9d9d-aa23af9251c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:13:40.347430 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:40.347407 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7810a030-f72f-4901-9d9d-aa23af9251c5-kube-api-access-j7k9q" (OuterVolumeSpecName: "kube-api-access-j7k9q") pod "7810a030-f72f-4901-9d9d-aa23af9251c5" (UID: "7810a030-f72f-4901-9d9d-aa23af9251c5"). InnerVolumeSpecName "kube-api-access-j7k9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:13:40.446026 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:40.445986 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7810a030-f72f-4901-9d9d-aa23af9251c5-dshm\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:13:40.446026 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:40.446017 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7810a030-f72f-4901-9d9d-aa23af9251c5-tls-certs\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:13:40.446026 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:40.446031 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7k9q\" (UniqueName: \"kubernetes.io/projected/7810a030-f72f-4901-9d9d-aa23af9251c5-kube-api-access-j7k9q\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:13:40.446264 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:40.446044 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7810a030-f72f-4901-9d9d-aa23af9251c5-model-cache\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:13:41.279473 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:41.279441 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" Jul 06 17:13:41.279917 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:41.279469 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8" event={"ID":"7810a030-f72f-4901-9d9d-aa23af9251c5","Type":"ContainerDied","Data":"5c0b77acefee0cf1b9ed998cde6316c8af703f8c9e60c3e74c3c5a751f7a5cf7"} Jul 06 17:13:41.279917 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:41.279542 2574 scope.go:117] "RemoveContainer" containerID="af9a13497dbbde8578660e0fac410dacf1d7de1e93f919b66cd22e9c1804b438" Jul 06 17:13:41.300218 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:41.300189 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8"] Jul 06 17:13:41.304683 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:41.304659 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-8bfb9318/e2e-lifecycle-model-8bfb9318-kserve-57ccbfdddc-w2fz8"] Jul 06 17:13:43.076228 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:13:43.076191 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7810a030-f72f-4901-9d9d-aa23af9251c5" path="/var/lib/kubelet/pods/7810a030-f72f-4901-9d9d-aa23af9251c5/volumes" Jul 06 17:14:04.396498 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:14:04.396466 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:14:13.603870 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:14:13.603829 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:14:25.291396 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:14:25.291311 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:15:17.202869 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:17.202832 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:15:26.790925 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:26.790891 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z"] Jul 06 17:15:26.791560 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:26.791298 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7810a030-f72f-4901-9d9d-aa23af9251c5" containerName="main" Jul 06 17:15:26.791560 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:26.791317 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7810a030-f72f-4901-9d9d-aa23af9251c5" containerName="main" Jul 06 17:15:26.791560 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:26.791446 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7810a030-f72f-4901-9d9d-aa23af9251c5" containerName="main" Jul 06 17:15:26.794416 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:26.794396 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" Jul 06 17:15:26.798475 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:26.798427 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-cdb744\"/\"kube-root-ca.crt\"" Jul 06 17:15:26.798475 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:26.798425 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-cdb744\"/\"test-model-cdb744-kserve-self-signed-certs\"" Jul 06 17:15:26.798681 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:26.798426 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-cdb744\"/\"openshift-service-ca.crt\"" Jul 06 17:15:26.798681 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:26.798479 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-cdb744\"/\"default-dockercfg-7jhsh\"" Jul 06 17:15:26.804734 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:26.804710 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z"] Jul 06 17:15:26.904962 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:26.904930 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a2077b1-8781-49ec-8467-1206ada1de8e-home\") pod \"test-model-cdb744-kserve-dfb8cb449-tcq7z\" (UID: \"0a2077b1-8781-49ec-8467-1206ada1de8e\") " pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" Jul 06 17:15:26.904962 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:26.904963 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a2077b1-8781-49ec-8467-1206ada1de8e-dshm\") pod \"test-model-cdb744-kserve-dfb8cb449-tcq7z\" (UID: \"0a2077b1-8781-49ec-8467-1206ada1de8e\") " pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" Jul 06 17:15:26.905192 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:26.904992 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a2077b1-8781-49ec-8467-1206ada1de8e-tmp-dir\") pod \"test-model-cdb744-kserve-dfb8cb449-tcq7z\" (UID: \"0a2077b1-8781-49ec-8467-1206ada1de8e\") " pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" Jul 06 17:15:26.905192 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:26.905009 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a2077b1-8781-49ec-8467-1206ada1de8e-tls-certs\") pod \"test-model-cdb744-kserve-dfb8cb449-tcq7z\" (UID: \"0a2077b1-8781-49ec-8467-1206ada1de8e\") " pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" Jul 06 17:15:26.905192 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:26.905063 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a2077b1-8781-49ec-8467-1206ada1de8e-model-cache\") pod \"test-model-cdb744-kserve-dfb8cb449-tcq7z\" (UID: \"0a2077b1-8781-49ec-8467-1206ada1de8e\") " pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" Jul 06 17:15:26.905192 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:26.905083 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjzhb\" (UniqueName: \"kubernetes.io/projected/0a2077b1-8781-49ec-8467-1206ada1de8e-kube-api-access-pjzhb\") pod \"test-model-cdb744-kserve-dfb8cb449-tcq7z\" (UID: \"0a2077b1-8781-49ec-8467-1206ada1de8e\") " pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" Jul 06 17:15:27.005719 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:27.005684 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a2077b1-8781-49ec-8467-1206ada1de8e-model-cache\") pod \"test-model-cdb744-kserve-dfb8cb449-tcq7z\" (UID: \"0a2077b1-8781-49ec-8467-1206ada1de8e\") " pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" Jul 06 17:15:27.005892 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:27.005728 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjzhb\" (UniqueName: \"kubernetes.io/projected/0a2077b1-8781-49ec-8467-1206ada1de8e-kube-api-access-pjzhb\") pod \"test-model-cdb744-kserve-dfb8cb449-tcq7z\" (UID: \"0a2077b1-8781-49ec-8467-1206ada1de8e\") " pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" Jul 06 17:15:27.005892 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:27.005782 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a2077b1-8781-49ec-8467-1206ada1de8e-home\") pod \"test-model-cdb744-kserve-dfb8cb449-tcq7z\" (UID: \"0a2077b1-8781-49ec-8467-1206ada1de8e\") " pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" Jul 06 17:15:27.005892 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:27.005798 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a2077b1-8781-49ec-8467-1206ada1de8e-dshm\") pod \"test-model-cdb744-kserve-dfb8cb449-tcq7z\" (UID: \"0a2077b1-8781-49ec-8467-1206ada1de8e\") " pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" Jul 06 17:15:27.005892 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:27.005828 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a2077b1-8781-49ec-8467-1206ada1de8e-tmp-dir\") pod \"test-model-cdb744-kserve-dfb8cb449-tcq7z\" (UID: \"0a2077b1-8781-49ec-8467-1206ada1de8e\") " pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" Jul 06 17:15:27.005892 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:27.005853 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a2077b1-8781-49ec-8467-1206ada1de8e-tls-certs\") pod \"test-model-cdb744-kserve-dfb8cb449-tcq7z\" (UID: \"0a2077b1-8781-49ec-8467-1206ada1de8e\") " pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" Jul 06 17:15:27.006182 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:27.006158 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a2077b1-8781-49ec-8467-1206ada1de8e-model-cache\") pod \"test-model-cdb744-kserve-dfb8cb449-tcq7z\" (UID: \"0a2077b1-8781-49ec-8467-1206ada1de8e\") " pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" Jul 06 17:15:27.006246 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:27.006192 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a2077b1-8781-49ec-8467-1206ada1de8e-home\") pod \"test-model-cdb744-kserve-dfb8cb449-tcq7z\" (UID: \"0a2077b1-8781-49ec-8467-1206ada1de8e\") " pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" Jul 06 17:15:27.006289 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:27.006264 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a2077b1-8781-49ec-8467-1206ada1de8e-tmp-dir\") pod \"test-model-cdb744-kserve-dfb8cb449-tcq7z\" (UID: \"0a2077b1-8781-49ec-8467-1206ada1de8e\") " pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" Jul 06 17:15:27.008021 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:27.007994 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a2077b1-8781-49ec-8467-1206ada1de8e-dshm\") pod \"test-model-cdb744-kserve-dfb8cb449-tcq7z\" (UID: \"0a2077b1-8781-49ec-8467-1206ada1de8e\") " pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" Jul 06 17:15:27.010333 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:27.010315 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-cdb744\"/\"test-model-cdb744-kserve-self-signed-certs\"" Jul 06 17:15:27.016865 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:27.016846 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-cdb744\"/\"kube-root-ca.crt\"" Jul 06 17:15:27.018351 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:27.018336 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a2077b1-8781-49ec-8467-1206ada1de8e-tls-certs\") pod \"test-model-cdb744-kserve-dfb8cb449-tcq7z\" (UID: \"0a2077b1-8781-49ec-8467-1206ada1de8e\") " pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" Jul 06 17:15:27.027496 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:27.027477 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-cdb744\"/\"openshift-service-ca.crt\"" Jul 06 17:15:27.037667 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:27.037636 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjzhb\" (UniqueName: \"kubernetes.io/projected/0a2077b1-8781-49ec-8467-1206ada1de8e-kube-api-access-pjzhb\") pod \"test-model-cdb744-kserve-dfb8cb449-tcq7z\" (UID: \"0a2077b1-8781-49ec-8467-1206ada1de8e\") " pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" Jul 06 17:15:27.108440 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:27.108364 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-cdb744\"/\"default-dockercfg-7jhsh\"" Jul 06 17:15:27.116198 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:27.116179 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" Jul 06 17:15:27.275081 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:27.275060 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z"] Jul 06 17:15:27.277307 ip-10-0-131-71 kubenswrapper[2574]: W0706 17:15:27.277283 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a2077b1_8781_49ec_8467_1206ada1de8e.slice/crio-9ee25b80bf93d1a588ce60d0734056c061dfaea4546a78f938cabc7b35967f9a WatchSource:0}: Error finding container 9ee25b80bf93d1a588ce60d0734056c061dfaea4546a78f938cabc7b35967f9a: Status 404 returned error can't find the container with id 9ee25b80bf93d1a588ce60d0734056c061dfaea4546a78f938cabc7b35967f9a Jul 06 17:15:27.496023 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:27.495980 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:15:27.788420 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:27.788321 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" event={"ID":"0a2077b1-8781-49ec-8467-1206ada1de8e","Type":"ContainerStarted","Data":"2598f6708328b01261486301733d18e675167dcfc3bec6ac29d19ef01f4204d2"} Jul 06 17:15:27.788420 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:27.788364 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" event={"ID":"0a2077b1-8781-49ec-8467-1206ada1de8e","Type":"ContainerStarted","Data":"9ee25b80bf93d1a588ce60d0734056c061dfaea4546a78f938cabc7b35967f9a"} Jul 06 17:15:27.837653 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:27.837596 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" podStartSLOduration=1.837578997 podStartE2EDuration="1.837578997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:15:27.834724199 +0000 UTC m=+2221.354215336" watchObservedRunningTime="2026-07-06 17:15:27.837578997 +0000 UTC m=+2221.357070113" Jul 06 17:15:37.117093 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:37.117061 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" Jul 06 17:15:37.117093 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:37.117099 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" Jul 06 17:15:37.129544 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:37.129508 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" Jul 06 17:15:37.851550 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:37.851506 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" Jul 06 17:15:38.785031 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:38.784997 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:15:59.402585 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.402540 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz"] Jul 06 17:15:59.406402 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.406377 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" Jul 06 17:15:59.409226 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.409202 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ebaa5d\"/\"openshift-service-ca.crt\"" Jul 06 17:15:59.410397 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.410375 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ebaa5d\"/\"kube-root-ca.crt\"" Jul 06 17:15:59.410532 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.410463 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ebaa5d\"/\"test-model-ebaa5d-kserve-self-signed-certs\"" Jul 06 17:15:59.410584 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.410463 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ebaa5d\"/\"default-dockercfg-22vnq\"" Jul 06 17:15:59.415730 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.415706 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz"] Jul 06 17:15:59.510272 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.510232 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08d71c7-c449-4ec6-857f-f0a9e5231f80-tmp-dir\") pod \"test-model-ebaa5d-kserve-647d8c8657-7b7hz\" (UID: \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\") " pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" Jul 06 17:15:59.510272 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.510270 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldvx9\" (UniqueName: \"kubernetes.io/projected/f08d71c7-c449-4ec6-857f-f0a9e5231f80-kube-api-access-ldvx9\") pod \"test-model-ebaa5d-kserve-647d8c8657-7b7hz\" (UID: \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\") " pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" Jul 06 17:15:59.510543 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.510288 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08d71c7-c449-4ec6-857f-f0a9e5231f80-tls-certs\") pod \"test-model-ebaa5d-kserve-647d8c8657-7b7hz\" (UID: \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\") " pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" Jul 06 17:15:59.510543 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.510337 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08d71c7-c449-4ec6-857f-f0a9e5231f80-dshm\") pod \"test-model-ebaa5d-kserve-647d8c8657-7b7hz\" (UID: \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\") " pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" Jul 06 17:15:59.510543 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.510389 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08d71c7-c449-4ec6-857f-f0a9e5231f80-home\") pod \"test-model-ebaa5d-kserve-647d8c8657-7b7hz\" (UID: \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\") " pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" Jul 06 17:15:59.510543 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.510433 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08d71c7-c449-4ec6-857f-f0a9e5231f80-model-cache\") pod \"test-model-ebaa5d-kserve-647d8c8657-7b7hz\" (UID: \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\") " pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" Jul 06 17:15:59.611914 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.611868 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08d71c7-c449-4ec6-857f-f0a9e5231f80-home\") pod \"test-model-ebaa5d-kserve-647d8c8657-7b7hz\" (UID: \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\") " pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" Jul 06 17:15:59.612113 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.611941 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08d71c7-c449-4ec6-857f-f0a9e5231f80-model-cache\") pod \"test-model-ebaa5d-kserve-647d8c8657-7b7hz\" (UID: \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\") " pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" Jul 06 17:15:59.612113 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.612040 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08d71c7-c449-4ec6-857f-f0a9e5231f80-tmp-dir\") pod \"test-model-ebaa5d-kserve-647d8c8657-7b7hz\" (UID: \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\") " pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" Jul 06 17:15:59.612113 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.612064 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldvx9\" (UniqueName: \"kubernetes.io/projected/f08d71c7-c449-4ec6-857f-f0a9e5231f80-kube-api-access-ldvx9\") pod \"test-model-ebaa5d-kserve-647d8c8657-7b7hz\" (UID: \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\") " pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" Jul 06 17:15:59.612113 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.612090 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08d71c7-c449-4ec6-857f-f0a9e5231f80-tls-certs\") pod \"test-model-ebaa5d-kserve-647d8c8657-7b7hz\" (UID: \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\") " pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" Jul 06 17:15:59.612314 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.612123 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08d71c7-c449-4ec6-857f-f0a9e5231f80-dshm\") pod \"test-model-ebaa5d-kserve-647d8c8657-7b7hz\" (UID: \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\") " pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" Jul 06 17:15:59.612372 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.612316 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08d71c7-c449-4ec6-857f-f0a9e5231f80-home\") pod \"test-model-ebaa5d-kserve-647d8c8657-7b7hz\" (UID: \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\") " pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" Jul 06 17:15:59.612429 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.612372 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08d71c7-c449-4ec6-857f-f0a9e5231f80-model-cache\") pod \"test-model-ebaa5d-kserve-647d8c8657-7b7hz\" (UID: \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\") " pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" Jul 06 17:15:59.612429 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.612397 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08d71c7-c449-4ec6-857f-f0a9e5231f80-tmp-dir\") pod \"test-model-ebaa5d-kserve-647d8c8657-7b7hz\" (UID: \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\") " pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" Jul 06 17:15:59.614398 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.614377 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08d71c7-c449-4ec6-857f-f0a9e5231f80-dshm\") pod \"test-model-ebaa5d-kserve-647d8c8657-7b7hz\" (UID: \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\") " pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" Jul 06 17:15:59.614734 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.614718 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08d71c7-c449-4ec6-857f-f0a9e5231f80-tls-certs\") pod \"test-model-ebaa5d-kserve-647d8c8657-7b7hz\" (UID: \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\") " pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" Jul 06 17:15:59.620431 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.620413 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldvx9\" (UniqueName: \"kubernetes.io/projected/f08d71c7-c449-4ec6-857f-f0a9e5231f80-kube-api-access-ldvx9\") pod \"test-model-ebaa5d-kserve-647d8c8657-7b7hz\" (UID: \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\") " pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" Jul 06 17:15:59.717398 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.717361 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" Jul 06 17:15:59.856044 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.856021 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz"] Jul 06 17:15:59.858398 ip-10-0-131-71 kubenswrapper[2574]: W0706 17:15:59.858368 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf08d71c7_c449_4ec6_857f_f0a9e5231f80.slice/crio-ae617a6e313521e0d20fbe7541350fceae10435638a10f45eb573d3dba632e23 WatchSource:0}: Error finding container ae617a6e313521e0d20fbe7541350fceae10435638a10f45eb573d3dba632e23: Status 404 returned error can't find the container with id ae617a6e313521e0d20fbe7541350fceae10435638a10f45eb573d3dba632e23 Jul 06 17:15:59.956154 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.956117 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" event={"ID":"f08d71c7-c449-4ec6-857f-f0a9e5231f80","Type":"ContainerStarted","Data":"8f39b222d6e2f8baa89e6447ff56c0cf004d677f90155ba721e1275e883415de"} Jul 06 17:15:59.956334 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.956163 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" event={"ID":"f08d71c7-c449-4ec6-857f-f0a9e5231f80","Type":"ContainerStarted","Data":"ae617a6e313521e0d20fbe7541350fceae10435638a10f45eb573d3dba632e23"} Jul 06 17:15:59.974142 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:15:59.974043 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" podStartSLOduration=0.974026615 podStartE2EDuration="974.026615ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:15:59.973318262 +0000 UTC m=+2253.492809403" watchObservedRunningTime="2026-07-06 17:15:59.974026615 +0000 UTC m=+2253.493517730" Jul 06 17:16:03.093297 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:03.093239 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:16:09.717702 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:09.717660 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" Jul 06 17:16:09.718229 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:09.717718 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" Jul 06 17:16:09.730290 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:09.730261 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" Jul 06 17:16:10.015036 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:10.014953 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" Jul 06 17:16:14.597762 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:14.597720 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:16:30.780965 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:30.780931 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z"] Jul 06 17:16:31.776459 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:31.776417 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz"] Jul 06 17:16:35.691397 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:35.691363 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:16:45.801293 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:45.801239 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" podUID="0a2077b1-8781-49ec-8467-1206ada1de8e" containerName="main" containerID="cri-o://2598f6708328b01261486301733d18e675167dcfc3bec6ac29d19ef01f4204d2" gracePeriod=45 Jul 06 17:16:46.064724 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.064661 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" Jul 06 17:16:46.128035 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.128002 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a2077b1-8781-49ec-8467-1206ada1de8e-dshm\") pod \"0a2077b1-8781-49ec-8467-1206ada1de8e\" (UID: \"0a2077b1-8781-49ec-8467-1206ada1de8e\") " Jul 06 17:16:46.128035 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.128046 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a2077b1-8781-49ec-8467-1206ada1de8e-tls-certs\") pod \"0a2077b1-8781-49ec-8467-1206ada1de8e\" (UID: \"0a2077b1-8781-49ec-8467-1206ada1de8e\") " Jul 06 17:16:46.128299 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.128085 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a2077b1-8781-49ec-8467-1206ada1de8e-tmp-dir\") pod \"0a2077b1-8781-49ec-8467-1206ada1de8e\" (UID: \"0a2077b1-8781-49ec-8467-1206ada1de8e\") " Jul 06 17:16:46.128299 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.128118 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a2077b1-8781-49ec-8467-1206ada1de8e-home\") pod \"0a2077b1-8781-49ec-8467-1206ada1de8e\" (UID: \"0a2077b1-8781-49ec-8467-1206ada1de8e\") " Jul 06 17:16:46.128299 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.128162 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a2077b1-8781-49ec-8467-1206ada1de8e-model-cache\") pod \"0a2077b1-8781-49ec-8467-1206ada1de8e\" (UID: \"0a2077b1-8781-49ec-8467-1206ada1de8e\") " Jul 06 17:16:46.128299 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.128238 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjzhb\" (UniqueName: \"kubernetes.io/projected/0a2077b1-8781-49ec-8467-1206ada1de8e-kube-api-access-pjzhb\") pod \"0a2077b1-8781-49ec-8467-1206ada1de8e\" (UID: \"0a2077b1-8781-49ec-8467-1206ada1de8e\") " Jul 06 17:16:46.128510 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.128408 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2077b1-8781-49ec-8467-1206ada1de8e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0a2077b1-8781-49ec-8467-1206ada1de8e" (UID: "0a2077b1-8781-49ec-8467-1206ada1de8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:16:46.128510 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.128434 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2077b1-8781-49ec-8467-1206ada1de8e-home" (OuterVolumeSpecName: "home") pod "0a2077b1-8781-49ec-8467-1206ada1de8e" (UID: "0a2077b1-8781-49ec-8467-1206ada1de8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:16:46.128625 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.128507 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2077b1-8781-49ec-8467-1206ada1de8e-model-cache" (OuterVolumeSpecName: "model-cache") pod "0a2077b1-8781-49ec-8467-1206ada1de8e" (UID: "0a2077b1-8781-49ec-8467-1206ada1de8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:16:46.128665 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.128622 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a2077b1-8781-49ec-8467-1206ada1de8e-tmp-dir\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:16:46.128665 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.128641 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a2077b1-8781-49ec-8467-1206ada1de8e-home\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:16:46.128665 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.128653 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a2077b1-8781-49ec-8467-1206ada1de8e-model-cache\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:16:46.130408 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.130378 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a2077b1-8781-49ec-8467-1206ada1de8e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0a2077b1-8781-49ec-8467-1206ada1de8e" (UID: "0a2077b1-8781-49ec-8467-1206ada1de8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:16:46.130540 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.130429 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2077b1-8781-49ec-8467-1206ada1de8e-dshm" (OuterVolumeSpecName: "dshm") pod "0a2077b1-8781-49ec-8467-1206ada1de8e" (UID: "0a2077b1-8781-49ec-8467-1206ada1de8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:16:46.130540 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.130449 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2077b1-8781-49ec-8467-1206ada1de8e-kube-api-access-pjzhb" (OuterVolumeSpecName: "kube-api-access-pjzhb") pod "0a2077b1-8781-49ec-8467-1206ada1de8e" (UID: "0a2077b1-8781-49ec-8467-1206ada1de8e"). InnerVolumeSpecName "kube-api-access-pjzhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:16:46.173142 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.173102 2574 generic.go:356] "Generic (PLEG): container finished" podID="0a2077b1-8781-49ec-8467-1206ada1de8e" containerID="2598f6708328b01261486301733d18e675167dcfc3bec6ac29d19ef01f4204d2" exitCode=0 Jul 06 17:16:46.173298 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.173150 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" Jul 06 17:16:46.173298 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.173192 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" event={"ID":"0a2077b1-8781-49ec-8467-1206ada1de8e","Type":"ContainerDied","Data":"2598f6708328b01261486301733d18e675167dcfc3bec6ac29d19ef01f4204d2"} Jul 06 17:16:46.173298 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.173240 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z" event={"ID":"0a2077b1-8781-49ec-8467-1206ada1de8e","Type":"ContainerDied","Data":"9ee25b80bf93d1a588ce60d0734056c061dfaea4546a78f938cabc7b35967f9a"} Jul 06 17:16:46.173298 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.173257 2574 scope.go:117] "RemoveContainer" containerID="2598f6708328b01261486301733d18e675167dcfc3bec6ac29d19ef01f4204d2" Jul 06 17:16:46.181781 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.181757 2574 scope.go:117] "RemoveContainer" containerID="2598f6708328b01261486301733d18e675167dcfc3bec6ac29d19ef01f4204d2" Jul 06 17:16:46.182051 ip-10-0-131-71 kubenswrapper[2574]: E0706 17:16:46.182031 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2598f6708328b01261486301733d18e675167dcfc3bec6ac29d19ef01f4204d2\": container with ID starting with 2598f6708328b01261486301733d18e675167dcfc3bec6ac29d19ef01f4204d2 not found: ID does not exist" containerID="2598f6708328b01261486301733d18e675167dcfc3bec6ac29d19ef01f4204d2" Jul 06 17:16:46.182121 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.182059 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2598f6708328b01261486301733d18e675167dcfc3bec6ac29d19ef01f4204d2"} err="failed to get container status \"2598f6708328b01261486301733d18e675167dcfc3bec6ac29d19ef01f4204d2\": rpc error: code = NotFound desc = could not find container \"2598f6708328b01261486301733d18e675167dcfc3bec6ac29d19ef01f4204d2\": container with ID starting with 2598f6708328b01261486301733d18e675167dcfc3bec6ac29d19ef01f4204d2 not found: ID does not exist" Jul 06 17:16:46.195723 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.195691 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z"] Jul 06 17:16:46.199773 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.199752 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-cdb744/test-model-cdb744-kserve-dfb8cb449-tcq7z"] Jul 06 17:16:46.229122 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.229090 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjzhb\" (UniqueName: \"kubernetes.io/projected/0a2077b1-8781-49ec-8467-1206ada1de8e-kube-api-access-pjzhb\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:16:46.229122 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.229116 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a2077b1-8781-49ec-8467-1206ada1de8e-dshm\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:16:46.229272 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.229128 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a2077b1-8781-49ec-8467-1206ada1de8e-tls-certs\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:16:46.796824 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:46.796768 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" podUID="f08d71c7-c449-4ec6-857f-f0a9e5231f80" containerName="main" containerID="cri-o://8f39b222d6e2f8baa89e6447ff56c0cf004d677f90155ba721e1275e883415de" gracePeriod=45 Jul 06 17:16:47.060103 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.060046 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" Jul 06 17:16:47.076220 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.076182 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a2077b1-8781-49ec-8467-1206ada1de8e" path="/var/lib/kubelet/pods/0a2077b1-8781-49ec-8467-1206ada1de8e/volumes" Jul 06 17:16:47.137347 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.137309 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldvx9\" (UniqueName: \"kubernetes.io/projected/f08d71c7-c449-4ec6-857f-f0a9e5231f80-kube-api-access-ldvx9\") pod \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\" (UID: \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\") " Jul 06 17:16:47.137504 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.137378 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08d71c7-c449-4ec6-857f-f0a9e5231f80-tls-certs\") pod \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\" (UID: \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\") " Jul 06 17:16:47.137504 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.137429 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08d71c7-c449-4ec6-857f-f0a9e5231f80-dshm\") pod \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\" (UID: \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\") " Jul 06 17:16:47.137504 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.137471 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08d71c7-c449-4ec6-857f-f0a9e5231f80-model-cache\") pod \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\" (UID: \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\") " Jul 06 17:16:47.137718 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.137576 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08d71c7-c449-4ec6-857f-f0a9e5231f80-home\") pod \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\" (UID: \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\") " Jul 06 17:16:47.137718 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.137611 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08d71c7-c449-4ec6-857f-f0a9e5231f80-tmp-dir\") pod \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\" (UID: \"f08d71c7-c449-4ec6-857f-f0a9e5231f80\") " Jul 06 17:16:47.137987 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.137945 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08d71c7-c449-4ec6-857f-f0a9e5231f80-home" (OuterVolumeSpecName: "home") pod "f08d71c7-c449-4ec6-857f-f0a9e5231f80" (UID: "f08d71c7-c449-4ec6-857f-f0a9e5231f80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:16:47.138103 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.138000 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08d71c7-c449-4ec6-857f-f0a9e5231f80-model-cache" (OuterVolumeSpecName: "model-cache") pod "f08d71c7-c449-4ec6-857f-f0a9e5231f80" (UID: "f08d71c7-c449-4ec6-857f-f0a9e5231f80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:16:47.138103 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.138073 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08d71c7-c449-4ec6-857f-f0a9e5231f80-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f08d71c7-c449-4ec6-857f-f0a9e5231f80" (UID: "f08d71c7-c449-4ec6-857f-f0a9e5231f80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:16:47.140148 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.140118 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f08d71c7-c449-4ec6-857f-f0a9e5231f80-dshm" (OuterVolumeSpecName: "dshm") pod "f08d71c7-c449-4ec6-857f-f0a9e5231f80" (UID: "f08d71c7-c449-4ec6-857f-f0a9e5231f80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:16:47.140276 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.140168 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f08d71c7-c449-4ec6-857f-f0a9e5231f80-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f08d71c7-c449-4ec6-857f-f0a9e5231f80" (UID: "f08d71c7-c449-4ec6-857f-f0a9e5231f80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:16:47.140276 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.140170 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08d71c7-c449-4ec6-857f-f0a9e5231f80-kube-api-access-ldvx9" (OuterVolumeSpecName: "kube-api-access-ldvx9") pod "f08d71c7-c449-4ec6-857f-f0a9e5231f80" (UID: "f08d71c7-c449-4ec6-857f-f0a9e5231f80"). InnerVolumeSpecName "kube-api-access-ldvx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:16:47.179563 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.179511 2574 generic.go:356] "Generic (PLEG): container finished" podID="f08d71c7-c449-4ec6-857f-f0a9e5231f80" containerID="8f39b222d6e2f8baa89e6447ff56c0cf004d677f90155ba721e1275e883415de" exitCode=0 Jul 06 17:16:47.179563 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.179561 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" Jul 06 17:16:47.179806 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.179593 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" event={"ID":"f08d71c7-c449-4ec6-857f-f0a9e5231f80","Type":"ContainerDied","Data":"8f39b222d6e2f8baa89e6447ff56c0cf004d677f90155ba721e1275e883415de"} Jul 06 17:16:47.179806 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.179636 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz" event={"ID":"f08d71c7-c449-4ec6-857f-f0a9e5231f80","Type":"ContainerDied","Data":"ae617a6e313521e0d20fbe7541350fceae10435638a10f45eb573d3dba632e23"} Jul 06 17:16:47.179806 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.179659 2574 scope.go:117] "RemoveContainer" containerID="8f39b222d6e2f8baa89e6447ff56c0cf004d677f90155ba721e1275e883415de" Jul 06 17:16:47.188998 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.188977 2574 scope.go:117] "RemoveContainer" containerID="8f39b222d6e2f8baa89e6447ff56c0cf004d677f90155ba721e1275e883415de" Jul 06 17:16:47.189235 ip-10-0-131-71 kubenswrapper[2574]: E0706 17:16:47.189216 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f39b222d6e2f8baa89e6447ff56c0cf004d677f90155ba721e1275e883415de\": container with ID starting with 8f39b222d6e2f8baa89e6447ff56c0cf004d677f90155ba721e1275e883415de not found: ID does not exist" containerID="8f39b222d6e2f8baa89e6447ff56c0cf004d677f90155ba721e1275e883415de" Jul 06 17:16:47.189287 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.189243 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f39b222d6e2f8baa89e6447ff56c0cf004d677f90155ba721e1275e883415de"} err="failed to get container status \"8f39b222d6e2f8baa89e6447ff56c0cf004d677f90155ba721e1275e883415de\": rpc error: code = NotFound desc = could not find container \"8f39b222d6e2f8baa89e6447ff56c0cf004d677f90155ba721e1275e883415de\": container with ID starting with 8f39b222d6e2f8baa89e6447ff56c0cf004d677f90155ba721e1275e883415de not found: ID does not exist" Jul 06 17:16:47.201712 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.201687 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz"] Jul 06 17:16:47.207054 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.207033 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ebaa5d/test-model-ebaa5d-kserve-647d8c8657-7b7hz"] Jul 06 17:16:47.239225 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.239200 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f08d71c7-c449-4ec6-857f-f0a9e5231f80-home\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:16:47.239225 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.239223 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f08d71c7-c449-4ec6-857f-f0a9e5231f80-tmp-dir\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:16:47.239388 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.239232 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldvx9\" (UniqueName: \"kubernetes.io/projected/f08d71c7-c449-4ec6-857f-f0a9e5231f80-kube-api-access-ldvx9\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:16:47.239388 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.239242 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f08d71c7-c449-4ec6-857f-f0a9e5231f80-tls-certs\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:16:47.239388 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.239252 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f08d71c7-c449-4ec6-857f-f0a9e5231f80-dshm\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:16:47.239388 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:47.239261 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f08d71c7-c449-4ec6-857f-f0a9e5231f80-model-cache\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:16:49.075885 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:16:49.075848 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f08d71c7-c449-4ec6-857f-f0a9e5231f80" path="/var/lib/kubelet/pods/f08d71c7-c449-4ec6-857f-f0a9e5231f80/volumes" Jul 06 17:17:19.090988 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:17:19.090903 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:17:33.305997 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:17:33.305957 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:17:45.304926 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:17:45.304891 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:17:58.007382 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:17:58.007351 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:19:00.795479 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:00.795436 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:19:12.365936 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:12.365898 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:19:30.925671 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:30.925633 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx"] Jul 06 17:19:30.926289 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:30.925993 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a2077b1-8781-49ec-8467-1206ada1de8e" containerName="main" Jul 06 17:19:30.926289 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:30.926004 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a2077b1-8781-49ec-8467-1206ada1de8e" containerName="main" Jul 06 17:19:30.926289 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:30.926019 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f08d71c7-c449-4ec6-857f-f0a9e5231f80" containerName="main" Jul 06 17:19:30.926289 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:30.926024 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f08d71c7-c449-4ec6-857f-f0a9e5231f80" containerName="main" Jul 06 17:19:30.926289 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:30.926081 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f08d71c7-c449-4ec6-857f-f0a9e5231f80" containerName="main" Jul 06 17:19:30.926289 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:30.926090 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a2077b1-8781-49ec-8467-1206ada1de8e" containerName="main" Jul 06 17:19:30.929472 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:30.929450 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" Jul 06 17:19:30.934317 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:30.934281 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2cd1eb\"/\"kube-root-ca.crt\"" Jul 06 17:19:30.935667 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:30.935642 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2cd1eb\"/\"default-dockercfg-slxr4\"" Jul 06 17:19:30.935798 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:30.935696 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2cd1eb\"/\"auth-test-model-2cd1eb-kserve-self-signed-certs\"" Jul 06 17:19:30.935798 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:30.935729 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2cd1eb\"/\"openshift-service-ca.crt\"" Jul 06 17:19:30.941667 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:30.941640 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx"] Jul 06 17:19:31.069391 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:31.069348 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-tls-certs\") pod \"auth-test-model-2cd1eb-kserve-66579d489f-9xdqx\" (UID: \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\") " pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" Jul 06 17:19:31.069391 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:31.069392 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-model-cache\") pod \"auth-test-model-2cd1eb-kserve-66579d489f-9xdqx\" (UID: \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\") " pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" Jul 06 17:19:31.069652 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:31.069426 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-tmp-dir\") pod \"auth-test-model-2cd1eb-kserve-66579d489f-9xdqx\" (UID: \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\") " pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" Jul 06 17:19:31.069652 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:31.069466 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-dshm\") pod \"auth-test-model-2cd1eb-kserve-66579d489f-9xdqx\" (UID: \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\") " pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" Jul 06 17:19:31.069652 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:31.069543 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgdnp\" (UniqueName: \"kubernetes.io/projected/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-kube-api-access-cgdnp\") pod \"auth-test-model-2cd1eb-kserve-66579d489f-9xdqx\" (UID: \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\") " pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" Jul 06 17:19:31.069652 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:31.069601 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-home\") pod \"auth-test-model-2cd1eb-kserve-66579d489f-9xdqx\" (UID: \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\") " pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" Jul 06 17:19:31.170156 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:31.170111 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgdnp\" (UniqueName: \"kubernetes.io/projected/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-kube-api-access-cgdnp\") pod \"auth-test-model-2cd1eb-kserve-66579d489f-9xdqx\" (UID: \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\") " pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" Jul 06 17:19:31.170365 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:31.170176 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-home\") pod \"auth-test-model-2cd1eb-kserve-66579d489f-9xdqx\" (UID: \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\") " pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" Jul 06 17:19:31.170365 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:31.170243 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-tls-certs\") pod \"auth-test-model-2cd1eb-kserve-66579d489f-9xdqx\" (UID: \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\") " pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" Jul 06 17:19:31.170365 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:31.170268 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-model-cache\") pod \"auth-test-model-2cd1eb-kserve-66579d489f-9xdqx\" (UID: \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\") " pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" Jul 06 17:19:31.170365 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:31.170296 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-tmp-dir\") pod \"auth-test-model-2cd1eb-kserve-66579d489f-9xdqx\" (UID: \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\") " pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" Jul 06 17:19:31.170365 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:31.170338 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-dshm\") pod \"auth-test-model-2cd1eb-kserve-66579d489f-9xdqx\" (UID: \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\") " pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" Jul 06 17:19:31.170712 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:31.170685 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-home\") pod \"auth-test-model-2cd1eb-kserve-66579d489f-9xdqx\" (UID: \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\") " pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" Jul 06 17:19:31.170796 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:31.170754 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-model-cache\") pod \"auth-test-model-2cd1eb-kserve-66579d489f-9xdqx\" (UID: \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\") " pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" Jul 06 17:19:31.170853 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:31.170801 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-tmp-dir\") pod \"auth-test-model-2cd1eb-kserve-66579d489f-9xdqx\" (UID: \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\") " pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" Jul 06 17:19:31.172651 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:31.172632 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-dshm\") pod \"auth-test-model-2cd1eb-kserve-66579d489f-9xdqx\" (UID: \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\") " pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" Jul 06 17:19:31.173034 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:31.173014 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-tls-certs\") pod \"auth-test-model-2cd1eb-kserve-66579d489f-9xdqx\" (UID: \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\") " pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" Jul 06 17:19:31.180166 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:31.180109 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgdnp\" (UniqueName: \"kubernetes.io/projected/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-kube-api-access-cgdnp\") pod \"auth-test-model-2cd1eb-kserve-66579d489f-9xdqx\" (UID: \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\") " pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" Jul 06 17:19:31.240281 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:31.240246 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" Jul 06 17:19:31.373080 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:31.373056 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx"] Jul 06 17:19:31.375204 ip-10-0-131-71 kubenswrapper[2574]: W0706 17:19:31.375164 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0b5754a_85d4_4a66_b13f_ff8aa3f6dd0b.slice/crio-27e7e43a72add94e67891471bfa552fb00230fe296a86b6804bb8ab7268fbb9f WatchSource:0}: Error finding container 27e7e43a72add94e67891471bfa552fb00230fe296a86b6804bb8ab7268fbb9f: Status 404 returned error can't find the container with id 27e7e43a72add94e67891471bfa552fb00230fe296a86b6804bb8ab7268fbb9f Jul 06 17:19:31.945295 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:31.945250 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" event={"ID":"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b","Type":"ContainerStarted","Data":"e5053e1ec1017a24922499a25bb9bb4c633d0a0ed1017e79240fc33160389137"} Jul 06 17:19:31.945737 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:31.945302 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" event={"ID":"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b","Type":"ContainerStarted","Data":"27e7e43a72add94e67891471bfa552fb00230fe296a86b6804bb8ab7268fbb9f"} Jul 06 17:19:31.970181 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:31.970132 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" podStartSLOduration=1.970116334 podStartE2EDuration="1.970116334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:19:31.968767286 +0000 UTC m=+2465.488258402" watchObservedRunningTime="2026-07-06 17:19:31.970116334 +0000 UTC m=+2465.489607449" Jul 06 17:19:34.197192 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:34.197163 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:19:41.240381 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:41.240343 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" Jul 06 17:19:41.240875 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:41.240398 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" Jul 06 17:19:41.253243 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:41.253213 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" Jul 06 17:19:42.001804 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:19:42.001767 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" Jul 06 17:20:16.409648 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:16.409613 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr"] Jul 06 17:20:16.413014 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:16.412998 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" Jul 06 17:20:16.416956 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:16.416932 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0ebdd9\"/\"kube-root-ca.crt\"" Jul 06 17:20:16.416956 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:16.416932 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0ebdd9\"/\"openshift-service-ca.crt\"" Jul 06 17:20:16.416956 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:16.416936 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0ebdd9\"/\"default-dockercfg-8nmft\"" Jul 06 17:20:16.417192 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:16.417178 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0ebdd9\"/\"auth-test-model-0ebdd9-kserve-self-signed-certs\"" Jul 06 17:20:16.422908 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:16.422887 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr"] Jul 06 17:20:16.473118 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:16.473088 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-tls-certs\") pod \"auth-test-model-0ebdd9-kserve-78fc759979-tbxpr\" (UID: \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\") " pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" Jul 06 17:20:16.473291 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:16.473135 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p44jp\" (UniqueName: \"kubernetes.io/projected/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-kube-api-access-p44jp\") pod \"auth-test-model-0ebdd9-kserve-78fc759979-tbxpr\" (UID: \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\") " pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" Jul 06 17:20:16.473291 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:16.473247 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-dshm\") pod \"auth-test-model-0ebdd9-kserve-78fc759979-tbxpr\" (UID: \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\") " pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" Jul 06 17:20:16.473291 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:16.473272 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-model-cache\") pod \"auth-test-model-0ebdd9-kserve-78fc759979-tbxpr\" (UID: \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\") " pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" Jul 06 17:20:16.473427 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:16.473308 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-home\") pod \"auth-test-model-0ebdd9-kserve-78fc759979-tbxpr\" (UID: \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\") " pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" Jul 06 17:20:16.473427 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:16.473352 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-tmp-dir\") pod \"auth-test-model-0ebdd9-kserve-78fc759979-tbxpr\" (UID: \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\") " pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" Jul 06 17:20:16.574315 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:16.574278 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-dshm\") pod \"auth-test-model-0ebdd9-kserve-78fc759979-tbxpr\" (UID: \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\") " pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" Jul 06 17:20:16.574315 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:16.574315 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-model-cache\") pod \"auth-test-model-0ebdd9-kserve-78fc759979-tbxpr\" (UID: \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\") " pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" Jul 06 17:20:16.574599 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:16.574337 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-home\") pod \"auth-test-model-0ebdd9-kserve-78fc759979-tbxpr\" (UID: \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\") " pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" Jul 06 17:20:16.574599 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:16.574542 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-tmp-dir\") pod \"auth-test-model-0ebdd9-kserve-78fc759979-tbxpr\" (UID: \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\") " pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" Jul 06 17:20:16.574599 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:16.574596 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-tls-certs\") pod \"auth-test-model-0ebdd9-kserve-78fc759979-tbxpr\" (UID: \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\") " pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" Jul 06 17:20:16.574758 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:16.574628 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p44jp\" (UniqueName: \"kubernetes.io/projected/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-kube-api-access-p44jp\") pod \"auth-test-model-0ebdd9-kserve-78fc759979-tbxpr\" (UID: \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\") " pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" Jul 06 17:20:16.574758 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:16.574740 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-model-cache\") pod \"auth-test-model-0ebdd9-kserve-78fc759979-tbxpr\" (UID: \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\") " pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" Jul 06 17:20:16.574859 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:16.574802 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-home\") pod \"auth-test-model-0ebdd9-kserve-78fc759979-tbxpr\" (UID: \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\") " pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" Jul 06 17:20:16.574897 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:16.574875 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-tmp-dir\") pod \"auth-test-model-0ebdd9-kserve-78fc759979-tbxpr\" (UID: \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\") " pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" Jul 06 17:20:16.576723 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:16.576689 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-dshm\") pod \"auth-test-model-0ebdd9-kserve-78fc759979-tbxpr\" (UID: \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\") " pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" Jul 06 17:20:16.576981 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:16.576964 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-tls-certs\") pod \"auth-test-model-0ebdd9-kserve-78fc759979-tbxpr\" (UID: \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\") " pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" Jul 06 17:20:16.582677 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:16.582654 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p44jp\" (UniqueName: \"kubernetes.io/projected/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-kube-api-access-p44jp\") pod \"auth-test-model-0ebdd9-kserve-78fc759979-tbxpr\" (UID: \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\") " pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" Jul 06 17:20:16.722964 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:16.722931 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" Jul 06 17:20:16.858569 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:16.858541 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr"] Jul 06 17:20:16.860618 ip-10-0-131-71 kubenswrapper[2574]: W0706 17:20:16.860592 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9abfaa10_affe_4cea_8ae3_d1b0a4dbeb39.slice/crio-c90f321dab28b5f26eaa7c885658c0504bcc3e48c77ccc40e32e791d8eb4e6cf WatchSource:0}: Error finding container c90f321dab28b5f26eaa7c885658c0504bcc3e48c77ccc40e32e791d8eb4e6cf: Status 404 returned error can't find the container with id c90f321dab28b5f26eaa7c885658c0504bcc3e48c77ccc40e32e791d8eb4e6cf Jul 06 17:20:17.153806 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:17.153715 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" event={"ID":"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39","Type":"ContainerStarted","Data":"58813186383d4e8a5cc8ed229b423a31a929292668d0bb15ca868f7561e851ca"} Jul 06 17:20:17.153806 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:17.153769 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" event={"ID":"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39","Type":"ContainerStarted","Data":"c90f321dab28b5f26eaa7c885658c0504bcc3e48c77ccc40e32e791d8eb4e6cf"} Jul 06 17:20:17.177461 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:17.177406 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" podStartSLOduration=1.1773906219999999 podStartE2EDuration="1.177390622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:20:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:20:17.175844721 +0000 UTC m=+2510.695335839" watchObservedRunningTime="2026-07-06 17:20:17.177390622 +0000 UTC m=+2510.696881798" Jul 06 17:20:20.001058 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:20.000974 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:20:26.723626 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:26.723584 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" Jul 06 17:20:26.723626 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:26.723637 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" Jul 06 17:20:26.736436 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:26.736411 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" Jul 06 17:20:27.221760 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:27.221734 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" Jul 06 17:20:38.595965 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:38.595933 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:20:52.202013 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:20:52.201973 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:21:08.996084 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:21:08.996046 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:21:26.488842 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:21:26.488809 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:21:44.397019 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:21:44.396983 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:22:01.309167 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:01.309129 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:22:14.869888 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:14.869855 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx"] Jul 06 17:22:15.839710 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:15.839677 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr"] Jul 06 17:22:18.591005 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:18.590971 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:22:29.889500 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:29.889432 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" podUID="f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b" containerName="main" containerID="cri-o://e5053e1ec1017a24922499a25bb9bb4c633d0a0ed1017e79240fc33160389137" gracePeriod=45 Jul 06 17:22:30.141271 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.141208 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" Jul 06 17:22:30.286411 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.286382 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-dshm\") pod \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\" (UID: \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\") " Jul 06 17:22:30.286609 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.286420 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-model-cache\") pod \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\" (UID: \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\") " Jul 06 17:22:30.286609 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.286476 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-tls-certs\") pod \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\" (UID: \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\") " Jul 06 17:22:30.286609 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.286498 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-tmp-dir\") pod \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\" (UID: \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\") " Jul 06 17:22:30.286609 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.286585 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-home\") pod \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\" (UID: \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\") " Jul 06 17:22:30.286836 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.286618 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgdnp\" (UniqueName: \"kubernetes.io/projected/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-kube-api-access-cgdnp\") pod \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\" (UID: \"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b\") " Jul 06 17:22:30.286836 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.286720 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-model-cache" (OuterVolumeSpecName: "model-cache") pod "f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b" (UID: "f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:22:30.286944 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.286867 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-model-cache\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:22:30.286944 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.286864 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b" (UID: "f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:22:30.286944 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.286883 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-home" (OuterVolumeSpecName: "home") pod "f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b" (UID: "f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:22:30.288586 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.288560 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-dshm" (OuterVolumeSpecName: "dshm") pod "f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b" (UID: "f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:22:30.288698 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.288590 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b" (UID: "f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:22:30.288698 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.288623 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-kube-api-access-cgdnp" (OuterVolumeSpecName: "kube-api-access-cgdnp") pod "f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b" (UID: "f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b"). InnerVolumeSpecName "kube-api-access-cgdnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:22:30.387931 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.387894 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-home\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:22:30.387931 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.387924 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgdnp\" (UniqueName: \"kubernetes.io/projected/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-kube-api-access-cgdnp\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:22:30.387931 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.387935 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-dshm\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:22:30.388163 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.387944 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-tls-certs\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:22:30.388163 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.387952 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b-tmp-dir\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:22:30.793284 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.793244 2574 generic.go:356] "Generic (PLEG): container finished" podID="f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b" containerID="e5053e1ec1017a24922499a25bb9bb4c633d0a0ed1017e79240fc33160389137" exitCode=0 Jul 06 17:22:30.793284 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.793281 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" Jul 06 17:22:30.793573 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.793329 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" event={"ID":"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b","Type":"ContainerDied","Data":"e5053e1ec1017a24922499a25bb9bb4c633d0a0ed1017e79240fc33160389137"} Jul 06 17:22:30.793573 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.793373 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx" event={"ID":"f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b","Type":"ContainerDied","Data":"27e7e43a72add94e67891471bfa552fb00230fe296a86b6804bb8ab7268fbb9f"} Jul 06 17:22:30.793573 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.793393 2574 scope.go:117] "RemoveContainer" containerID="e5053e1ec1017a24922499a25bb9bb4c633d0a0ed1017e79240fc33160389137" Jul 06 17:22:30.802066 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.802049 2574 scope.go:117] "RemoveContainer" containerID="e5053e1ec1017a24922499a25bb9bb4c633d0a0ed1017e79240fc33160389137" Jul 06 17:22:30.802311 ip-10-0-131-71 kubenswrapper[2574]: E0706 17:22:30.802287 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5053e1ec1017a24922499a25bb9bb4c633d0a0ed1017e79240fc33160389137\": container with ID starting with e5053e1ec1017a24922499a25bb9bb4c633d0a0ed1017e79240fc33160389137 not found: ID does not exist" containerID="e5053e1ec1017a24922499a25bb9bb4c633d0a0ed1017e79240fc33160389137" Jul 06 17:22:30.802361 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.802320 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5053e1ec1017a24922499a25bb9bb4c633d0a0ed1017e79240fc33160389137"} err="failed to get container status \"e5053e1ec1017a24922499a25bb9bb4c633d0a0ed1017e79240fc33160389137\": rpc error: code = NotFound desc = could not find container \"e5053e1ec1017a24922499a25bb9bb4c633d0a0ed1017e79240fc33160389137\": container with ID starting with e5053e1ec1017a24922499a25bb9bb4c633d0a0ed1017e79240fc33160389137 not found: ID does not exist" Jul 06 17:22:30.818684 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.818656 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx"] Jul 06 17:22:30.822662 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.822643 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-2cd1eb/auth-test-model-2cd1eb-kserve-66579d489f-9xdqx"] Jul 06 17:22:30.859503 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:30.859436 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" podUID="9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39" containerName="main" containerID="cri-o://58813186383d4e8a5cc8ed229b423a31a929292668d0bb15ca868f7561e851ca" gracePeriod=45 Jul 06 17:22:31.077279 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.077202 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b" path="/var/lib/kubelet/pods/f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b/volumes" Jul 06 17:22:31.121980 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.121954 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" Jul 06 17:22:31.195482 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.195433 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-dshm\") pod \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\" (UID: \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\") " Jul 06 17:22:31.195670 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.195584 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-model-cache\") pod \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\" (UID: \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\") " Jul 06 17:22:31.195670 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.195633 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p44jp\" (UniqueName: \"kubernetes.io/projected/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-kube-api-access-p44jp\") pod \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\" (UID: \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\") " Jul 06 17:22:31.195670 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.195659 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-tmp-dir\") pod \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\" (UID: \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\") " Jul 06 17:22:31.195850 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.195691 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-home\") pod \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\" (UID: \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\") " Jul 06 17:22:31.195850 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.195721 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-tls-certs\") pod \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\" (UID: \"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39\") " Jul 06 17:22:31.195959 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.195858 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-model-cache" (OuterVolumeSpecName: "model-cache") pod "9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39" (UID: "9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:22:31.195959 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.195926 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39" (UID: "9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:22:31.195959 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.195951 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-home" (OuterVolumeSpecName: "home") pod "9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39" (UID: "9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:22:31.196124 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.196071 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-model-cache\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:22:31.196124 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.196095 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-tmp-dir\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:22:31.196124 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.196113 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-home\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:22:31.197816 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.197791 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-dshm" (OuterVolumeSpecName: "dshm") pod "9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39" (UID: "9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:22:31.197816 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.197804 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39" (UID: "9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:22:31.197960 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.197882 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-kube-api-access-p44jp" (OuterVolumeSpecName: "kube-api-access-p44jp") pod "9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39" (UID: "9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39"). InnerVolumeSpecName "kube-api-access-p44jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:22:31.296618 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.296573 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p44jp\" (UniqueName: \"kubernetes.io/projected/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-kube-api-access-p44jp\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:22:31.296618 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.296614 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-tls-certs\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:22:31.296618 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.296634 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39-dshm\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:22:31.799678 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.799642 2574 generic.go:356] "Generic (PLEG): container finished" podID="9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39" containerID="58813186383d4e8a5cc8ed229b423a31a929292668d0bb15ca868f7561e851ca" exitCode=0 Jul 06 17:22:31.799897 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.799693 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" Jul 06 17:22:31.799897 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.799774 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" event={"ID":"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39","Type":"ContainerDied","Data":"58813186383d4e8a5cc8ed229b423a31a929292668d0bb15ca868f7561e851ca"} Jul 06 17:22:31.799897 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.799809 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr" event={"ID":"9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39","Type":"ContainerDied","Data":"c90f321dab28b5f26eaa7c885658c0504bcc3e48c77ccc40e32e791d8eb4e6cf"} Jul 06 17:22:31.799897 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.799824 2574 scope.go:117] "RemoveContainer" containerID="58813186383d4e8a5cc8ed229b423a31a929292668d0bb15ca868f7561e851ca" Jul 06 17:22:31.810904 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.810866 2574 scope.go:117] "RemoveContainer" containerID="58813186383d4e8a5cc8ed229b423a31a929292668d0bb15ca868f7561e851ca" Jul 06 17:22:31.811323 ip-10-0-131-71 kubenswrapper[2574]: E0706 17:22:31.811301 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58813186383d4e8a5cc8ed229b423a31a929292668d0bb15ca868f7561e851ca\": container with ID starting with 58813186383d4e8a5cc8ed229b423a31a929292668d0bb15ca868f7561e851ca not found: ID does not exist" containerID="58813186383d4e8a5cc8ed229b423a31a929292668d0bb15ca868f7561e851ca" Jul 06 17:22:31.811424 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.811331 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58813186383d4e8a5cc8ed229b423a31a929292668d0bb15ca868f7561e851ca"} err="failed to get container status \"58813186383d4e8a5cc8ed229b423a31a929292668d0bb15ca868f7561e851ca\": rpc error: code = NotFound desc = could not find container \"58813186383d4e8a5cc8ed229b423a31a929292668d0bb15ca868f7561e851ca\": container with ID starting with 58813186383d4e8a5cc8ed229b423a31a929292668d0bb15ca868f7561e851ca not found: ID does not exist" Jul 06 17:22:31.825673 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.825645 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr"] Jul 06 17:22:31.832801 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:31.832777 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-0ebdd9/auth-test-model-0ebdd9-kserve-78fc759979-tbxpr"] Jul 06 17:22:33.076288 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:33.076255 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39" path="/var/lib/kubelet/pods/9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39/volumes" Jul 06 17:22:41.157378 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.157340 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd"] Jul 06 17:22:41.157918 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.157885 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39" containerName="main" Jul 06 17:22:41.157918 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.157905 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39" containerName="main" Jul 06 17:22:41.158040 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.157939 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b" containerName="main" Jul 06 17:22:41.158040 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.157949 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b" containerName="main" Jul 06 17:22:41.158135 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.158048 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9abfaa10-affe-4cea-8ae3-d1b0a4dbeb39" containerName="main" Jul 06 17:22:41.158135 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.158062 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0b5754a-85d4-4a66-b13f-ff8aa3f6dd0b" containerName="main" Jul 06 17:22:41.161077 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.161054 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" Jul 06 17:22:41.193347 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.193323 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2cd1eb\"/\"kube-root-ca.crt\"" Jul 06 17:22:41.193459 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.193324 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2cd1eb\"/\"openshift-service-ca.crt\"" Jul 06 17:22:41.210272 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.210239 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2cd1eb\"/\"sub-test-model-2cd1eb-kserve-self-signed-certs\"" Jul 06 17:22:41.233149 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.233123 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd"] Jul 06 17:22:41.239723 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.239699 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2cd1eb\"/\"default-dockercfg-slxr4\"" Jul 06 17:22:41.281542 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.281482 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-home\") pod \"sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd\" (UID: \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\") " pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" Jul 06 17:22:41.281748 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.281594 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-tls-certs\") pod \"sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd\" (UID: \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\") " pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" Jul 06 17:22:41.281748 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.281624 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-dshm\") pod \"sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd\" (UID: \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\") " pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" Jul 06 17:22:41.281748 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.281663 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-tmp-dir\") pod \"sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd\" (UID: \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\") " pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" Jul 06 17:22:41.281748 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.281684 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4fdq\" (UniqueName: \"kubernetes.io/projected/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-kube-api-access-r4fdq\") pod \"sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd\" (UID: \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\") " pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" Jul 06 17:22:41.281748 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.281718 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-model-cache\") pod \"sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd\" (UID: \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\") " pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" Jul 06 17:22:41.382835 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.382786 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-dshm\") pod \"sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd\" (UID: \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\") " pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" Jul 06 17:22:41.382835 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.382844 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-tmp-dir\") pod \"sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd\" (UID: \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\") " pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" Jul 06 17:22:41.383085 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.382866 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4fdq\" (UniqueName: \"kubernetes.io/projected/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-kube-api-access-r4fdq\") pod \"sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd\" (UID: \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\") " pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" Jul 06 17:22:41.383085 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.382885 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-model-cache\") pod \"sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd\" (UID: \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\") " pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" Jul 06 17:22:41.383085 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.382925 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-home\") pod \"sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd\" (UID: \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\") " pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" Jul 06 17:22:41.383085 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.383005 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-tls-certs\") pod \"sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd\" (UID: \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\") " pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" Jul 06 17:22:41.383312 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.383288 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-tmp-dir\") pod \"sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd\" (UID: \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\") " pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" Jul 06 17:22:41.383312 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.383299 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-model-cache\") pod \"sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd\" (UID: \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\") " pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" Jul 06 17:22:41.383464 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.383368 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-home\") pod \"sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd\" (UID: \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\") " pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" Jul 06 17:22:41.385062 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.385037 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-dshm\") pod \"sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd\" (UID: \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\") " pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" Jul 06 17:22:41.385370 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.385351 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-tls-certs\") pod \"sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd\" (UID: \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\") " pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" Jul 06 17:22:41.413287 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.413216 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4fdq\" (UniqueName: \"kubernetes.io/projected/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-kube-api-access-r4fdq\") pod \"sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd\" (UID: \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\") " pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" Jul 06 17:22:41.470410 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.470370 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" Jul 06 17:22:41.621503 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.621364 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd"] Jul 06 17:22:41.624207 ip-10-0-131-71 kubenswrapper[2574]: W0706 17:22:41.624180 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode593d1c0_082b_4fec_a3ee_f22d04d24b6e.slice/crio-99860d1728072f060dc961822a4f32bd1cc1242409d174f15b2ff94ee047eb21 WatchSource:0}: Error finding container 99860d1728072f060dc961822a4f32bd1cc1242409d174f15b2ff94ee047eb21: Status 404 returned error can't find the container with id 99860d1728072f060dc961822a4f32bd1cc1242409d174f15b2ff94ee047eb21 Jul 06 17:22:41.850725 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.850691 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" event={"ID":"e593d1c0-082b-4fec-a3ee-f22d04d24b6e","Type":"ContainerStarted","Data":"98d13b3528410b0afab9b154a072369dcc453c27b168468f26121661cdc49828"} Jul 06 17:22:41.850725 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.850730 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" event={"ID":"e593d1c0-082b-4fec-a3ee-f22d04d24b6e","Type":"ContainerStarted","Data":"99860d1728072f060dc961822a4f32bd1cc1242409d174f15b2ff94ee047eb21"} Jul 06 17:22:41.878833 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:41.878781 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" podStartSLOduration=0.878765572 podStartE2EDuration="878.765572ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:22:41.877748891 +0000 UTC m=+2655.397240008" watchObservedRunningTime="2026-07-06 17:22:41.878765572 +0000 UTC m=+2655.398256682" Jul 06 17:22:44.830511 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:44.830477 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:22:51.471530 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:51.471472 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" Jul 06 17:22:51.471933 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:51.471548 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" Jul 06 17:22:51.484400 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:51.484373 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" Jul 06 17:22:51.916683 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:22:51.916595 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" Jul 06 17:23:01.840265 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:01.840226 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn"] Jul 06 17:23:01.843953 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:01.843935 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" Jul 06 17:23:01.847913 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:01.847885 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0ebdd9\"/\"sub-test-model-0ebdd9-kserve-self-signed-certs\"" Jul 06 17:23:01.847913 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:01.847907 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0ebdd9\"/\"openshift-service-ca.crt\"" Jul 06 17:23:01.848088 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:01.847924 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0ebdd9\"/\"default-dockercfg-8nmft\"" Jul 06 17:23:01.848088 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:01.847937 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0ebdd9\"/\"kube-root-ca.crt\"" Jul 06 17:23:01.863329 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:01.863301 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn"] Jul 06 17:23:01.872667 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:01.872638 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6c02a6-3dca-41b6-a26d-278842600342-model-cache\") pod \"sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn\" (UID: \"ce6c02a6-3dca-41b6-a26d-278842600342\") " pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" Jul 06 17:23:01.872804 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:01.872699 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce6c02a6-3dca-41b6-a26d-278842600342-home\") pod \"sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn\" (UID: \"ce6c02a6-3dca-41b6-a26d-278842600342\") " pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" Jul 06 17:23:01.872804 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:01.872720 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce6c02a6-3dca-41b6-a26d-278842600342-dshm\") pod \"sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn\" (UID: \"ce6c02a6-3dca-41b6-a26d-278842600342\") " pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" Jul 06 17:23:01.872804 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:01.872759 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce6c02a6-3dca-41b6-a26d-278842600342-tmp-dir\") pod \"sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn\" (UID: \"ce6c02a6-3dca-41b6-a26d-278842600342\") " pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" Jul 06 17:23:01.872804 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:01.872796 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce6c02a6-3dca-41b6-a26d-278842600342-tls-certs\") pod \"sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn\" (UID: \"ce6c02a6-3dca-41b6-a26d-278842600342\") " pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" Jul 06 17:23:01.872952 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:01.872811 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftb8l\" (UniqueName: \"kubernetes.io/projected/ce6c02a6-3dca-41b6-a26d-278842600342-kube-api-access-ftb8l\") pod \"sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn\" (UID: \"ce6c02a6-3dca-41b6-a26d-278842600342\") " pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" Jul 06 17:23:01.974028 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:01.973992 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce6c02a6-3dca-41b6-a26d-278842600342-tls-certs\") pod \"sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn\" (UID: \"ce6c02a6-3dca-41b6-a26d-278842600342\") " pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" Jul 06 17:23:01.974028 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:01.974030 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftb8l\" (UniqueName: \"kubernetes.io/projected/ce6c02a6-3dca-41b6-a26d-278842600342-kube-api-access-ftb8l\") pod \"sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn\" (UID: \"ce6c02a6-3dca-41b6-a26d-278842600342\") " pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" Jul 06 17:23:01.974295 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:01.974084 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6c02a6-3dca-41b6-a26d-278842600342-model-cache\") pod \"sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn\" (UID: \"ce6c02a6-3dca-41b6-a26d-278842600342\") " pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" Jul 06 17:23:01.974295 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:01.974145 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce6c02a6-3dca-41b6-a26d-278842600342-home\") pod \"sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn\" (UID: \"ce6c02a6-3dca-41b6-a26d-278842600342\") " pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" Jul 06 17:23:01.974295 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:01.974168 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce6c02a6-3dca-41b6-a26d-278842600342-dshm\") pod \"sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn\" (UID: \"ce6c02a6-3dca-41b6-a26d-278842600342\") " pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" Jul 06 17:23:01.974295 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:01.974192 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce6c02a6-3dca-41b6-a26d-278842600342-tmp-dir\") pod \"sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn\" (UID: \"ce6c02a6-3dca-41b6-a26d-278842600342\") " pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" Jul 06 17:23:01.974635 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:01.974609 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce6c02a6-3dca-41b6-a26d-278842600342-home\") pod \"sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn\" (UID: \"ce6c02a6-3dca-41b6-a26d-278842600342\") " pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" Jul 06 17:23:01.974635 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:01.974626 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6c02a6-3dca-41b6-a26d-278842600342-model-cache\") pod \"sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn\" (UID: \"ce6c02a6-3dca-41b6-a26d-278842600342\") " pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" Jul 06 17:23:01.974851 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:01.974687 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce6c02a6-3dca-41b6-a26d-278842600342-tmp-dir\") pod \"sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn\" (UID: \"ce6c02a6-3dca-41b6-a26d-278842600342\") " pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" Jul 06 17:23:01.976428 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:01.976407 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce6c02a6-3dca-41b6-a26d-278842600342-dshm\") pod \"sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn\" (UID: \"ce6c02a6-3dca-41b6-a26d-278842600342\") " pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" Jul 06 17:23:01.976567 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:01.976549 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce6c02a6-3dca-41b6-a26d-278842600342-tls-certs\") pod \"sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn\" (UID: \"ce6c02a6-3dca-41b6-a26d-278842600342\") " pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" Jul 06 17:23:01.985020 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:01.984995 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftb8l\" (UniqueName: \"kubernetes.io/projected/ce6c02a6-3dca-41b6-a26d-278842600342-kube-api-access-ftb8l\") pod \"sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn\" (UID: \"ce6c02a6-3dca-41b6-a26d-278842600342\") " pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" Jul 06 17:23:02.154954 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:02.154828 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" Jul 06 17:23:02.292785 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:02.292718 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn"] Jul 06 17:23:02.296246 ip-10-0-131-71 kubenswrapper[2574]: W0706 17:23:02.296219 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce6c02a6_3dca_41b6_a26d_278842600342.slice/crio-f087c1496180e7391efb9ae78b8e003854e62b73482aa27947899c9eb0e2e431 WatchSource:0}: Error finding container f087c1496180e7391efb9ae78b8e003854e62b73482aa27947899c9eb0e2e431: Status 404 returned error can't find the container with id f087c1496180e7391efb9ae78b8e003854e62b73482aa27947899c9eb0e2e431 Jul 06 17:23:02.960414 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:02.960377 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" event={"ID":"ce6c02a6-3dca-41b6-a26d-278842600342","Type":"ContainerStarted","Data":"80429c2655bbe3b46194d357bc0db265364a477934bab034cc4912b15c920337"} Jul 06 17:23:02.960414 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:02.960421 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" event={"ID":"ce6c02a6-3dca-41b6-a26d-278842600342","Type":"ContainerStarted","Data":"f087c1496180e7391efb9ae78b8e003854e62b73482aa27947899c9eb0e2e431"} Jul 06 17:23:02.990402 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:02.990348 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" podStartSLOduration=1.990328031 podStartE2EDuration="1.990328031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:23:02.990223469 +0000 UTC m=+2676.509714587" watchObservedRunningTime="2026-07-06 17:23:02.990328031 +0000 UTC m=+2676.509819148" Jul 06 17:23:06.024559 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:06.024526 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:23:12.156026 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:12.155978 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" Jul 06 17:23:12.156026 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:12.156033 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" Jul 06 17:23:12.168735 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:12.168706 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" Jul 06 17:23:13.019575 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:13.019545 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" Jul 06 17:23:20.160920 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:20.160843 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:23:38.823990 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:38.823951 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:23:45.554819 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:45.554783 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd"] Jul 06 17:23:46.472768 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:46.472735 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn"] Jul 06 17:23:58.401850 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:23:58.401811 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:24:00.574510 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:00.574468 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" podUID="e593d1c0-082b-4fec-a3ee-f22d04d24b6e" containerName="main" containerID="cri-o://98d13b3528410b0afab9b154a072369dcc453c27b168468f26121661cdc49828" gracePeriod=45 Jul 06 17:24:00.827787 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:00.827723 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" Jul 06 17:24:00.916268 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:00.916228 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-tls-certs\") pod \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\" (UID: \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\") " Jul 06 17:24:00.916462 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:00.916299 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-tmp-dir\") pod \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\" (UID: \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\") " Jul 06 17:24:00.916462 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:00.916348 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4fdq\" (UniqueName: \"kubernetes.io/projected/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-kube-api-access-r4fdq\") pod \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\" (UID: \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\") " Jul 06 17:24:00.916462 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:00.916405 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-home\") pod \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\" (UID: \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\") " Jul 06 17:24:00.916711 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:00.916463 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-dshm\") pod \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\" (UID: \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\") " Jul 06 17:24:00.916711 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:00.916487 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-model-cache\") pod \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\" (UID: \"e593d1c0-082b-4fec-a3ee-f22d04d24b6e\") " Jul 06 17:24:00.916711 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:00.916622 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e593d1c0-082b-4fec-a3ee-f22d04d24b6e" (UID: "e593d1c0-082b-4fec-a3ee-f22d04d24b6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:24:00.916874 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:00.916721 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-home" (OuterVolumeSpecName: "home") pod "e593d1c0-082b-4fec-a3ee-f22d04d24b6e" (UID: "e593d1c0-082b-4fec-a3ee-f22d04d24b6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:24:00.916874 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:00.916773 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-model-cache" (OuterVolumeSpecName: "model-cache") pod "e593d1c0-082b-4fec-a3ee-f22d04d24b6e" (UID: "e593d1c0-082b-4fec-a3ee-f22d04d24b6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:24:00.916874 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:00.916857 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-home\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:24:00.916874 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:00.916872 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-tmp-dir\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:24:00.918487 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:00.918466 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e593d1c0-082b-4fec-a3ee-f22d04d24b6e" (UID: "e593d1c0-082b-4fec-a3ee-f22d04d24b6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:24:00.918602 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:00.918536 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-kube-api-access-r4fdq" (OuterVolumeSpecName: "kube-api-access-r4fdq") pod "e593d1c0-082b-4fec-a3ee-f22d04d24b6e" (UID: "e593d1c0-082b-4fec-a3ee-f22d04d24b6e"). InnerVolumeSpecName "kube-api-access-r4fdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:24:00.918602 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:00.918556 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-dshm" (OuterVolumeSpecName: "dshm") pod "e593d1c0-082b-4fec-a3ee-f22d04d24b6e" (UID: "e593d1c0-082b-4fec-a3ee-f22d04d24b6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:24:01.017529 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.017492 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-dshm\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:24:01.017702 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.017541 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-model-cache\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:24:01.017702 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.017552 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-tls-certs\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:24:01.017702 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.017560 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4fdq\" (UniqueName: \"kubernetes.io/projected/e593d1c0-082b-4fec-a3ee-f22d04d24b6e-kube-api-access-r4fdq\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:24:01.235348 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.235307 2574 generic.go:356] "Generic (PLEG): container finished" podID="e593d1c0-082b-4fec-a3ee-f22d04d24b6e" containerID="98d13b3528410b0afab9b154a072369dcc453c27b168468f26121661cdc49828" exitCode=0 Jul 06 17:24:01.235348 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.235360 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" Jul 06 17:24:01.235678 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.235392 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" event={"ID":"e593d1c0-082b-4fec-a3ee-f22d04d24b6e","Type":"ContainerDied","Data":"98d13b3528410b0afab9b154a072369dcc453c27b168468f26121661cdc49828"} Jul 06 17:24:01.235678 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.235452 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd" event={"ID":"e593d1c0-082b-4fec-a3ee-f22d04d24b6e","Type":"ContainerDied","Data":"99860d1728072f060dc961822a4f32bd1cc1242409d174f15b2ff94ee047eb21"} Jul 06 17:24:01.235678 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.235476 2574 scope.go:117] "RemoveContainer" containerID="98d13b3528410b0afab9b154a072369dcc453c27b168468f26121661cdc49828" Jul 06 17:24:01.244712 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.244687 2574 scope.go:117] "RemoveContainer" containerID="98d13b3528410b0afab9b154a072369dcc453c27b168468f26121661cdc49828" Jul 06 17:24:01.244979 ip-10-0-131-71 kubenswrapper[2574]: E0706 17:24:01.244959 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98d13b3528410b0afab9b154a072369dcc453c27b168468f26121661cdc49828\": container with ID starting with 98d13b3528410b0afab9b154a072369dcc453c27b168468f26121661cdc49828 not found: ID does not exist" containerID="98d13b3528410b0afab9b154a072369dcc453c27b168468f26121661cdc49828" Jul 06 17:24:01.245066 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.244988 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98d13b3528410b0afab9b154a072369dcc453c27b168468f26121661cdc49828"} err="failed to get container status \"98d13b3528410b0afab9b154a072369dcc453c27b168468f26121661cdc49828\": rpc error: code = NotFound desc = could not find container \"98d13b3528410b0afab9b154a072369dcc453c27b168468f26121661cdc49828\": container with ID starting with 98d13b3528410b0afab9b154a072369dcc453c27b168468f26121661cdc49828 not found: ID does not exist" Jul 06 17:24:01.261331 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.261294 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd"] Jul 06 17:24:01.267946 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.267923 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-2cd1eb/sub-test-model-2cd1eb-kserve-59f564f9bf-pd8gd"] Jul 06 17:24:01.492366 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.492330 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" podUID="ce6c02a6-3dca-41b6-a26d-278842600342" containerName="main" containerID="cri-o://80429c2655bbe3b46194d357bc0db265364a477934bab034cc4912b15c920337" gracePeriod=45 Jul 06 17:24:01.745336 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.745272 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" Jul 06 17:24:01.824222 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.824180 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce6c02a6-3dca-41b6-a26d-278842600342-tmp-dir\") pod \"ce6c02a6-3dca-41b6-a26d-278842600342\" (UID: \"ce6c02a6-3dca-41b6-a26d-278842600342\") " Jul 06 17:24:01.824427 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.824251 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce6c02a6-3dca-41b6-a26d-278842600342-dshm\") pod \"ce6c02a6-3dca-41b6-a26d-278842600342\" (UID: \"ce6c02a6-3dca-41b6-a26d-278842600342\") " Jul 06 17:24:01.824427 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.824317 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce6c02a6-3dca-41b6-a26d-278842600342-tls-certs\") pod \"ce6c02a6-3dca-41b6-a26d-278842600342\" (UID: \"ce6c02a6-3dca-41b6-a26d-278842600342\") " Jul 06 17:24:01.824427 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.824345 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftb8l\" (UniqueName: \"kubernetes.io/projected/ce6c02a6-3dca-41b6-a26d-278842600342-kube-api-access-ftb8l\") pod \"ce6c02a6-3dca-41b6-a26d-278842600342\" (UID: \"ce6c02a6-3dca-41b6-a26d-278842600342\") " Jul 06 17:24:01.824427 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.824379 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6c02a6-3dca-41b6-a26d-278842600342-model-cache\") pod \"ce6c02a6-3dca-41b6-a26d-278842600342\" (UID: \"ce6c02a6-3dca-41b6-a26d-278842600342\") " Jul 06 17:24:01.824675 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.824436 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce6c02a6-3dca-41b6-a26d-278842600342-home\") pod \"ce6c02a6-3dca-41b6-a26d-278842600342\" (UID: \"ce6c02a6-3dca-41b6-a26d-278842600342\") " Jul 06 17:24:01.824675 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.824480 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce6c02a6-3dca-41b6-a26d-278842600342-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ce6c02a6-3dca-41b6-a26d-278842600342" (UID: "ce6c02a6-3dca-41b6-a26d-278842600342"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:24:01.824769 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.824716 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce6c02a6-3dca-41b6-a26d-278842600342-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce6c02a6-3dca-41b6-a26d-278842600342" (UID: "ce6c02a6-3dca-41b6-a26d-278842600342"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:24:01.824769 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.824730 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce6c02a6-3dca-41b6-a26d-278842600342-home" (OuterVolumeSpecName: "home") pod "ce6c02a6-3dca-41b6-a26d-278842600342" (UID: "ce6c02a6-3dca-41b6-a26d-278842600342"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:24:01.824769 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.824763 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce6c02a6-3dca-41b6-a26d-278842600342-tmp-dir\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:24:01.826550 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.826509 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce6c02a6-3dca-41b6-a26d-278842600342-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce6c02a6-3dca-41b6-a26d-278842600342" (UID: "ce6c02a6-3dca-41b6-a26d-278842600342"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:24:01.826667 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.826581 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce6c02a6-3dca-41b6-a26d-278842600342-kube-api-access-ftb8l" (OuterVolumeSpecName: "kube-api-access-ftb8l") pod "ce6c02a6-3dca-41b6-a26d-278842600342" (UID: "ce6c02a6-3dca-41b6-a26d-278842600342"). InnerVolumeSpecName "kube-api-access-ftb8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:24:01.826667 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.826597 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce6c02a6-3dca-41b6-a26d-278842600342-dshm" (OuterVolumeSpecName: "dshm") pod "ce6c02a6-3dca-41b6-a26d-278842600342" (UID: "ce6c02a6-3dca-41b6-a26d-278842600342"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:24:01.925927 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.925886 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce6c02a6-3dca-41b6-a26d-278842600342-dshm\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:24:01.925927 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.925919 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce6c02a6-3dca-41b6-a26d-278842600342-tls-certs\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:24:01.925927 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.925933 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftb8l\" (UniqueName: \"kubernetes.io/projected/ce6c02a6-3dca-41b6-a26d-278842600342-kube-api-access-ftb8l\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:24:01.926173 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.925947 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce6c02a6-3dca-41b6-a26d-278842600342-model-cache\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:24:01.926173 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:01.925958 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce6c02a6-3dca-41b6-a26d-278842600342-home\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:24:02.243260 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:02.243223 2574 generic.go:356] "Generic (PLEG): container finished" podID="ce6c02a6-3dca-41b6-a26d-278842600342" containerID="80429c2655bbe3b46194d357bc0db265364a477934bab034cc4912b15c920337" exitCode=0 Jul 06 17:24:02.243413 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:02.243269 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" Jul 06 17:24:02.243413 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:02.243270 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" event={"ID":"ce6c02a6-3dca-41b6-a26d-278842600342","Type":"ContainerDied","Data":"80429c2655bbe3b46194d357bc0db265364a477934bab034cc4912b15c920337"} Jul 06 17:24:02.243413 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:02.243368 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn" event={"ID":"ce6c02a6-3dca-41b6-a26d-278842600342","Type":"ContainerDied","Data":"f087c1496180e7391efb9ae78b8e003854e62b73482aa27947899c9eb0e2e431"} Jul 06 17:24:02.243413 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:02.243389 2574 scope.go:117] "RemoveContainer" containerID="80429c2655bbe3b46194d357bc0db265364a477934bab034cc4912b15c920337" Jul 06 17:24:02.252277 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:02.252257 2574 scope.go:117] "RemoveContainer" containerID="80429c2655bbe3b46194d357bc0db265364a477934bab034cc4912b15c920337" Jul 06 17:24:02.252556 ip-10-0-131-71 kubenswrapper[2574]: E0706 17:24:02.252532 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80429c2655bbe3b46194d357bc0db265364a477934bab034cc4912b15c920337\": container with ID starting with 80429c2655bbe3b46194d357bc0db265364a477934bab034cc4912b15c920337 not found: ID does not exist" containerID="80429c2655bbe3b46194d357bc0db265364a477934bab034cc4912b15c920337" Jul 06 17:24:02.252609 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:02.252566 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80429c2655bbe3b46194d357bc0db265364a477934bab034cc4912b15c920337"} err="failed to get container status \"80429c2655bbe3b46194d357bc0db265364a477934bab034cc4912b15c920337\": rpc error: code = NotFound desc = could not find container \"80429c2655bbe3b46194d357bc0db265364a477934bab034cc4912b15c920337\": container with ID starting with 80429c2655bbe3b46194d357bc0db265364a477934bab034cc4912b15c920337 not found: ID does not exist" Jul 06 17:24:02.269558 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:02.266936 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn"] Jul 06 17:24:02.272221 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:02.272194 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-0ebdd9/sub-test-model-0ebdd9-kserve-f75fb67d7-5qsfn"] Jul 06 17:24:03.076372 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:03.076338 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce6c02a6-3dca-41b6-a26d-278842600342" path="/var/lib/kubelet/pods/ce6c02a6-3dca-41b6-a26d-278842600342/volumes" Jul 06 17:24:03.076800 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:03.076773 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e593d1c0-082b-4fec-a3ee-f22d04d24b6e" path="/var/lib/kubelet/pods/e593d1c0-082b-4fec-a3ee-f22d04d24b6e/volumes" Jul 06 17:24:22.654013 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.653983 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm"] Jul 06 17:24:22.654481 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.654357 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e593d1c0-082b-4fec-a3ee-f22d04d24b6e" containerName="main" Jul 06 17:24:22.654481 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.654368 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e593d1c0-082b-4fec-a3ee-f22d04d24b6e" containerName="main" Jul 06 17:24:22.654481 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.654387 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce6c02a6-3dca-41b6-a26d-278842600342" containerName="main" Jul 06 17:24:22.654481 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.654392 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce6c02a6-3dca-41b6-a26d-278842600342" containerName="main" Jul 06 17:24:22.654481 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.654450 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e593d1c0-082b-4fec-a3ee-f22d04d24b6e" containerName="main" Jul 06 17:24:22.654481 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.654459 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce6c02a6-3dca-41b6-a26d-278842600342" containerName="main" Jul 06 17:24:22.657460 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.657446 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" Jul 06 17:24:22.661384 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.661363 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0ebdd9\"/\"openshift-service-ca.crt\"" Jul 06 17:24:22.662078 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.662054 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0ebdd9\"/\"default-dockercfg-8nmft\"" Jul 06 17:24:22.662181 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.662111 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-0ebdd9\"/\"kube-root-ca.crt\"" Jul 06 17:24:22.662392 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.662376 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-0ebdd9\"/\"rate-test-model-0ebdd9-kserve-self-signed-certs\"" Jul 06 17:24:22.683374 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.683350 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm"] Jul 06 17:24:22.713597 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.713571 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-dshm\") pod \"rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm\" (UID: \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\") " pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" Jul 06 17:24:22.713739 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.713609 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-model-cache\") pod \"rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm\" (UID: \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\") " pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" Jul 06 17:24:22.713739 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.713698 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-home\") pod \"rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm\" (UID: \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\") " pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" Jul 06 17:24:22.713739 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.713732 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-tmp-dir\") pod \"rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm\" (UID: \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\") " pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" Jul 06 17:24:22.713904 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.713775 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-tls-certs\") pod \"rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm\" (UID: \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\") " pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" Jul 06 17:24:22.713904 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.713831 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twthc\" (UniqueName: \"kubernetes.io/projected/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-kube-api-access-twthc\") pod \"rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm\" (UID: \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\") " pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" Jul 06 17:24:22.814463 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.814429 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-home\") pod \"rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm\" (UID: \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\") " pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" Jul 06 17:24:22.814463 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.814462 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-tmp-dir\") pod \"rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm\" (UID: \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\") " pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" Jul 06 17:24:22.814728 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.814484 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-tls-certs\") pod \"rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm\" (UID: \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\") " pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" Jul 06 17:24:22.814728 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.814533 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twthc\" (UniqueName: \"kubernetes.io/projected/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-kube-api-access-twthc\") pod \"rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm\" (UID: \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\") " pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" Jul 06 17:24:22.814728 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.814563 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-dshm\") pod \"rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm\" (UID: \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\") " pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" Jul 06 17:24:22.814728 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.814585 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-model-cache\") pod \"rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm\" (UID: \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\") " pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" Jul 06 17:24:22.814958 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.814881 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-home\") pod \"rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm\" (UID: \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\") " pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" Jul 06 17:24:22.814958 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.814943 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-tmp-dir\") pod \"rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm\" (UID: \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\") " pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" Jul 06 17:24:22.815029 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.815014 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-model-cache\") pod \"rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm\" (UID: \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\") " pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" Jul 06 17:24:22.816857 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.816831 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-dshm\") pod \"rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm\" (UID: \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\") " pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" Jul 06 17:24:22.817148 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.817127 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-tls-certs\") pod \"rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm\" (UID: \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\") " pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" Jul 06 17:24:22.830946 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.830922 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twthc\" (UniqueName: \"kubernetes.io/projected/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-kube-api-access-twthc\") pod \"rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm\" (UID: \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\") " pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" Jul 06 17:24:22.966933 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:22.966901 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" Jul 06 17:24:23.116559 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:23.116508 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm"] Jul 06 17:24:23.119789 ip-10-0-131-71 kubenswrapper[2574]: W0706 17:24:23.119762 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f2c7b20_ab4b_46aa_8d18_16fb09c2181f.slice/crio-caa75187c73abd1482a9765c309ec3ddd514f552462b56d0c4d11fa38acb747d WatchSource:0}: Error finding container caa75187c73abd1482a9765c309ec3ddd514f552462b56d0c4d11fa38acb747d: Status 404 returned error can't find the container with id caa75187c73abd1482a9765c309ec3ddd514f552462b56d0c4d11fa38acb747d Jul 06 17:24:23.345044 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:23.344950 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" event={"ID":"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f","Type":"ContainerStarted","Data":"459a7c36d2f8f5ba58a76957bae7eda6de2e92bdbe9cc8cb404b42dd0273de2b"} Jul 06 17:24:23.345044 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:23.345007 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" event={"ID":"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f","Type":"ContainerStarted","Data":"caa75187c73abd1482a9765c309ec3ddd514f552462b56d0c4d11fa38acb747d"} Jul 06 17:24:23.368649 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:23.368605 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" podStartSLOduration=1.368585747 podStartE2EDuration="1.368585747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:24:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:24:23.367307181 +0000 UTC m=+2756.886798309" watchObservedRunningTime="2026-07-06 17:24:23.368585747 +0000 UTC m=+2756.888076862" Jul 06 17:24:26.949453 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:26.949420 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:24:32.967674 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:32.967635 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" Jul 06 17:24:32.968088 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:32.967689 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" Jul 06 17:24:32.980264 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:32.980234 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" Jul 06 17:24:33.404129 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:33.404046 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" Jul 06 17:24:40.106794 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:40.106752 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:24:54.736678 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:24:54.736645 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:25:01.792164 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:01.792133 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm"] Jul 06 17:25:08.142869 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:08.142823 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:25:16.811743 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:16.811673 2574 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" podUID="8f2c7b20-ab4b-46aa-8d18-16fb09c2181f" containerName="main" containerID="cri-o://459a7c36d2f8f5ba58a76957bae7eda6de2e92bdbe9cc8cb404b42dd0273de2b" gracePeriod=45 Jul 06 17:25:17.073296 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.073226 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" Jul 06 17:25:17.097557 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.097496 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twthc\" (UniqueName: \"kubernetes.io/projected/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-kube-api-access-twthc\") pod \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\" (UID: \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\") " Jul 06 17:25:17.097557 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.097562 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-tmp-dir\") pod \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\" (UID: \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\") " Jul 06 17:25:17.097821 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.097605 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-home\") pod \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\" (UID: \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\") " Jul 06 17:25:17.097821 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.097691 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-tls-certs\") pod \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\" (UID: \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\") " Jul 06 17:25:17.097821 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.097753 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-dshm\") pod \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\" (UID: \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\") " Jul 06 17:25:17.097821 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.097783 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-model-cache\") pod \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\" (UID: \"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f\") " Jul 06 17:25:17.098015 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.097860 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8f2c7b20-ab4b-46aa-8d18-16fb09c2181f" (UID: "8f2c7b20-ab4b-46aa-8d18-16fb09c2181f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:25:17.098015 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.097864 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-home" (OuterVolumeSpecName: "home") pod "8f2c7b20-ab4b-46aa-8d18-16fb09c2181f" (UID: "8f2c7b20-ab4b-46aa-8d18-16fb09c2181f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:25:17.098180 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.098061 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-model-cache" (OuterVolumeSpecName: "model-cache") pod "8f2c7b20-ab4b-46aa-8d18-16fb09c2181f" (UID: "8f2c7b20-ab4b-46aa-8d18-16fb09c2181f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:25:17.098255 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.098235 2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-home\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:25:17.098321 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.098260 2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-model-cache\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:25:17.098321 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.098275 2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-tmp-dir\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:25:17.099950 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.099922 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-dshm" (OuterVolumeSpecName: "dshm") pod "8f2c7b20-ab4b-46aa-8d18-16fb09c2181f" (UID: "8f2c7b20-ab4b-46aa-8d18-16fb09c2181f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:25:17.100067 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.099996 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-kube-api-access-twthc" (OuterVolumeSpecName: "kube-api-access-twthc") pod "8f2c7b20-ab4b-46aa-8d18-16fb09c2181f" (UID: "8f2c7b20-ab4b-46aa-8d18-16fb09c2181f"). InnerVolumeSpecName "kube-api-access-twthc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:25:17.100067 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.100038 2574 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f2c7b20-ab4b-46aa-8d18-16fb09c2181f" (UID: "8f2c7b20-ab4b-46aa-8d18-16fb09c2181f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 17:25:17.198987 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.198947 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twthc\" (UniqueName: \"kubernetes.io/projected/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-kube-api-access-twthc\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:25:17.198987 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.198980 2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-tls-certs\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:25:17.198987 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.198992 2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f-dshm\") on node \"ip-10-0-131-71.ec2.internal\" DevicePath \"\"" Jul 06 17:25:17.606257 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.606220 2574 generic.go:356] "Generic (PLEG): container finished" podID="8f2c7b20-ab4b-46aa-8d18-16fb09c2181f" containerID="459a7c36d2f8f5ba58a76957bae7eda6de2e92bdbe9cc8cb404b42dd0273de2b" exitCode=0 Jul 06 17:25:17.606449 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.606265 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" Jul 06 17:25:17.606449 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.606316 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" event={"ID":"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f","Type":"ContainerDied","Data":"459a7c36d2f8f5ba58a76957bae7eda6de2e92bdbe9cc8cb404b42dd0273de2b"} Jul 06 17:25:17.606449 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.606364 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm" event={"ID":"8f2c7b20-ab4b-46aa-8d18-16fb09c2181f","Type":"ContainerDied","Data":"caa75187c73abd1482a9765c309ec3ddd514f552462b56d0c4d11fa38acb747d"} Jul 06 17:25:17.606449 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.606382 2574 scope.go:117] "RemoveContainer" containerID="459a7c36d2f8f5ba58a76957bae7eda6de2e92bdbe9cc8cb404b42dd0273de2b" Jul 06 17:25:17.615147 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.615128 2574 scope.go:117] "RemoveContainer" containerID="459a7c36d2f8f5ba58a76957bae7eda6de2e92bdbe9cc8cb404b42dd0273de2b" Jul 06 17:25:17.615407 ip-10-0-131-71 kubenswrapper[2574]: E0706 17:25:17.615388 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"459a7c36d2f8f5ba58a76957bae7eda6de2e92bdbe9cc8cb404b42dd0273de2b\": container with ID starting with 459a7c36d2f8f5ba58a76957bae7eda6de2e92bdbe9cc8cb404b42dd0273de2b not found: ID does not exist" containerID="459a7c36d2f8f5ba58a76957bae7eda6de2e92bdbe9cc8cb404b42dd0273de2b" Jul 06 17:25:17.615452 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.615415 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"459a7c36d2f8f5ba58a76957bae7eda6de2e92bdbe9cc8cb404b42dd0273de2b"} err="failed to get container status \"459a7c36d2f8f5ba58a76957bae7eda6de2e92bdbe9cc8cb404b42dd0273de2b\": rpc error: code = NotFound desc = could not find container \"459a7c36d2f8f5ba58a76957bae7eda6de2e92bdbe9cc8cb404b42dd0273de2b\": container with ID starting with 459a7c36d2f8f5ba58a76957bae7eda6de2e92bdbe9cc8cb404b42dd0273de2b not found: ID does not exist" Jul 06 17:25:17.634996 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.634971 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm"] Jul 06 17:25:17.638258 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:17.638236 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-0ebdd9/rate-test-model-0ebdd9-kserve-7bd84bc5d7-st5wm"] Jul 06 17:25:19.075912 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:19.075881 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f2c7b20-ab4b-46aa-8d18-16fb09c2181f" path="/var/lib/kubelet/pods/8f2c7b20-ab4b-46aa-8d18-16fb09c2181f/volumes" Jul 06 17:25:50.503195 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:50.503154 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-q294l"] Jul 06 17:25:59.548224 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:59.548189 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-zkvgc_6a166ea3-4b8f-4785-90e7-d807751e6567/manager/0.log" Jul 06 17:25:59.669619 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:25:59.669585 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-xzhs8_3b7e6bbe-d837-4656-b585-d2f9b254830e/manager/0.log" Jul 06 17:26:00.730319 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:00.730290 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d66bcd77d-vfkkk_39214b4b-e5c5-499c-ba7b-44b81db5a24a/copy-manifests/0.log" Jul 06 17:26:00.736943 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:00.736918 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d66bcd77d-vfkkk_39214b4b-e5c5-499c-ba7b-44b81db5a24a/manager/0.log" Jul 06 17:26:00.852324 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:00.852286 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-ndsgq_a6fcd064-9328-429a-8356-84b50309e4fd/postgres/0.log" Jul 06 17:26:02.241249 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:02.241191 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-bwhj8_6ad1e1c6-a52e-419e-9bc3-5fbd83b79ec9/manager/0.log" Jul 06 17:26:02.355165 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:02.355133 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-bhhj5_fb360155-5817-4af0-a955-2f55fb125800/manager/0.log" Jul 06 17:26:02.462512 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:02.462486 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-8f2qf_0d3ad447-6e87-43dd-bd9b-82847c08dd5e/kuadrant-console-plugin/0.log" Jul 06 17:26:02.576997 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:02.576913 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-nr28z_ea5197b4-ea14-4d28-b4c9-16fcd3f6bf69/registry-server/0.log" Jul 06 17:26:02.698804 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:02.698774 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-pbk8v_6305877e-6719-41f7-9d9f-3ec5534c8cdd/manager/0.log" Jul 06 17:26:02.813055 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:02.813028 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-q294l_9e689a68-5912-40d5-a37f-b9d8d37ccd47/limitador/0.log" Jul 06 17:26:03.496489 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:03.496451 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5548f7d4db-d8r2s_d74d37ae-0e47-45c8-9520-b490f54badf9/kube-auth-proxy/0.log" Jul 06 17:26:03.743918 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:03.743884 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-mqc82_9ccabcba-7d81-48aa-919f-150b01a4959c/istio-proxy/1.log" Jul 06 17:26:03.995012 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:03.994978 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-cm8wv_ef7d126c-7733-4313-9644-7b8de3b45a93/payload-processing/0.log" Jul 06 17:26:04.553428 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:04.553395 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-jfbsk_88361605-3306-42ad-b7bd-ffab7e1f3313/main/0.log" Jul 06 17:26:04.677977 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:04.677950 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-mn95s_4486b3ed-42d1-4b23-ac20-08c5247ff5f5/main/0.log" Jul 06 17:26:05.021218 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:05.021190 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-tmnjg_92e4ce7c-9cc9-446a-ba9a-b4505caddd7f/main/0.log" Jul 06 17:26:12.449407 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:12.449375 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nnd4r_9ce13f97-b996-4f40-86fb-e0453bd3a476/global-pull-secret-syncer/0.log" Jul 06 17:26:12.662662 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:12.662628 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xswlf_94a73cbd-9f23-4895-9127-fe752b38a3a5/konnectivity-agent/0.log" Jul 06 17:26:12.710485 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:12.710408 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-71.ec2.internal_03b7a04aec66663a9ae8155170654101/haproxy/0.log" Jul 06 17:26:18.184866 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:18.184812 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-bwhj8_6ad1e1c6-a52e-419e-9bc3-5fbd83b79ec9/manager/0.log" Jul 06 17:26:18.211354 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:18.211323 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-bhhj5_fb360155-5817-4af0-a955-2f55fb125800/manager/0.log" Jul 06 17:26:18.248577 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:18.248555 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-8f2qf_0d3ad447-6e87-43dd-bd9b-82847c08dd5e/kuadrant-console-plugin/0.log" Jul 06 17:26:18.305199 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:18.305170 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-nr28z_ea5197b4-ea14-4d28-b4c9-16fcd3f6bf69/registry-server/0.log" Jul 06 17:26:18.459462 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:18.459428 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-pbk8v_6305877e-6719-41f7-9d9f-3ec5534c8cdd/manager/0.log" Jul 06 17:26:18.488224 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:18.488195 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-q294l_9e689a68-5912-40d5-a37f-b9d8d37ccd47/limitador/0.log" Jul 06 17:26:20.575918 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:20.575884 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-lwx68_2743adb0-29e5-4b14-999b-dd6b06bbab78/kube-state-metrics/0.log" Jul 06 17:26:20.592164 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:20.592140 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-lwx68_2743adb0-29e5-4b14-999b-dd6b06bbab78/kube-rbac-proxy-main/0.log" Jul 06 17:26:20.614028 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:20.613999 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-lwx68_2743adb0-29e5-4b14-999b-dd6b06bbab78/kube-rbac-proxy-self/0.log" Jul 06 17:26:20.720875 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:20.720840 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mgt5v_9ac30c81-a78b-44ab-aa37-711dcc8be401/node-exporter/0.log" Jul 06 17:26:20.741803 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:20.741773 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mgt5v_9ac30c81-a78b-44ab-aa37-711dcc8be401/kube-rbac-proxy/0.log" Jul 06 17:26:20.763688 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:20.763666 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mgt5v_9ac30c81-a78b-44ab-aa37-711dcc8be401/init-textfile/0.log" Jul 06 17:26:22.261741 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.261706 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hztcg/perf-node-gather-daemonset-kwm7p"] Jul 06 17:26:22.262118 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.262068 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f2c7b20-ab4b-46aa-8d18-16fb09c2181f" containerName="main" Jul 06 17:26:22.262118 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.262078 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f2c7b20-ab4b-46aa-8d18-16fb09c2181f" containerName="main" Jul 06 17:26:22.262214 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.262136 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f2c7b20-ab4b-46aa-8d18-16fb09c2181f" containerName="main" Jul 06 17:26:22.265341 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.265319 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-kwm7p" Jul 06 17:26:22.268888 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.268866 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hztcg\"/\"kube-root-ca.crt\"" Jul 06 17:26:22.270058 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.270034 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hztcg\"/\"default-dockercfg-7dvzm\"" Jul 06 17:26:22.270164 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.270046 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hztcg\"/\"openshift-service-ca.crt\"" Jul 06 17:26:22.281441 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.281411 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hztcg/perf-node-gather-daemonset-kwm7p"] Jul 06 17:26:22.378825 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.378792 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f10473b9-e4a8-4d74-a5fb-70bd001d6891-podres\") pod \"perf-node-gather-daemonset-kwm7p\" (UID: \"f10473b9-e4a8-4d74-a5fb-70bd001d6891\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-kwm7p" Jul 06 17:26:22.379022 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.378830 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f10473b9-e4a8-4d74-a5fb-70bd001d6891-lib-modules\") pod \"perf-node-gather-daemonset-kwm7p\" (UID: \"f10473b9-e4a8-4d74-a5fb-70bd001d6891\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-kwm7p" Jul 06 17:26:22.379022 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.378856 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f10473b9-e4a8-4d74-a5fb-70bd001d6891-proc\") pod \"perf-node-gather-daemonset-kwm7p\" (UID: \"f10473b9-e4a8-4d74-a5fb-70bd001d6891\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-kwm7p" Jul 06 17:26:22.379022 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.378950 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f10473b9-e4a8-4d74-a5fb-70bd001d6891-sys\") pod \"perf-node-gather-daemonset-kwm7p\" (UID: \"f10473b9-e4a8-4d74-a5fb-70bd001d6891\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-kwm7p" Jul 06 17:26:22.379022 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.378993 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlmpn\" (UniqueName: \"kubernetes.io/projected/f10473b9-e4a8-4d74-a5fb-70bd001d6891-kube-api-access-wlmpn\") pod \"perf-node-gather-daemonset-kwm7p\" (UID: \"f10473b9-e4a8-4d74-a5fb-70bd001d6891\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-kwm7p" Jul 06 17:26:22.479792 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.479756 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f10473b9-e4a8-4d74-a5fb-70bd001d6891-podres\") pod \"perf-node-gather-daemonset-kwm7p\" (UID: \"f10473b9-e4a8-4d74-a5fb-70bd001d6891\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-kwm7p" Jul 06 17:26:22.479792 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.479795 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f10473b9-e4a8-4d74-a5fb-70bd001d6891-lib-modules\") pod \"perf-node-gather-daemonset-kwm7p\" (UID: \"f10473b9-e4a8-4d74-a5fb-70bd001d6891\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-kwm7p" Jul 06 17:26:22.480039 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.479826 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f10473b9-e4a8-4d74-a5fb-70bd001d6891-proc\") pod \"perf-node-gather-daemonset-kwm7p\" (UID: \"f10473b9-e4a8-4d74-a5fb-70bd001d6891\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-kwm7p" Jul 06 17:26:22.480039 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.479861 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f10473b9-e4a8-4d74-a5fb-70bd001d6891-sys\") pod \"perf-node-gather-daemonset-kwm7p\" (UID: \"f10473b9-e4a8-4d74-a5fb-70bd001d6891\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-kwm7p" Jul 06 17:26:22.480039 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.479898 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlmpn\" (UniqueName: \"kubernetes.io/projected/f10473b9-e4a8-4d74-a5fb-70bd001d6891-kube-api-access-wlmpn\") pod \"perf-node-gather-daemonset-kwm7p\" (UID: \"f10473b9-e4a8-4d74-a5fb-70bd001d6891\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-kwm7p" Jul 06 17:26:22.480039 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.479949 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f10473b9-e4a8-4d74-a5fb-70bd001d6891-podres\") pod \"perf-node-gather-daemonset-kwm7p\" (UID: \"f10473b9-e4a8-4d74-a5fb-70bd001d6891\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-kwm7p" Jul 06 17:26:22.480039 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.479960 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f10473b9-e4a8-4d74-a5fb-70bd001d6891-lib-modules\") pod \"perf-node-gather-daemonset-kwm7p\" (UID: \"f10473b9-e4a8-4d74-a5fb-70bd001d6891\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-kwm7p" Jul 06 17:26:22.480039 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.479969 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f10473b9-e4a8-4d74-a5fb-70bd001d6891-sys\") pod \"perf-node-gather-daemonset-kwm7p\" (UID: \"f10473b9-e4a8-4d74-a5fb-70bd001d6891\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-kwm7p" Jul 06 17:26:22.480039 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.479959 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f10473b9-e4a8-4d74-a5fb-70bd001d6891-proc\") pod \"perf-node-gather-daemonset-kwm7p\" (UID: \"f10473b9-e4a8-4d74-a5fb-70bd001d6891\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-kwm7p" Jul 06 17:26:22.493419 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.493389 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlmpn\" (UniqueName: \"kubernetes.io/projected/f10473b9-e4a8-4d74-a5fb-70bd001d6891-kube-api-access-wlmpn\") pod \"perf-node-gather-daemonset-kwm7p\" (UID: \"f10473b9-e4a8-4d74-a5fb-70bd001d6891\") " pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-kwm7p" Jul 06 17:26:22.575779 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.575710 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-kwm7p" Jul 06 17:26:22.709873 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.709843 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hztcg/perf-node-gather-daemonset-kwm7p"] Jul 06 17:26:22.713246 ip-10-0-131-71 kubenswrapper[2574]: W0706 17:26:22.713218 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf10473b9_e4a8_4d74_a5fb_70bd001d6891.slice/crio-a5856062bd2a62cf4cdbfda48f9b325268cdfc1a393eaadb452a59164c444c4a WatchSource:0}: Error finding container a5856062bd2a62cf4cdbfda48f9b325268cdfc1a393eaadb452a59164c444c4a: Status 404 returned error can't find the container with id a5856062bd2a62cf4cdbfda48f9b325268cdfc1a393eaadb452a59164c444c4a Jul 06 17:26:22.908073 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.908023 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-kwm7p" event={"ID":"f10473b9-e4a8-4d74-a5fb-70bd001d6891","Type":"ContainerStarted","Data":"5932b0f403654f1a8ec519114ec42851091cfe4965d3878d27820e13656a9df2"} Jul 06 17:26:22.908256 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.908215 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-kwm7p" event={"ID":"f10473b9-e4a8-4d74-a5fb-70bd001d6891","Type":"ContainerStarted","Data":"a5856062bd2a62cf4cdbfda48f9b325268cdfc1a393eaadb452a59164c444c4a"} Jul 06 17:26:22.908539 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.908494 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-kwm7p" Jul 06 17:26:22.936344 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:22.936295 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-kwm7p" podStartSLOduration=0.936282173 podStartE2EDuration="936.282173ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 17:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:26:22.936252233 +0000 UTC m=+2876.455743350" watchObservedRunningTime="2026-07-06 17:26:22.936282173 +0000 UTC m=+2876.455773320" Jul 06 17:26:23.101852 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:23.101758 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-b2rtp_a9e75cc4-a9a1-4be1-b52b-0522e2f176d3/networking-console-plugin/0.log" Jul 06 17:26:24.491670 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:24.491643 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-9dd6795c7-td257_43edfcba-7c81-4ce5-bcbd-22ba27f2e0e7/console/0.log" Jul 06 17:26:24.549629 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:24.549585 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-nh4dr_832a849a-0ae6-4242-8ba1-f64b0f12038a/download-server/0.log" Jul 06 17:26:26.606355 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:26.606322 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n82pj_f2d0b8d6-be30-423c-840e-f20cebd694fd/dns/0.log" Jul 06 17:26:26.644283 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:26.644245 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n82pj_f2d0b8d6-be30-423c-840e-f20cebd694fd/kube-rbac-proxy/0.log" Jul 06 17:26:26.809257 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:26.809228 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nf6xc_5aa71c0b-c28a-43a4-92d7-3d617a604f55/dns-node-resolver/0.log" Jul 06 17:26:27.593886 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:27.593853 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lxbqr_0f6a8b1b-224d-459c-aad2-1d86d1c38c79/node-ca/0.log" Jul 06 17:26:28.850379 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:28.850315 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5548f7d4db-d8r2s_d74d37ae-0e47-45c8-9520-b490f54badf9/kube-auth-proxy/0.log" Jul 06 17:26:28.909141 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:28.909113 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-mqc82_9ccabcba-7d81-48aa-919f-150b01a4959c/istio-proxy/0.log" Jul 06 17:26:28.924437 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:28.924408 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hztcg/perf-node-gather-daemonset-kwm7p" Jul 06 17:26:28.964124 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:28.964093 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-mqc82_9ccabcba-7d81-48aa-919f-150b01a4959c/istio-proxy/1.log" Jul 06 17:26:29.066737 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:29.066706 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-cm8wv_ef7d126c-7733-4313-9644-7b8de3b45a93/payload-processing/0.log" Jul 06 17:26:29.859713 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:29.859682 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pdh4h_5b350fbc-4cba-421e-89d3-1f0df3318c34/serve-healthcheck-canary/0.log" Jul 06 17:26:30.512489 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:30.512459 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fqkj2_48e427b3-b218-4b68-90ba-0a4750c49f43/kube-rbac-proxy/0.log" Jul 06 17:26:30.538462 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:30.538436 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fqkj2_48e427b3-b218-4b68-90ba-0a4750c49f43/exporter/0.log" Jul 06 17:26:30.560858 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:30.560826 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fqkj2_48e427b3-b218-4b68-90ba-0a4750c49f43/extractor/0.log" Jul 06 17:26:33.096966 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:33.096923 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-zkvgc_6a166ea3-4b8f-4785-90e7-d807751e6567/manager/0.log" Jul 06 17:26:33.237386 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:33.237354 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5548f7d4db-d8r2s_d74d37ae-0e47-45c8-9520-b490f54badf9/kube-auth-proxy/0.log" Jul 06 17:26:33.309007 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:33.308918 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-xzhs8_3b7e6bbe-d837-4656-b585-d2f9b254830e/manager/0.log" Jul 06 17:26:33.429000 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:33.428963 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-mqc82_9ccabcba-7d81-48aa-919f-150b01a4959c/istio-proxy/0.log" Jul 06 17:26:33.480498 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:33.480443 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-mqc82_9ccabcba-7d81-48aa-919f-150b01a4959c/istio-proxy/1.log" Jul 06 17:26:33.676479 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:33.676393 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-cm8wv_ef7d126c-7733-4313-9644-7b8de3b45a93/payload-processing/0.log" Jul 06 17:26:34.381742 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:34.381707 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d66bcd77d-vfkkk_39214b4b-e5c5-499c-ba7b-44b81db5a24a/manager/0.log" Jul 06 17:26:34.429293 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:34.429266 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d66bcd77d-vfkkk_39214b4b-e5c5-499c-ba7b-44b81db5a24a/copy-manifests/0.log" Jul 06 17:26:34.486383 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:34.486350 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-ndsgq_a6fcd064-9328-429a-8356-84b50309e4fd/postgres/0.log" Jul 06 17:26:35.077779 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:35.077681 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5548f7d4db-d8r2s_d74d37ae-0e47-45c8-9520-b490f54badf9/kube-auth-proxy/0.log" Jul 06 17:26:35.126745 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:35.126717 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-mqc82_9ccabcba-7d81-48aa-919f-150b01a4959c/istio-proxy/0.log" Jul 06 17:26:35.176590 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:35.176560 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-mqc82_9ccabcba-7d81-48aa-919f-150b01a4959c/istio-proxy/1.log" Jul 06 17:26:35.404539 ip-10-0-131-71 kubenswrapper[2574]: I0706 17:26:35.404440 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-cm8wv_ef7d126c-7733-4313-9644-7b8de3b45a93/payload-processing/0.log"