Jul 10 00:15:36.277461 ip-10-0-132-30 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 10 00:15:36.277470 ip-10-0-132-30 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 10 00:15:36.277477 ip-10-0-132-30 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 10 00:15:36.277686 ip-10-0-132-30 systemd[1]: Failed to start Kubernetes Kubelet. Jul 10 00:15:46.334917 ip-10-0-132-30 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 10 00:15:46.334936 ip-10-0-132-30 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot fd9739c16a894d02b80a1602c490592c -- Jul 10 00:17:45.741543 ip-10-0-132-30 systemd[1]: Starting Kubernetes Kubelet... Jul 10 00:17:46.176686 ip-10-0-132-30 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 00:17:46.176686 ip-10-0-132-30 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 00:17:46.176686 ip-10-0-132-30 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 00:17:46.176686 ip-10-0-132-30 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 00:17:46.176686 ip-10-0-132-30 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 00:17:46.176686 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.152170 2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 00:17:46.176686 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157598 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:17:46.176686 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157619 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:17:46.176686 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157622 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:17:46.176686 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157625 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:17:46.176686 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157631 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:17:46.176686 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157634 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:17:46.176686 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157637 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:17:46.176686 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157640 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:17:46.180530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157644 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:17:46.180530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157647 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:17:46.180530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157650 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:17:46.180530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157653 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:17:46.180530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157656 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:17:46.180530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157658 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:17:46.180530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157661 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:17:46.180530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157664 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:17:46.180530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157674 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:17:46.180530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157677 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:17:46.180530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157680 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:17:46.180530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157683 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:17:46.180530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157686 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:17:46.180530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157689 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:17:46.180530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157692 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:17:46.180530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157695 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:17:46.180530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157697 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:17:46.180530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157700 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:17:46.180530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157702 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:17:46.181395 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157705 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:17:46.181395 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157707 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:17:46.181395 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157710 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:17:46.181395 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157713 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:17:46.181395 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157717 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:17:46.181395 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157720 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:17:46.181395 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157725 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:17:46.181395 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157730 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:17:46.181395 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157733 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:17:46.181395 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157736 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:17:46.181395 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157738 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:17:46.181395 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157741 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:17:46.181395 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157743 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:17:46.181395 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157746 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:17:46.181395 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157748 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:17:46.181395 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157751 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:17:46.181395 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157755 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:17:46.181395 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157757 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:17:46.181395 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157761 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:17:46.181395 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157763 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:17:46.186320 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157767 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:17:46.186320 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157771 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:17:46.186320 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157775 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:17:46.186320 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157778 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:17:46.186320 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157781 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:17:46.186320 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157784 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:17:46.186320 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157787 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:17:46.186320 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157790 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:17:46.186320 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157793 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:17:46.186320 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157795 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:17:46.186320 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157798 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:17:46.186320 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157801 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:17:46.186320 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157803 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:17:46.186320 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157806 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:17:46.186320 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157808 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:17:46.186320 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157811 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:17:46.186320 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157814 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:17:46.186320 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157816 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:17:46.186320 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157819 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:17:46.186320 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157822 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:17:46.187046 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157824 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:17:46.187046 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157827 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:17:46.187046 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157830 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:17:46.187046 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157832 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:17:46.187046 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157836 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:17:46.187046 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157839 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:17:46.187046 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157842 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:17:46.187046 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157844 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:17:46.187046 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157847 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:17:46.187046 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157849 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:17:46.187046 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157852 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:17:46.187046 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157855 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:17:46.187046 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157859 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:17:46.187046 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157863 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:17:46.187046 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157866 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:17:46.187046 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157869 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:17:46.187046 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157872 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:17:46.187046 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157875 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:17:46.187046 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157878 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:17:46.187580 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157881 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:17:46.187580 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157884 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:17:46.187580 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157886 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:17:46.187580 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157889 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:17:46.187580 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157892 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:17:46.187580 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157894 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:17:46.187580 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157897 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:17:46.187580 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157899 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:17:46.187580 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157903 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:17:46.187580 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157905 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:17:46.187580 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157908 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:17:46.187580 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157910 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:17:46.187580 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.157913 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:17:46.187580 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158757 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:17:46.187580 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158766 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:17:46.187580 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158769 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:17:46.187580 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158772 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:17:46.187580 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158775 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:17:46.187580 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158778 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:17:46.188138 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158782 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:17:46.188138 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158786 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:17:46.188138 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158790 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:17:46.188138 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158794 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:17:46.188138 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158798 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:17:46.188138 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158801 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:17:46.188138 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158803 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:17:46.188138 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158807 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:17:46.188138 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158811 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:17:46.188138 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158814 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:17:46.188138 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158816 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:17:46.188138 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158819 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:17:46.188138 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158822 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:17:46.188138 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158824 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:17:46.188138 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158827 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:17:46.188138 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158829 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:17:46.188138 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158832 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:17:46.188138 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158834 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:17:46.188138 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158838 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:17:46.188723 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158840 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:17:46.188723 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158843 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:17:46.188723 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158845 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:17:46.188723 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158848 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:17:46.188723 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158851 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:17:46.188723 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158853 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:17:46.188723 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158856 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:17:46.188723 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158859 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:17:46.188723 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158862 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:17:46.188723 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158865 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:17:46.188723 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158867 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:17:46.188723 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158870 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:17:46.188723 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158872 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:17:46.188723 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158875 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:17:46.188723 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158877 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:17:46.188723 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158880 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:17:46.188723 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158883 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:17:46.188723 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158886 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:17:46.188723 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158889 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:17:46.188723 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158891 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:17:46.189379 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158894 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:17:46.189379 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158899 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:17:46.189379 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158903 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:17:46.189379 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158906 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:17:46.189379 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158909 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:17:46.189379 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158912 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:17:46.189379 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158916 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:17:46.189379 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158918 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:17:46.189379 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158921 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:17:46.189379 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158924 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:17:46.189379 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158926 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:17:46.189379 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158929 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:17:46.189379 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158931 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:17:46.189379 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158934 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:17:46.189379 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158936 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:17:46.189379 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158939 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:17:46.189379 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158942 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:17:46.189379 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158944 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:17:46.189379 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158947 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:17:47.526135 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158950 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:17:47.526135 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158953 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:17:47.526135 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158956 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:17:47.526135 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158958 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:17:47.526135 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158961 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:17:47.526135 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158964 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:17:47.526135 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158966 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:17:47.526135 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158969 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:17:47.526135 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158971 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:17:47.526135 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158974 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:17:47.526135 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158977 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:17:47.526135 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158979 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:17:47.526135 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158983 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:17:47.526135 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158986 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:17:47.526135 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158988 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:17:47.526135 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158991 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:17:47.526135 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158994 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:17:47.526135 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158997 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:17:47.526135 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.158999 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:17:47.526135 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159002 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:17:46.229491 ip-10-0-132-30 systemd[1]: Started Kubernetes Kubelet. Jul 10 00:17:47.527775 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159004 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:17:47.527775 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159008 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:17:47.527775 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159010 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:17:47.527775 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159013 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:17:47.527775 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159015 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:17:47.527775 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159018 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:17:47.527775 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159021 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:17:47.527775 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159024 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:17:47.527775 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159027 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:17:47.527775 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159029 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:17:47.527775 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159032 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:17:47.527775 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159035 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:17:47.527775 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159041 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:17:47.527775 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159044 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:17:47.527775 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159047 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:17:47.527775 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159480 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:17:47.527775 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159485 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:17:47.527775 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159489 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:17:47.527775 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159492 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:17:47.527775 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159494 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:17:47.529009 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159498 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:17:47.529009 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159501 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:17:47.529009 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159504 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:17:47.529009 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159506 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:17:47.529009 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159509 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:17:47.529009 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159512 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:17:47.529009 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159515 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:17:47.529009 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159517 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:17:47.529009 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159521 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:17:47.529009 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159524 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:17:47.529009 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159526 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:17:47.529009 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159529 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:17:47.529009 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159531 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:17:47.529009 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159534 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:17:47.529009 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159537 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:17:47.529009 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159539 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:17:47.529009 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159542 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:17:47.529009 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159544 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:17:47.529009 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159547 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:17:47.529009 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159549 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:17:47.530075 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159552 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:17:47.530075 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159554 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:17:47.530075 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159557 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:17:47.530075 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159560 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:17:47.530075 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159565 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:17:47.530075 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159567 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:17:47.530075 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159570 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:17:47.530075 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159572 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:17:47.530075 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159575 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:17:47.530075 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159577 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:17:47.530075 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159580 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:17:47.530075 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159583 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:17:47.530075 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159586 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:17:47.530075 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159588 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:17:47.530075 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159591 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:17:47.530075 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159593 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:17:47.530075 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159596 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:17:47.530075 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159599 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:17:47.530075 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159601 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:17:47.530075 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159604 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:17:47.531099 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159606 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:17:47.531099 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159610 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:17:47.531099 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159612 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:17:47.531099 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159615 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:17:47.531099 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159618 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:17:47.531099 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159620 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:17:47.531099 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159623 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:17:47.531099 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159625 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:17:47.531099 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159628 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:17:47.531099 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159630 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:17:47.531099 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159633 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:17:47.531099 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159635 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:17:47.531099 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159638 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:17:47.531099 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159640 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:17:47.531099 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159644 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:17:47.531099 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159648 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:17:47.531099 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159653 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:17:47.531099 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159657 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:17:47.531099 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159660 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:17:47.531968 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159664 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:17:47.531968 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159666 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:17:47.531968 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159670 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:17:47.531968 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159674 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:17:47.531968 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159677 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:17:47.531968 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159680 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:17:47.531968 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159683 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:17:47.531968 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159686 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:17:47.531968 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159690 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:17:47.531968 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159693 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:17:47.531968 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159697 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:17:47.531968 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159700 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:17:47.531968 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159703 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:17:47.531968 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159705 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:17:47.531968 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159708 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:17:47.531968 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159711 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:17:47.531968 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159715 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:17:47.531968 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159718 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:17:47.531968 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159720 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:17:47.532837 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159723 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:17:47.532837 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159725 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:17:47.532837 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159727 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:17:47.532837 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159730 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:17:47.532837 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159732 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:17:47.532837 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159735 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:17:47.532837 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159737 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:17:47.532837 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159740 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:17:47.532837 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159742 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:17:47.532837 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159745 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:17:47.532837 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159749 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:17:47.532837 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159752 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:17:47.532837 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159755 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:17:47.532837 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159757 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:17:47.532837 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159760 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:17:47.532837 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.159763 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:17:47.532837 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159851 2559 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 00:17:47.532837 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159860 2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 00:17:47.532837 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159867 2559 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 00:17:47.532837 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159872 2559 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 00:17:47.532837 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159877 2559 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 00:17:47.533912 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159880 2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 00:17:47.533912 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159884 2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 00:17:47.533912 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159889 2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 00:17:47.533912 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159893 2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 00:17:47.533912 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159896 2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 00:17:47.533912 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159900 2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 00:17:47.533912 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159903 2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 00:17:47.533912 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159906 2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 00:17:47.533912 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159910 2559 flags.go:64] FLAG: --cgroup-root="" Jul 10 00:17:47.533912 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159913 2559 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 00:17:47.533912 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159916 2559 flags.go:64] FLAG: --client-ca-file="" Jul 10 00:17:47.533912 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159920 2559 flags.go:64] FLAG: --cloud-provider="external" Jul 10 00:17:47.533912 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159923 2559 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 00:17:47.533912 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159927 2559 flags.go:64] FLAG: --cluster-domain="" Jul 10 00:17:47.533912 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159930 2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 00:17:47.533912 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159933 2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 00:17:47.533912 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159937 2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 00:17:47.533912 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159940 2559 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 00:17:47.533912 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159944 2559 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 00:17:47.533912 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159948 2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 00:17:47.533912 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159951 2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 00:17:47.533912 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159956 2559 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 00:17:47.533912 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159960 2559 flags.go:64] FLAG: --contention-profiling="false" Jul 10 00:17:47.533912 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159963 2559 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 00:17:47.534973 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159966 2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 00:17:47.534973 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159970 2559 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 00:17:47.534973 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159973 2559 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 00:17:47.534973 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159977 2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 00:17:47.534973 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159980 2559 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 00:17:47.534973 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159983 2559 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 00:17:47.534973 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159987 2559 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 00:17:47.534973 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159990 2559 flags.go:64] FLAG: --enable-server="true" Jul 10 00:17:47.534973 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159993 2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 00:17:47.534973 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.159999 2559 flags.go:64] FLAG: --event-burst="100" Jul 10 00:17:47.534973 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160002 2559 flags.go:64] FLAG: --event-qps="50" Jul 10 00:17:47.534973 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160005 2559 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 00:17:47.534973 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160009 2559 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 00:17:47.534973 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160012 2559 flags.go:64] FLAG: --eviction-hard="" Jul 10 00:17:47.534973 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160016 2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 00:17:47.534973 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160019 2559 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 00:17:47.534973 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160022 2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 00:17:47.534973 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160026 2559 flags.go:64] FLAG: --eviction-soft="" Jul 10 00:17:47.534973 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160029 2559 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 00:17:47.534973 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160032 2559 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 00:17:47.534973 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160035 2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 00:17:47.534973 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160038 2559 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 00:17:47.534973 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160041 2559 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 00:17:47.534973 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160043 2559 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 00:17:47.534973 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160046 2559 flags.go:64] FLAG: --feature-gates="" Jul 10 00:17:47.536150 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160052 2559 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 00:17:47.536150 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160055 2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 00:17:47.536150 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160058 2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 00:17:47.536150 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160061 2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 00:17:47.536150 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160066 2559 flags.go:64] FLAG: --healthz-port="10248" Jul 10 00:17:47.536150 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160069 2559 flags.go:64] FLAG: --help="false" Jul 10 00:17:47.536150 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160072 2559 flags.go:64] FLAG: --hostname-override="ip-10-0-132-30.ec2.internal" Jul 10 00:17:47.536150 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160075 2559 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 00:17:47.536150 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160078 2559 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 00:17:47.536150 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160081 2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 00:17:47.536150 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160084 2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 00:17:47.536150 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160087 2559 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 00:17:47.536150 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160090 2559 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 00:17:47.536150 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160093 2559 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 00:17:47.536150 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160096 2559 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 00:17:47.536150 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160099 2559 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 00:17:47.536150 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160101 2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 00:17:47.536150 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160104 2559 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 00:17:47.536150 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160107 2559 flags.go:64] FLAG: --kube-reserved="" Jul 10 00:17:47.536150 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160111 2559 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 00:17:47.536150 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160113 2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 00:17:47.536150 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160116 2559 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 00:17:47.536150 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160119 2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 00:17:47.536150 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160122 2559 flags.go:64] FLAG: --lock-file="" Jul 10 00:17:47.537546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160125 2559 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 00:17:47.537546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160128 2559 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 00:17:47.537546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160131 2559 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 00:17:47.537546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160137 2559 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 00:17:47.537546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160140 2559 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 00:17:47.537546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160143 2559 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 00:17:47.537546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160146 2559 flags.go:64] FLAG: --logging-format="text" Jul 10 00:17:47.537546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160149 2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 00:17:47.537546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160154 2559 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 00:17:47.537546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160157 2559 flags.go:64] FLAG: --manifest-url="" Jul 10 00:17:47.537546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160160 2559 flags.go:64] FLAG: --manifest-url-header="" Jul 10 00:17:47.537546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160164 2559 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 00:17:47.537546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160170 2559 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 00:17:47.537546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160175 2559 flags.go:64] FLAG: --max-pods="110" Jul 10 00:17:47.537546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160178 2559 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 00:17:47.537546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160181 2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 00:17:47.537546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160184 2559 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 00:17:47.537546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160187 2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 00:17:47.537546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160191 2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 00:17:47.537546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160194 2559 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 00:17:47.537546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160196 2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 00:17:47.537546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160204 2559 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 00:17:47.537546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160207 2559 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 00:17:47.537546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160211 2559 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 00:17:47.537546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160214 2559 flags.go:64] FLAG: --pod-cidr="" Jul 10 00:17:47.538632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160217 2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 00:17:47.538632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160237 2559 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 00:17:47.538632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160240 2559 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 00:17:47.538632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160244 2559 flags.go:64] FLAG: --pods-per-core="0" Jul 10 00:17:47.538632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160247 2559 flags.go:64] FLAG: --port="10250" Jul 10 00:17:47.538632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160250 2559 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 00:17:47.538632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160253 2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09b6a7e2a5815fc96" Jul 10 00:17:47.538632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160257 2559 flags.go:64] FLAG: --qos-reserved="" Jul 10 00:17:47.538632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160260 2559 flags.go:64] FLAG: --read-only-port="10255" Jul 10 00:17:47.538632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160263 2559 flags.go:64] FLAG: --register-node="true" Jul 10 00:17:47.538632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160266 2559 flags.go:64] FLAG: --register-with-taints="" Jul 10 00:17:47.538632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160270 2559 flags.go:64] FLAG: --registry-burst="10" Jul 10 00:17:47.538632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160273 2559 flags.go:64] FLAG: --registry-qps="5" Jul 10 00:17:47.538632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160276 2559 flags.go:64] FLAG: --reserved-cpus="" Jul 10 00:17:47.538632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160279 2559 flags.go:64] FLAG: --reserved-memory="" Jul 10 00:17:47.538632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160282 2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 00:17:47.538632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160287 2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 00:17:47.538632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160290 2559 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 00:17:47.538632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160293 2559 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 00:17:47.538632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160297 2559 flags.go:64] FLAG: --runonce="false" Jul 10 00:17:47.538632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160300 2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 00:17:47.538632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160303 2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 00:17:47.538632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160306 2559 flags.go:64] FLAG: --seccomp-default="false" Jul 10 00:17:47.538632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160309 2559 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 00:17:47.538632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160312 2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 00:17:47.539723 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160315 2559 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 00:17:47.539723 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160318 2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 00:17:47.539723 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160321 2559 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 00:17:47.539723 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160324 2559 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 00:17:47.539723 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160327 2559 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 00:17:47.539723 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160330 2559 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 00:17:47.539723 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160333 2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 00:17:47.539723 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160336 2559 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 00:17:47.539723 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160339 2559 flags.go:64] FLAG: --system-cgroups="" Jul 10 00:17:47.539723 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160342 2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 00:17:47.539723 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160347 2559 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 00:17:47.539723 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160350 2559 flags.go:64] FLAG: --tls-cert-file="" Jul 10 00:17:47.539723 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160353 2559 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 00:17:47.539723 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160358 2559 flags.go:64] FLAG: --tls-min-version="" Jul 10 00:17:47.539723 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160360 2559 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 00:17:47.539723 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160363 2559 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 00:17:47.539723 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160366 2559 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 00:17:47.539723 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160370 2559 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 00:17:47.539723 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160373 2559 flags.go:64] FLAG: --v="2" Jul 10 00:17:47.539723 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160376 2559 flags.go:64] FLAG: --version="false" Jul 10 00:17:47.539723 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160380 2559 flags.go:64] FLAG: --vmodule="" Jul 10 00:17:47.539723 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160384 2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 00:17:47.539723 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.160387 2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 00:17:47.539723 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160508 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:17:47.539723 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160513 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:17:47.540744 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160516 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:17:47.540744 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160522 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:17:47.540744 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160526 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:17:47.540744 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160529 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:17:47.540744 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160532 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:17:47.540744 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160535 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:17:47.540744 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160538 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:17:47.540744 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160541 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:17:47.540744 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160543 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:17:47.540744 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160546 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:17:47.540744 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160549 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:17:47.540744 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160552 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:17:47.540744 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160554 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:17:47.540744 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160557 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:17:47.540744 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160559 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:17:47.540744 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160562 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:17:47.540744 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160564 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:17:47.540744 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160567 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:17:47.540744 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160571 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:17:47.545201 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160575 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:17:47.545201 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160578 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:17:47.545201 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160581 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:17:47.545201 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160584 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:17:47.545201 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160587 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:17:47.545201 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160590 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:17:47.545201 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160593 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:17:47.545201 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160596 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:17:47.545201 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160598 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:17:47.545201 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160601 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:17:47.545201 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160603 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:17:47.545201 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160607 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:17:47.545201 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160610 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:17:47.545201 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160612 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:17:47.545201 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160616 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:17:47.545201 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160619 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:17:47.545201 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160622 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:17:47.545201 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160624 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:17:47.545201 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160627 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:17:47.545201 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160630 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:17:47.567733 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160632 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:17:47.567733 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160635 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:17:47.567733 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160639 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:17:47.567733 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160642 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:17:47.567733 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160645 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:17:47.567733 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160648 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:17:47.567733 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160650 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:17:47.567733 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160653 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:17:47.567733 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160655 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:17:47.567733 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160658 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:17:47.567733 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160660 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:17:47.567733 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160663 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:17:47.567733 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160665 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:17:47.567733 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160668 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:17:47.567733 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160670 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:17:47.567733 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160673 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:17:47.567733 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160675 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:17:47.567733 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160678 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:17:47.567733 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160680 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:17:47.568370 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160683 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:17:47.568370 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160685 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:17:47.568370 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160688 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:17:47.568370 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160691 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:17:47.568370 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160696 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:17:47.568370 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160699 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:17:47.568370 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160702 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:17:47.568370 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160707 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:17:47.568370 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160710 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:17:47.568370 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160712 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:17:47.568370 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160715 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:17:47.568370 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160718 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:17:47.568370 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160720 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:17:47.568370 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160723 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:17:47.568370 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160725 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:17:47.568370 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160728 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:17:47.568370 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160730 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:17:47.568370 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160733 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:17:47.568370 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160735 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:17:47.568370 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160738 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:17:47.568940 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160740 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:17:47.568940 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160743 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:17:47.568940 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160746 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:17:47.568940 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160748 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:17:47.568940 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160751 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:17:47.568940 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160753 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:17:47.568940 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160756 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:17:47.568940 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160758 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:17:47.568940 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160761 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:17:47.568940 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160763 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:17:47.568940 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160766 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:17:47.568940 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160768 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:17:47.568940 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160771 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:17:47.568940 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160774 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:17:47.568940 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160777 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:17:47.568940 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160779 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:17:47.568940 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160785 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:17:47.568940 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160788 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:17:47.568940 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.160791 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:17:47.569598 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.161432 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 00:17:47.569598 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.168762 2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 00:17:47.569598 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.168778 2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 00:17:47.569598 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.168798 2559 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 00:17:47.569598 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.168808 2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 00:17:47.569598 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168872 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:17:47.569598 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168877 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:17:47.569598 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168880 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:17:47.569598 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168884 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:17:47.569598 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168887 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:17:47.569598 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168890 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:17:47.569598 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168893 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:17:47.569598 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168896 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:17:47.569598 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168899 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:17:47.569598 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168901 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:17:47.570028 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168904 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:17:47.570028 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168906 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:17:47.570028 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168909 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:17:47.570028 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168911 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:17:47.570028 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168914 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:17:47.570028 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168917 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:17:47.570028 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168919 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:17:47.570028 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168922 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:17:47.570028 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168927 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:17:47.570028 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168932 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:17:47.570028 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168936 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:17:47.570028 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168945 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:17:47.570028 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168948 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:17:47.570028 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168951 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:17:47.570028 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168954 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:17:47.570028 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168956 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:17:47.570028 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168959 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:17:47.570028 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168961 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:17:47.570028 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168965 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:17:47.570967 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168967 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:17:47.570967 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168970 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:17:47.570967 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168973 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:17:47.570967 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168975 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:17:47.570967 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168978 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:17:47.570967 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168981 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:17:47.570967 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168983 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:17:47.570967 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168986 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:17:47.570967 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168989 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:17:47.570967 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168991 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:17:47.570967 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168994 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:17:47.570967 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168997 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:17:47.570967 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.168999 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:17:47.570967 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169002 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:17:47.570967 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169005 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:17:47.570967 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169007 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:17:47.570967 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169010 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:17:47.570967 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169012 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:17:47.570967 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169015 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:17:47.570967 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169017 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:17:47.571530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169019 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:17:47.571530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169022 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:17:47.571530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169025 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:17:47.571530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169028 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:17:47.571530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169031 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:17:47.571530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169039 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:17:47.571530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169042 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:17:47.571530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169045 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:17:47.571530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169048 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:17:47.571530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169050 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:17:47.571530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169053 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:17:47.571530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169056 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:17:47.571530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169059 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:17:47.571530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169061 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:17:47.571530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169064 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:17:47.571530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169066 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:17:47.571530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169069 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:17:47.571530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169072 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:17:47.571530 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169074 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:17:47.785999 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169077 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:17:47.785999 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169079 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:17:47.785999 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169082 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:17:47.785999 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169084 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:17:47.785999 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169087 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:17:47.785999 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169089 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:17:47.785999 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169092 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:17:47.785999 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169094 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:17:47.785999 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169097 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:17:47.785999 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169100 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:17:47.785999 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169102 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:17:47.785999 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169104 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:17:47.785999 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169107 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:17:47.785999 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169110 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:17:47.785999 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169116 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:17:47.785999 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169118 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:17:47.785999 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169121 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:17:47.785999 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169123 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:17:47.785999 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169126 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:17:47.785999 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169137 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:17:47.787826 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169139 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:17:47.787826 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169142 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:17:47.787826 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169144 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:17:47.787826 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169147 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:17:47.787826 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169149 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:17:47.787826 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169153 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:17:47.787826 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169157 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:17:47.787826 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169160 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:17:47.787826 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169163 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:17:47.787826 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169166 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:17:47.787826 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169169 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:17:47.787826 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.169176 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 00:17:47.787826 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169362 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:17:47.787826 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169368 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:17:47.787826 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169371 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:17:47.788869 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169374 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:17:47.788869 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169377 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:17:47.788869 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169380 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:17:47.788869 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169382 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:17:47.788869 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169385 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:17:47.788869 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169388 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:17:47.788869 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169390 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:17:47.788869 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169394 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:17:47.788869 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169396 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:17:47.788869 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169398 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:17:47.788869 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169401 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:17:47.788869 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169404 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:17:47.788869 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169406 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:17:47.788869 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169409 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:17:47.788869 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169411 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:17:47.788869 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169414 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:17:47.788869 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169418 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:17:47.788869 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169427 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:17:47.788869 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169430 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:17:47.788869 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169432 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:17:47.791084 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169435 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:17:47.791084 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169437 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:17:47.791084 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169440 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:17:47.791084 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169443 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:17:47.791084 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169445 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:17:47.791084 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169448 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:17:47.791084 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169451 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:17:47.791084 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169453 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:17:47.791084 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169455 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:17:47.791084 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169459 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:17:47.791084 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169462 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:17:47.791084 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169465 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:17:47.791084 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169467 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:17:47.791084 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169470 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:17:47.791084 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169472 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:17:47.791084 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169475 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:17:47.791084 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169477 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:17:47.791084 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169480 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:17:47.791084 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169482 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:17:47.791669 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169485 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:17:47.791669 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169487 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:17:47.791669 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169490 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:17:47.791669 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169494 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:17:47.791669 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169497 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:17:47.791669 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169500 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:17:47.791669 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169503 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:17:47.791669 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169505 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:17:47.791669 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169508 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:17:47.791669 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169512 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:17:47.791669 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169515 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:17:47.791669 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169525 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:17:47.791669 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169528 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:17:47.791669 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169531 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:17:47.791669 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169534 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:17:47.791669 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169537 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:17:47.791669 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169539 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:17:47.791669 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169542 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:17:47.791669 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169545 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:17:47.792229 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169547 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:17:47.792229 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169550 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:17:47.792229 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169552 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:17:47.792229 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169556 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:17:47.792229 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169559 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:17:47.792229 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169561 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:17:47.792229 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169564 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:17:47.792229 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169566 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:17:47.792229 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169569 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:17:47.792229 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169571 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:17:47.792229 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169574 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:17:47.792229 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169577 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:17:47.792229 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169579 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:17:47.792229 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169582 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:17:47.792229 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169585 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:17:47.792229 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169587 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:17:47.792229 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169590 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:17:47.792229 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169592 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:17:47.792229 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169595 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:17:47.792229 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169598 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:17:47.792904 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169600 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:17:47.792904 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169603 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:17:47.792904 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169607 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:17:47.792904 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169610 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:17:47.792904 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169612 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:17:47.792904 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169620 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:17:47.792904 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169625 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:17:47.792904 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169628 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:17:47.792904 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169631 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:17:47.792904 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169634 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:17:47.792904 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169637 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:17:47.792904 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169640 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:17:47.792904 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169643 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:17:47.792904 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169645 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:17:47.792904 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169648 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:17:47.792904 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169652 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:17:47.792904 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169655 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:17:47.792904 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:46.169658 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:17:47.793457 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.169663 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 00:17:47.793457 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.169810 2559 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 00:17:47.793457 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.171745 2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 00:17:47.793457 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.173128 2559 server.go:1019] "Starting client certificate rotation" Jul 10 00:17:47.793457 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.173256 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 00:17:47.793457 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.173819 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 00:17:47.793457 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.175515 2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 00:17:47.793457 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.177660 2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 00:17:47.793457 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.195584 2559 log.go:25] "Validated CRI v1 runtime API" Jul 10 00:17:47.793457 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.197483 2559 log.go:25] "Validated CRI v1 image API" Jul 10 00:17:47.793457 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.198264 2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 00:17:47.793457 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.201774 2559 fs.go:135] Filesystem UUIDs: map[05cc0579-7d6c-4fa0-8bb2-c050844b4b6e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 af6fe266-6857-4c00-b6c2-4090985bb694:/dev/nvme0n1p4] Jul 10 00:17:47.793850 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.201791 2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 00:17:47.793850 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.203537 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 00:17:47.794213 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.206878 2559 manager.go:216] Machine: {Timestamp:2026-07-10 00:17:46.20548756 +0000 UTC m=+0.343465152 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099054 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c4a5409002c1f945d74e03fce0db2 SystemUUID:ec2c4a54-0900-2c1f-945d-74e03fce0db2 BootID:fd9739c1-6a89-4d02-b80a-1602c490592c Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:83:f4:a6:2c:35 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:83:f4:a6:2c:35 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:3c:60:52:ec:94 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 00:17:47.794213 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.206977 2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 00:17:47.794213 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.207048 2559 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 00:17:47.794213 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.210272 2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 00:17:47.794213 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.210308 2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-30.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 00:17:47.794213 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.211729 2559 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 00:17:47.794213 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.211740 2559 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 00:17:47.794213 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.211747 2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 00:17:47.794213 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.212291 2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 00:17:47.794213 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.213675 2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 00:17:47.794213 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.217422 2559 state_mem.go:36] "Initialized new in-memory state store" Jul 10 00:17:47.794213 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.217527 2559 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 00:17:47.794213 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.217552 2559 kubelet.go:486] "Attempting to sync node with API server" Jul 10 00:17:47.794213 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.217561 2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 00:17:47.794213 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.217579 2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 00:17:47.794213 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.217587 2559 kubelet.go:398] "Adding apiserver pod source" Jul 10 00:17:47.794213 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.217596 2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 00:17:47.794213 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.217900 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.217916 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.220657 2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.222642 2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.222684 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g5ggp" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.222703 2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.224850 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.224864 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.224871 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.224877 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.224887 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.224893 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.224901 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.224907 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.224915 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.224922 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.224931 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.224938 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.224964 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.224970 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.228525 2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-30.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.228627 2559 server.go:1268] "Started kubelet" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.228758 2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.228725 2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.228876 2559 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:46.228773 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-30.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:46.228774 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.230597 2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.231431 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g5ggp" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.232770 2559 server.go:310] "Adding debug handlers to kubelet server" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:46.234582 2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:46.236522 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-30.ec2.internal.18c0c58e06c93e42 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-30.ec2.internal,UID:ip-10-0-132-30.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-30.ec2.internal,},FirstTimestamp:2026-07-10 00:17:46.22859629 +0000 UTC m=+0.366573884,LastTimestamp:2026-07-10 00:17:46.22859629 +0000 UTC m=+0.366573884,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-30.ec2.internal,}" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.237855 2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.237863 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:46.237932 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-30.ec2.internal\" not found" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.238307 2559 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.238313 2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.238322 2559 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.238441 2559 reconstruct.go:97] "Volume reconstruction finished" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.238451 2559 reconciler.go:29] "Reconciler: start to sync state" Jul 10 00:17:47.794849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.238596 2559 factory.go:55] Registering systemd factory Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.238610 2559 factory.go:223] Registration of the systemd container factory successfully Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.238803 2559 factory.go:153] Registering CRI-O factory Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.238991 2559 factory.go:223] Registration of the crio container factory successfully Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.239091 2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.239127 2559 factory.go:103] Registering Raw factory Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.239145 2559 manager.go:1196] Started watching for new ooms in manager Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.239711 2559 manager.go:349] Starting recovery of all containers Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.246932 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:46.249137 2559 manager.go:354] Recovery completed Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:47.454633 2559 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-30.ec2.internal\" not found" node="ip-10-0-132-30.ec2.internal" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.459878 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.463889 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-30.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.463912 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-30.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.463921 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-30.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.464586 2559 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.464597 2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.464616 2559 state_mem.go:36] "Initialized new in-memory state store" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.464800 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.466270 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-30.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.466292 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-30.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.466306 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-30.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.467634 2559 policy_none.go:49] "None policy: Start" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.467648 2559 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.467661 2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.469734 2559 policy_none.go:47] "Start" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.495952 2559 manager.go:337] "Starting Device Plugin manager" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:47.495980 2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.495989 2559 server.go:87] "Starting device plugin registration server" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.496240 2559 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.496253 2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.496347 2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.496437 2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.496448 2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:47.497520 2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:47.497562 2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-30.ec2.internal\" not found" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.597238 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.598056 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-30.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.598082 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-30.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.598092 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-30.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.598130 2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-30.ec2.internal" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.605725 2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-30.ec2.internal" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:47.605745 2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-30.ec2.internal\": node \"ip-10-0-132-30.ec2.internal\" not found" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:47.622540 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-30.ec2.internal\" not found" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.642011 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.643291 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.643325 2559 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.643347 2559 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:47.643388 2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 00:17:47.796005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.646298 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:17:47.797395 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:47.722824 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-30.ec2.internal\" not found" Jul 10 00:17:47.797395 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.744068 2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-30.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-30.ec2.internal"] Jul 10 00:17:47.797395 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.744150 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:17:47.797395 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.745096 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-30.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:17:47.797395 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.745147 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-30.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:17:47.797395 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.745164 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-30.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:17:47.797395 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.747627 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:17:47.797395 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.747748 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-30.ec2.internal" Jul 10 00:17:47.797395 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.747783 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:17:47.797395 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.748329 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-30.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:17:47.797395 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.748358 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-30.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:17:47.797395 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.748374 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-30.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:17:47.797395 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.748408 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-30.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:17:47.797395 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.748430 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-30.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:17:47.797395 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.748444 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-30.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:17:47.797395 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.750662 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-30.ec2.internal" Jul 10 00:17:47.797395 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.750686 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:17:47.797395 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.751330 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-30.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:17:47.797395 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.751360 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-30.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:17:47.797395 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.751372 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-30.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:17:47.797395 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:47.766716 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-30.ec2.internal\" not found" node="ip-10-0-132-30.ec2.internal" Jul 10 00:17:47.797395 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:47.772235 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-30.ec2.internal\" not found" node="ip-10-0-132-30.ec2.internal" Jul 10 00:17:48.015149 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:47.823449 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-30.ec2.internal\" not found" Jul 10 00:17:48.015149 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.850006 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fea6ac5a21d75add74f7c2411fe7019d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-30.ec2.internal\" (UID: \"fea6ac5a21d75add74f7c2411fe7019d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-30.ec2.internal" Jul 10 00:17:48.015149 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.850035 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71ce3f1111f97924efae8ea5aa51d9e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-30.ec2.internal\" (UID: \"71ce3f1111f97924efae8ea5aa51d9e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-30.ec2.internal" Jul 10 00:17:48.015149 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.850053 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71ce3f1111f97924efae8ea5aa51d9e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-30.ec2.internal\" (UID: \"71ce3f1111f97924efae8ea5aa51d9e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-30.ec2.internal" Jul 10 00:17:48.015149 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:47.924015 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-30.ec2.internal\" not found" Jul 10 00:17:48.015149 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.950184 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71ce3f1111f97924efae8ea5aa51d9e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-30.ec2.internal\" (UID: \"71ce3f1111f97924efae8ea5aa51d9e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-30.ec2.internal" Jul 10 00:17:48.015149 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.950218 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71ce3f1111f97924efae8ea5aa51d9e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-30.ec2.internal\" (UID: \"71ce3f1111f97924efae8ea5aa51d9e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-30.ec2.internal" Jul 10 00:17:48.015149 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.950247 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/71ce3f1111f97924efae8ea5aa51d9e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-30.ec2.internal\" (UID: \"71ce3f1111f97924efae8ea5aa51d9e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-30.ec2.internal" Jul 10 00:17:48.015149 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.950247 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fea6ac5a21d75add74f7c2411fe7019d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-30.ec2.internal\" (UID: \"fea6ac5a21d75add74f7c2411fe7019d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-30.ec2.internal" Jul 10 00:17:48.015149 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.950286 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71ce3f1111f97924efae8ea5aa51d9e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-30.ec2.internal\" (UID: \"71ce3f1111f97924efae8ea5aa51d9e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-30.ec2.internal" Jul 10 00:17:48.015149 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:47.950306 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fea6ac5a21d75add74f7c2411fe7019d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-30.ec2.internal\" (UID: \"fea6ac5a21d75add74f7c2411fe7019d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-30.ec2.internal" Jul 10 00:17:48.205292 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:48.025310 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-30.ec2.internal\" not found" Jul 10 00:17:48.205292 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:48.070772 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-30.ec2.internal" Jul 10 00:17:48.205292 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:48.075757 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-30.ec2.internal" Jul 10 00:17:48.205292 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:48.126342 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-30.ec2.internal\" not found" Jul 10 00:17:48.389967 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:48.226784 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-30.ec2.internal\" not found" Jul 10 00:17:48.389967 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:48.327236 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-30.ec2.internal\" not found" Jul 10 00:17:48.389967 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:48.377689 2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 00:17:48.389967 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:48.377817 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 00:17:48.389967 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:48.377849 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 00:17:48.568285 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:48.428154 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-30.ec2.internal\" not found" Jul 10 00:17:48.568285 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:48.442443 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 00:12:47 +0000 UTC" deadline="2028-01-02 18:38:42.320500606 +0000 UTC" Jul 10 00:17:48.568285 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:48.442465 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13002h20m53.87803913s" Jul 10 00:17:48.568285 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:48.528257 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-30.ec2.internal\" not found" Jul 10 00:17:48.692409 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:48.628764 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-30.ec2.internal\" not found" Jul 10 00:17:48.849968 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:48.706012 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ce3f1111f97924efae8ea5aa51d9e5.slice/crio-b6e3cbe9327b528f87493a0bf8e454c4161bd01696e007563fc202d2bc66fc3c WatchSource:0}: Error finding container b6e3cbe9327b528f87493a0bf8e454c4161bd01696e007563fc202d2bc66fc3c: Status 404 returned error can't find the container with id b6e3cbe9327b528f87493a0bf8e454c4161bd01696e007563fc202d2bc66fc3c Jul 10 00:17:48.849968 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:48.706700 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea6ac5a21d75add74f7c2411fe7019d.slice/crio-1887fa084889092305120fceab3cab960ef80a06ccf5dee6dedd8c83d83c7db7 WatchSource:0}: Error finding container 1887fa084889092305120fceab3cab960ef80a06ccf5dee6dedd8c83d83c7db7: Status 404 returned error can't find the container with id 1887fa084889092305120fceab3cab960ef80a06ccf5dee6dedd8c83d83c7db7 Jul 10 00:17:48.849968 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:48.718706 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 00:17:48.849968 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:48.729611 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-30.ec2.internal\" not found" Jul 10 00:17:48.849968 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:48.775052 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:17:48.849968 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:48.829941 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-30.ec2.internal\" not found" Jul 10 00:17:48.849968 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:48.832841 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:17:48.849968 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:48.843374 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-30.ec2.internal" Jul 10 00:17:49.071084 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:48.859618 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 00:17:49.071084 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:48.860557 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-30.ec2.internal" Jul 10 00:17:49.071084 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:48.866199 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 00:17:49.436903 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.424012 2559 apiserver.go:52] "Watching apiserver" Jul 10 00:17:49.436903 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.435004 2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 00:17:49.436903 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.435398 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-30.ec2.internal","openshift-multus/network-metrics-daemon-ms9mf","kube-system/global-pull-secret-syncer-zqzc6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp","openshift-multus/multus-additional-cni-plugins-8hp4r","openshift-multus/multus-hf6sh","openshift-network-diagnostics/network-check-target-wrzt4","openshift-network-operator/iptables-alerter-n9prh","openshift-ovn-kubernetes/ovnkube-node-p7zrr","kube-system/konnectivity-agent-k2hs8","kube-system/kube-apiserver-proxy-ip-10-0-132-30.ec2.internal","openshift-cluster-node-tuning-operator/tuned-nsx5f","openshift-dns/node-resolver-gzg2p","openshift-image-registry/node-ca-jdrfw"] Jul 10 00:17:49.438168 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.437987 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:17:49.438168 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:49.438074 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrzt4" podUID="4c396c5d-d578-4bc2-b02e-0d1f129d8247" Jul 10 00:17:49.440631 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.440137 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gzg2p" Jul 10 00:17:49.442291 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.442250 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 00:17:49.442504 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.442453 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lbvjh\"" Jul 10 00:17:49.442598 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.442544 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 00:17:49.443059 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.442874 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:17:49.443059 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:49.442941 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqzc6" podUID="b1c3645d-2300-4b32-b0ef-e373064e5aad" Jul 10 00:17:49.443805 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.443601 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 00:17:49.445351 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.445111 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" Jul 10 00:17:49.447440 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.447076 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9dtgm\"" Jul 10 00:17:49.447440 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.447127 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 00:17:49.447440 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.447297 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 00:17:49.447440 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.447373 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 00:17:49.447440 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.447401 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8hp4r" Jul 10 00:17:49.450416 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.450042 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.451167 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.450876 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 00:17:49.451167 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.450941 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8c825\"" Jul 10 00:17:49.451167 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.450876 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 00:17:49.451167 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.450991 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 00:17:49.451167 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.451151 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 00:17:49.451620 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.451602 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 00:17:49.452040 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.452028 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jdrfw" Jul 10 00:17:49.453035 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.453016 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4pdxd\"" Jul 10 00:17:49.453183 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.453061 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 00:17:49.454424 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.453955 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 00:17:49.454424 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.454249 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 00:17:49.454424 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.454278 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cnzl5\"" Jul 10 00:17:49.454424 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.454302 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 00:17:49.454718 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.454699 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n9prh" Jul 10 00:17:49.455165 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.455147 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 00:17:49.456631 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.456502 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 00:17:49.457405 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.456679 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wf4s2\"" Jul 10 00:17:49.457405 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.456820 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/57273bce-c5a5-4b08-86b5-8f69c2eed2de-etc-selinux\") pod \"aws-ebs-csi-driver-node-2hkpp\" (UID: \"57273bce-c5a5-4b08-86b5-8f69c2eed2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" Jul 10 00:17:49.457405 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.456852 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/733f8b73-939c-414d-9d90-7791508e5b84-hosts-file\") pod \"node-resolver-gzg2p\" (UID: \"733f8b73-939c-414d-9d90-7791508e5b84\") " pod="openshift-dns/node-resolver-gzg2p" Jul 10 00:17:49.457405 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.456880 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1c3645d-2300-4b32-b0ef-e373064e5aad-original-pull-secret\") pod \"global-pull-secret-syncer-zqzc6\" (UID: \"b1c3645d-2300-4b32-b0ef-e373064e5aad\") " pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:17:49.457405 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.456907 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/57273bce-c5a5-4b08-86b5-8f69c2eed2de-socket-dir\") pod \"aws-ebs-csi-driver-node-2hkpp\" (UID: \"57273bce-c5a5-4b08-86b5-8f69c2eed2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" Jul 10 00:17:49.457405 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.456929 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/57273bce-c5a5-4b08-86b5-8f69c2eed2de-registration-dir\") pod \"aws-ebs-csi-driver-node-2hkpp\" (UID: \"57273bce-c5a5-4b08-86b5-8f69c2eed2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" Jul 10 00:17:49.457405 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.456950 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/57273bce-c5a5-4b08-86b5-8f69c2eed2de-device-dir\") pod \"aws-ebs-csi-driver-node-2hkpp\" (UID: \"57273bce-c5a5-4b08-86b5-8f69c2eed2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" Jul 10 00:17:49.457405 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.456975 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdbw5\" (UniqueName: \"kubernetes.io/projected/4c396c5d-d578-4bc2-b02e-0d1f129d8247-kube-api-access-kdbw5\") pod \"network-check-target-wrzt4\" (UID: \"4c396c5d-d578-4bc2-b02e-0d1f129d8247\") " pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:17:49.457405 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.456996 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/733f8b73-939c-414d-9d90-7791508e5b84-tmp-dir\") pod \"node-resolver-gzg2p\" (UID: \"733f8b73-939c-414d-9d90-7791508e5b84\") " pod="openshift-dns/node-resolver-gzg2p" Jul 10 00:17:49.457405 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.457019 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dqhp\" (UniqueName: \"kubernetes.io/projected/733f8b73-939c-414d-9d90-7791508e5b84-kube-api-access-8dqhp\") pod \"node-resolver-gzg2p\" (UID: \"733f8b73-939c-414d-9d90-7791508e5b84\") " pod="openshift-dns/node-resolver-gzg2p" Jul 10 00:17:49.457405 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.457043 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bprg\" (UniqueName: \"kubernetes.io/projected/57273bce-c5a5-4b08-86b5-8f69c2eed2de-kube-api-access-2bprg\") pod \"aws-ebs-csi-driver-node-2hkpp\" (UID: \"57273bce-c5a5-4b08-86b5-8f69c2eed2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" Jul 10 00:17:49.457405 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.457065 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b1c3645d-2300-4b32-b0ef-e373064e5aad-kubelet-config\") pod \"global-pull-secret-syncer-zqzc6\" (UID: \"b1c3645d-2300-4b32-b0ef-e373064e5aad\") " pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:17:49.457405 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.457092 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b1c3645d-2300-4b32-b0ef-e373064e5aad-dbus\") pod \"global-pull-secret-syncer-zqzc6\" (UID: \"b1c3645d-2300-4b32-b0ef-e373064e5aad\") " pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:17:49.457405 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.457131 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/57273bce-c5a5-4b08-86b5-8f69c2eed2de-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2hkpp\" (UID: \"57273bce-c5a5-4b08-86b5-8f69c2eed2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" Jul 10 00:17:49.457405 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.457180 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/57273bce-c5a5-4b08-86b5-8f69c2eed2de-sys-fs\") pod \"aws-ebs-csi-driver-node-2hkpp\" (UID: \"57273bce-c5a5-4b08-86b5-8f69c2eed2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" Jul 10 00:17:49.458371 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.457829 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:17:49.458371 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:49.457938 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9mf" podUID="376630cb-0d38-46f3-a311-7cf9af6f2d10" Jul 10 00:17:49.458715 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.458597 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 00:17:49.458715 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.458597 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 00:17:49.459570 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.459539 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k2hs8" Jul 10 00:17:49.461700 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.461582 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-z5t2p\"" Jul 10 00:17:49.461700 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.461598 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 00:17:49.461700 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.461620 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 00:17:49.462218 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.462198 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.464092 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.464075 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 00:17:49.464092 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.464086 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 00:17:49.464249 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.464182 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ns7hv\"" Jul 10 00:17:49.467964 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.465204 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.467964 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.467091 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 00:17:49.467964 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.467385 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 00:17:49.467964 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.467509 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qjs5d\"" Jul 10 00:17:49.467964 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.467667 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 00:17:49.467964 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.467724 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 00:17:49.469681 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.468233 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 00:17:49.470764 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.470651 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 00:17:49.476040 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.476000 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-69x6r" Jul 10 00:17:49.484203 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.484178 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-69x6r" Jul 10 00:17:49.544251 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.544229 2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 00:17:49.558300 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.558269 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-etc-kubernetes\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.558445 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.558310 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-systemd-units\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.558445 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.558337 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-ovn-node-metrics-cert\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.558445 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.558362 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-system-cni-dir\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.558585 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.558465 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-multus-socket-dir-parent\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.558585 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.558490 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-host-var-lib-cni-multus\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.558585 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.558514 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nstk\" (UniqueName: \"kubernetes.io/projected/de6e5b48-34f6-43bb-8b9d-d1e651860753-kube-api-access-5nstk\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.558585 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.558540 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d642e8f4-ca88-46f0-ba74-e17a26ec2f7f-system-cni-dir\") pod \"multus-additional-cni-plugins-8hp4r\" (UID: \"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f\") " pod="openshift-multus/multus-additional-cni-plugins-8hp4r" Jul 10 00:17:49.558772 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.558585 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bprg\" (UniqueName: \"kubernetes.io/projected/57273bce-c5a5-4b08-86b5-8f69c2eed2de-kube-api-access-2bprg\") pod \"aws-ebs-csi-driver-node-2hkpp\" (UID: \"57273bce-c5a5-4b08-86b5-8f69c2eed2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" Jul 10 00:17:49.558772 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.558643 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-etc-sysconfig\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.558772 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.558670 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-host-kubelet\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.558772 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.558710 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-ovnkube-config\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.558772 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.558739 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-host-var-lib-kubelet\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.558772 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.558765 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d642e8f4-ca88-46f0-ba74-e17a26ec2f7f-cni-binary-copy\") pod \"multus-additional-cni-plugins-8hp4r\" (UID: \"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f\") " pod="openshift-multus/multus-additional-cni-plugins-8hp4r" Jul 10 00:17:49.559046 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.558789 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d642e8f4-ca88-46f0-ba74-e17a26ec2f7f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8hp4r\" (UID: \"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f\") " pod="openshift-multus/multus-additional-cni-plugins-8hp4r" Jul 10 00:17:49.559046 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.558818 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b1c3645d-2300-4b32-b0ef-e373064e5aad-dbus\") pod \"global-pull-secret-syncer-zqzc6\" (UID: \"b1c3645d-2300-4b32-b0ef-e373064e5aad\") " pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:17:49.559046 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.558841 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9de3defe-9135-40a5-994e-2a2b3c888c6d-etc-tuned\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.559046 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.558876 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-env-overrides\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.559046 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.558913 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-host-run-k8s-cni-cncf-io\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.559046 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.558944 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-hostroot\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.559046 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.558973 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-lib-modules\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.559046 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.558976 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b1c3645d-2300-4b32-b0ef-e373064e5aad-dbus\") pod \"global-pull-secret-syncer-zqzc6\" (UID: \"b1c3645d-2300-4b32-b0ef-e373064e5aad\") " pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:17:49.559046 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.558994 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-run-systemd\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.559432 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559051 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f158539b-d568-492a-a0b6-0266deb4109e-iptables-alerter-script\") pod \"iptables-alerter-n9prh\" (UID: \"f158539b-d568-492a-a0b6-0266deb4109e\") " pod="openshift-network-operator/iptables-alerter-n9prh" Jul 10 00:17:49.559432 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559104 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d642e8f4-ca88-46f0-ba74-e17a26ec2f7f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8hp4r\" (UID: \"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f\") " pod="openshift-multus/multus-additional-cni-plugins-8hp4r" Jul 10 00:17:49.559432 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559167 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/57273bce-c5a5-4b08-86b5-8f69c2eed2de-registration-dir\") pod \"aws-ebs-csi-driver-node-2hkpp\" (UID: \"57273bce-c5a5-4b08-86b5-8f69c2eed2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" Jul 10 00:17:49.559432 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559192 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-etc-sysctl-conf\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.559432 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559214 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-run\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.559432 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559241 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d642e8f4-ca88-46f0-ba74-e17a26ec2f7f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8hp4r\" (UID: \"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f\") " pod="openshift-multus/multus-additional-cni-plugins-8hp4r" Jul 10 00:17:49.559432 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559251 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/57273bce-c5a5-4b08-86b5-8f69c2eed2de-registration-dir\") pod \"aws-ebs-csi-driver-node-2hkpp\" (UID: \"57273bce-c5a5-4b08-86b5-8f69c2eed2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" Jul 10 00:17:49.559432 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559285 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs\") pod \"network-metrics-daemon-ms9mf\" (UID: \"376630cb-0d38-46f3-a311-7cf9af6f2d10\") " pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:17:49.559432 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559320 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdbw5\" (UniqueName: \"kubernetes.io/projected/4c396c5d-d578-4bc2-b02e-0d1f129d8247-kube-api-access-kdbw5\") pod \"network-check-target-wrzt4\" (UID: \"4c396c5d-d578-4bc2-b02e-0d1f129d8247\") " pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:17:49.559432 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559345 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/733f8b73-939c-414d-9d90-7791508e5b84-tmp-dir\") pod \"node-resolver-gzg2p\" (UID: \"733f8b73-939c-414d-9d90-7791508e5b84\") " pod="openshift-dns/node-resolver-gzg2p" Jul 10 00:17:49.559432 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559372 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dqhp\" (UniqueName: \"kubernetes.io/projected/733f8b73-939c-414d-9d90-7791508e5b84-kube-api-access-8dqhp\") pod \"node-resolver-gzg2p\" (UID: \"733f8b73-939c-414d-9d90-7791508e5b84\") " pod="openshift-dns/node-resolver-gzg2p" Jul 10 00:17:49.559432 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559397 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-etc-systemd\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.559948 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559440 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-run-ovn\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.559948 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559469 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxmgw\" (UniqueName: \"kubernetes.io/projected/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-kube-api-access-zxmgw\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.559948 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559494 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-host-var-lib-cni-bin\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.559948 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559516 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-multus-conf-dir\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.559948 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559541 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b1c3645d-2300-4b32-b0ef-e373064e5aad-kubelet-config\") pod \"global-pull-secret-syncer-zqzc6\" (UID: \"b1c3645d-2300-4b32-b0ef-e373064e5aad\") " pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:17:49.559948 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559583 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/57273bce-c5a5-4b08-86b5-8f69c2eed2de-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2hkpp\" (UID: \"57273bce-c5a5-4b08-86b5-8f69c2eed2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" Jul 10 00:17:49.559948 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559615 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-host-slash\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.559948 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559629 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b1c3645d-2300-4b32-b0ef-e373064e5aad-kubelet-config\") pod \"global-pull-secret-syncer-zqzc6\" (UID: \"b1c3645d-2300-4b32-b0ef-e373064e5aad\") " pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:17:49.559948 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559654 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de6e5b48-34f6-43bb-8b9d-d1e651860753-multus-daemon-config\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.559948 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559679 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/733f8b73-939c-414d-9d90-7791508e5b84-tmp-dir\") pod \"node-resolver-gzg2p\" (UID: \"733f8b73-939c-414d-9d90-7791508e5b84\") " pod="openshift-dns/node-resolver-gzg2p" Jul 10 00:17:49.559948 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559692 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/57273bce-c5a5-4b08-86b5-8f69c2eed2de-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2hkpp\" (UID: \"57273bce-c5a5-4b08-86b5-8f69c2eed2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" Jul 10 00:17:49.559948 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559683 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d642e8f4-ca88-46f0-ba74-e17a26ec2f7f-os-release\") pod \"multus-additional-cni-plugins-8hp4r\" (UID: \"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f\") " pod="openshift-multus/multus-additional-cni-plugins-8hp4r" Jul 10 00:17:49.559948 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559758 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-node-log\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.559948 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559785 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f158539b-d568-492a-a0b6-0266deb4109e-host-slash\") pod \"iptables-alerter-n9prh\" (UID: \"f158539b-d568-492a-a0b6-0266deb4109e\") " pod="openshift-network-operator/iptables-alerter-n9prh" Jul 10 00:17:49.559948 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559816 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/57273bce-c5a5-4b08-86b5-8f69c2eed2de-socket-dir\") pod \"aws-ebs-csi-driver-node-2hkpp\" (UID: \"57273bce-c5a5-4b08-86b5-8f69c2eed2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" Jul 10 00:17:49.559948 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559838 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-multus-cni-dir\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.559948 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559864 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de6e5b48-34f6-43bb-8b9d-d1e651860753-cni-binary-copy\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.560805 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559886 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-host-run-netns\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.560805 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559907 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/57273bce-c5a5-4b08-86b5-8f69c2eed2de-socket-dir\") pod \"aws-ebs-csi-driver-node-2hkpp\" (UID: \"57273bce-c5a5-4b08-86b5-8f69c2eed2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" Jul 10 00:17:49.560805 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559920 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-host-run-multus-certs\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.560805 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559948 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-var-lib-kubelet\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.560805 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559964 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-run-openvswitch\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.560805 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559980 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-cnibin\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.560805 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.559994 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-etc-kubernetes\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.560805 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560024 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f59c94d4-93a7-4602-bf06-b2e0a4b2e312-konnectivity-ca\") pod \"konnectivity-agent-k2hs8\" (UID: \"f59c94d4-93a7-4602-bf06-b2e0a4b2e312\") " pod="kube-system/konnectivity-agent-k2hs8" Jul 10 00:17:49.560805 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560062 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-etc-modprobe-d\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.560805 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560100 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-etc-sysctl-d\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.560805 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560141 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-host\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.560805 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560170 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slz7c\" (UniqueName: \"kubernetes.io/projected/d642e8f4-ca88-46f0-ba74-e17a26ec2f7f-kube-api-access-slz7c\") pod \"multus-additional-cni-plugins-8hp4r\" (UID: \"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f\") " pod="openshift-multus/multus-additional-cni-plugins-8hp4r" Jul 10 00:17:49.560805 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560195 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-sys\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.560805 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560214 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-etc-openvswitch\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.560805 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560228 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/613ac839-60fd-4f20-a127-a072d8a574bb-host\") pod \"node-ca-jdrfw\" (UID: \"613ac839-60fd-4f20-a127-a072d8a574bb\") " pod="openshift-image-registry/node-ca-jdrfw" Jul 10 00:17:49.560805 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560255 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/733f8b73-939c-414d-9d90-7791508e5b84-hosts-file\") pod \"node-resolver-gzg2p\" (UID: \"733f8b73-939c-414d-9d90-7791508e5b84\") " pod="openshift-dns/node-resolver-gzg2p" Jul 10 00:17:49.560805 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560279 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1c3645d-2300-4b32-b0ef-e373064e5aad-original-pull-secret\") pod \"global-pull-secret-syncer-zqzc6\" (UID: \"b1c3645d-2300-4b32-b0ef-e373064e5aad\") " pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:17:49.561583 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560295 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/57273bce-c5a5-4b08-86b5-8f69c2eed2de-device-dir\") pod \"aws-ebs-csi-driver-node-2hkpp\" (UID: \"57273bce-c5a5-4b08-86b5-8f69c2eed2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" Jul 10 00:17:49.561583 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560310 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb7xv\" (UniqueName: \"kubernetes.io/projected/9de3defe-9135-40a5-994e-2a2b3c888c6d-kube-api-access-qb7xv\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.561583 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560330 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.561583 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560355 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-os-release\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.561583 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560378 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d642e8f4-ca88-46f0-ba74-e17a26ec2f7f-cnibin\") pod \"multus-additional-cni-plugins-8hp4r\" (UID: \"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f\") " pod="openshift-multus/multus-additional-cni-plugins-8hp4r" Jul 10 00:17:49.561583 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560399 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-var-lib-openvswitch\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.561583 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560424 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-host-run-ovn-kubernetes\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.561583 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560448 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-host-cni-netd\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.561583 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560452 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/733f8b73-939c-414d-9d90-7791508e5b84-hosts-file\") pod \"node-resolver-gzg2p\" (UID: \"733f8b73-939c-414d-9d90-7791508e5b84\") " pod="openshift-dns/node-resolver-gzg2p" Jul 10 00:17:49.561583 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560470 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/613ac839-60fd-4f20-a127-a072d8a574bb-serviceca\") pod \"node-ca-jdrfw\" (UID: \"613ac839-60fd-4f20-a127-a072d8a574bb\") " pod="openshift-image-registry/node-ca-jdrfw" Jul 10 00:17:49.561583 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560495 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/57273bce-c5a5-4b08-86b5-8f69c2eed2de-sys-fs\") pod \"aws-ebs-csi-driver-node-2hkpp\" (UID: \"57273bce-c5a5-4b08-86b5-8f69c2eed2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" Jul 10 00:17:49.561583 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560516 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9de3defe-9135-40a5-994e-2a2b3c888c6d-tmp\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.561583 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560538 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-log-socket\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.561583 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560560 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-host-cni-bin\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.561583 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560577 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/57273bce-c5a5-4b08-86b5-8f69c2eed2de-sys-fs\") pod \"aws-ebs-csi-driver-node-2hkpp\" (UID: \"57273bce-c5a5-4b08-86b5-8f69c2eed2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" Jul 10 00:17:49.561583 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560584 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5hb9\" (UniqueName: \"kubernetes.io/projected/f158539b-d568-492a-a0b6-0266deb4109e-kube-api-access-t5hb9\") pod \"iptables-alerter-n9prh\" (UID: \"f158539b-d568-492a-a0b6-0266deb4109e\") " pod="openshift-network-operator/iptables-alerter-n9prh" Jul 10 00:17:49.562318 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560610 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f59c94d4-93a7-4602-bf06-b2e0a4b2e312-agent-certs\") pod \"konnectivity-agent-k2hs8\" (UID: \"f59c94d4-93a7-4602-bf06-b2e0a4b2e312\") " pod="kube-system/konnectivity-agent-k2hs8" Jul 10 00:17:49.562318 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560633 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/57273bce-c5a5-4b08-86b5-8f69c2eed2de-device-dir\") pod \"aws-ebs-csi-driver-node-2hkpp\" (UID: \"57273bce-c5a5-4b08-86b5-8f69c2eed2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" Jul 10 00:17:49.562318 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560634 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2m9w\" (UniqueName: \"kubernetes.io/projected/376630cb-0d38-46f3-a311-7cf9af6f2d10-kube-api-access-f2m9w\") pod \"network-metrics-daemon-ms9mf\" (UID: \"376630cb-0d38-46f3-a311-7cf9af6f2d10\") " pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:17:49.562318 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:49.560402 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:17:49.562318 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560695 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/57273bce-c5a5-4b08-86b5-8f69c2eed2de-etc-selinux\") pod \"aws-ebs-csi-driver-node-2hkpp\" (UID: \"57273bce-c5a5-4b08-86b5-8f69c2eed2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" Jul 10 00:17:49.562318 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:49.560746 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1c3645d-2300-4b32-b0ef-e373064e5aad-original-pull-secret podName:b1c3645d-2300-4b32-b0ef-e373064e5aad nodeName:}" failed. No retries permitted until 2026-07-10 00:17:50.060707002 +0000 UTC m=+2.993791305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1c3645d-2300-4b32-b0ef-e373064e5aad-original-pull-secret") pod "global-pull-secret-syncer-zqzc6" (UID: "b1c3645d-2300-4b32-b0ef-e373064e5aad") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:17:49.562318 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560766 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-host-run-netns\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.562318 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560791 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-ovnkube-script-lib\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.562318 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560768 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/57273bce-c5a5-4b08-86b5-8f69c2eed2de-etc-selinux\") pod \"aws-ebs-csi-driver-node-2hkpp\" (UID: \"57273bce-c5a5-4b08-86b5-8f69c2eed2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" Jul 10 00:17:49.562318 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.560827 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv5vw\" (UniqueName: \"kubernetes.io/projected/613ac839-60fd-4f20-a127-a072d8a574bb-kube-api-access-mv5vw\") pod \"node-ca-jdrfw\" (UID: \"613ac839-60fd-4f20-a127-a072d8a574bb\") " pod="openshift-image-registry/node-ca-jdrfw" Jul 10 00:17:49.573021 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.572994 2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 00:17:49.574310 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:49.573551 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:17:49.574310 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:49.573570 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:17:49.574310 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:49.573582 2559 projected.go:196] Error preparing data for projected volume kube-api-access-kdbw5 for pod openshift-network-diagnostics/network-check-target-wrzt4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:17:49.574310 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:49.573648 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c396c5d-d578-4bc2-b02e-0d1f129d8247-kube-api-access-kdbw5 podName:4c396c5d-d578-4bc2-b02e-0d1f129d8247 nodeName:}" failed. No retries permitted until 2026-07-10 00:17:50.073631451 +0000 UTC m=+3.006715752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kdbw5" (UniqueName: "kubernetes.io/projected/4c396c5d-d578-4bc2-b02e-0d1f129d8247-kube-api-access-kdbw5") pod "network-check-target-wrzt4" (UID: "4c396c5d-d578-4bc2-b02e-0d1f129d8247") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:17:49.576863 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.576833 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dqhp\" (UniqueName: \"kubernetes.io/projected/733f8b73-939c-414d-9d90-7791508e5b84-kube-api-access-8dqhp\") pod \"node-resolver-gzg2p\" (UID: \"733f8b73-939c-414d-9d90-7791508e5b84\") " pod="openshift-dns/node-resolver-gzg2p" Jul 10 00:17:49.576863 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.576850 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bprg\" (UniqueName: \"kubernetes.io/projected/57273bce-c5a5-4b08-86b5-8f69c2eed2de-kube-api-access-2bprg\") pod \"aws-ebs-csi-driver-node-2hkpp\" (UID: \"57273bce-c5a5-4b08-86b5-8f69c2eed2de\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" Jul 10 00:17:49.649411 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.649347 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-30.ec2.internal" event={"ID":"fea6ac5a21d75add74f7c2411fe7019d","Type":"ContainerStarted","Data":"1887fa084889092305120fceab3cab960ef80a06ccf5dee6dedd8c83d83c7db7"} Jul 10 00:17:49.650849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.650821 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-30.ec2.internal" event={"ID":"71ce3f1111f97924efae8ea5aa51d9e5","Type":"ContainerStarted","Data":"b6e3cbe9327b528f87493a0bf8e454c4161bd01696e007563fc202d2bc66fc3c"} Jul 10 00:17:49.661451 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.661421 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9de3defe-9135-40a5-994e-2a2b3c888c6d-tmp\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.661557 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.661461 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-log-socket\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.661557 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.661488 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-host-cni-bin\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.661557 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.661513 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5hb9\" (UniqueName: \"kubernetes.io/projected/f158539b-d568-492a-a0b6-0266deb4109e-kube-api-access-t5hb9\") pod \"iptables-alerter-n9prh\" (UID: \"f158539b-d568-492a-a0b6-0266deb4109e\") " pod="openshift-network-operator/iptables-alerter-n9prh" Jul 10 00:17:49.661557 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.661538 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f59c94d4-93a7-4602-bf06-b2e0a4b2e312-agent-certs\") pod \"konnectivity-agent-k2hs8\" (UID: \"f59c94d4-93a7-4602-bf06-b2e0a4b2e312\") " pod="kube-system/konnectivity-agent-k2hs8" Jul 10 00:17:49.661747 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.661562 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2m9w\" (UniqueName: \"kubernetes.io/projected/376630cb-0d38-46f3-a311-7cf9af6f2d10-kube-api-access-f2m9w\") pod \"network-metrics-daemon-ms9mf\" (UID: \"376630cb-0d38-46f3-a311-7cf9af6f2d10\") " pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:17:49.661747 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.661585 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-host-run-netns\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.661747 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.661608 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-ovnkube-script-lib\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.661747 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.661631 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv5vw\" (UniqueName: \"kubernetes.io/projected/613ac839-60fd-4f20-a127-a072d8a574bb-kube-api-access-mv5vw\") pod \"node-ca-jdrfw\" (UID: \"613ac839-60fd-4f20-a127-a072d8a574bb\") " pod="openshift-image-registry/node-ca-jdrfw" Jul 10 00:17:49.661747 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.661654 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-etc-kubernetes\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.661747 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.661679 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-systemd-units\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.661747 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.661705 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-ovn-node-metrics-cert\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.661747 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.661728 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-system-cni-dir\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.662136 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.661751 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-multus-socket-dir-parent\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.662136 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.661751 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-host-cni-bin\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.662136 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.661774 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-host-var-lib-cni-multus\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.662136 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.661804 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nstk\" (UniqueName: \"kubernetes.io/projected/de6e5b48-34f6-43bb-8b9d-d1e651860753-kube-api-access-5nstk\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.662136 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.661830 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d642e8f4-ca88-46f0-ba74-e17a26ec2f7f-system-cni-dir\") pod \"multus-additional-cni-plugins-8hp4r\" (UID: \"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f\") " pod="openshift-multus/multus-additional-cni-plugins-8hp4r" Jul 10 00:17:49.662136 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.661854 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-etc-sysconfig\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.662136 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.661867 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-log-socket\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.662136 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.661877 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-host-kubelet\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.662136 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.661913 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-ovnkube-config\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.662136 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.661940 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-system-cni-dir\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.662136 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.661951 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-host-var-lib-kubelet\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.662136 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.661974 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d642e8f4-ca88-46f0-ba74-e17a26ec2f7f-cni-binary-copy\") pod \"multus-additional-cni-plugins-8hp4r\" (UID: \"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f\") " pod="openshift-multus/multus-additional-cni-plugins-8hp4r" Jul 10 00:17:49.662136 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.661984 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-host-run-netns\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.662136 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.661997 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d642e8f4-ca88-46f0-ba74-e17a26ec2f7f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8hp4r\" (UID: \"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f\") " pod="openshift-multus/multus-additional-cni-plugins-8hp4r" Jul 10 00:17:49.662136 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662022 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9de3defe-9135-40a5-994e-2a2b3c888c6d-etc-tuned\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.662136 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662044 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-env-overrides\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.662136 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662068 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-host-run-k8s-cni-cncf-io\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.662913 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662090 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-hostroot\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.662913 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662129 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-lib-modules\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.662913 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662153 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-run-systemd\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.662913 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662177 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f158539b-d568-492a-a0b6-0266deb4109e-iptables-alerter-script\") pod \"iptables-alerter-n9prh\" (UID: \"f158539b-d568-492a-a0b6-0266deb4109e\") " pod="openshift-network-operator/iptables-alerter-n9prh" Jul 10 00:17:49.662913 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662203 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d642e8f4-ca88-46f0-ba74-e17a26ec2f7f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8hp4r\" (UID: \"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f\") " pod="openshift-multus/multus-additional-cni-plugins-8hp4r" Jul 10 00:17:49.662913 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662231 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-etc-sysctl-conf\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.662913 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662256 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-run\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.662913 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662280 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d642e8f4-ca88-46f0-ba74-e17a26ec2f7f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8hp4r\" (UID: \"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f\") " pod="openshift-multus/multus-additional-cni-plugins-8hp4r" Jul 10 00:17:49.662913 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662304 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs\") pod \"network-metrics-daemon-ms9mf\" (UID: \"376630cb-0d38-46f3-a311-7cf9af6f2d10\") " pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:17:49.662913 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662343 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-etc-systemd\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.662913 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662369 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-run-ovn\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.662913 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662392 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxmgw\" (UniqueName: \"kubernetes.io/projected/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-kube-api-access-zxmgw\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.662913 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662416 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-host-var-lib-cni-bin\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.662913 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662440 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-multus-conf-dir\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.662913 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662473 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-host-slash\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.662913 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662498 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de6e5b48-34f6-43bb-8b9d-d1e651860753-multus-daemon-config\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.662913 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662521 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d642e8f4-ca88-46f0-ba74-e17a26ec2f7f-os-release\") pod \"multus-additional-cni-plugins-8hp4r\" (UID: \"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f\") " pod="openshift-multus/multus-additional-cni-plugins-8hp4r" Jul 10 00:17:49.663814 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662543 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-node-log\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.663814 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662567 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f158539b-d568-492a-a0b6-0266deb4109e-host-slash\") pod \"iptables-alerter-n9prh\" (UID: \"f158539b-d568-492a-a0b6-0266deb4109e\") " pod="openshift-network-operator/iptables-alerter-n9prh" Jul 10 00:17:49.663814 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662593 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-multus-cni-dir\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.663814 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662618 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de6e5b48-34f6-43bb-8b9d-d1e651860753-cni-binary-copy\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.663814 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662625 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-ovnkube-script-lib\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.663814 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662643 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-host-run-netns\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.663814 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662671 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-host-run-multus-certs\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.663814 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662726 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-host-run-multus-certs\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.663814 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662726 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-etc-kubernetes\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.663814 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662750 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-var-lib-kubelet\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.663814 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662781 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-run-openvswitch\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.663814 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662444 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-systemd-units\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.663814 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662807 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-cnibin\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.663814 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662834 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-etc-kubernetes\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.663814 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662859 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f59c94d4-93a7-4602-bf06-b2e0a4b2e312-konnectivity-ca\") pod \"konnectivity-agent-k2hs8\" (UID: \"f59c94d4-93a7-4602-bf06-b2e0a4b2e312\") " pod="kube-system/konnectivity-agent-k2hs8" Jul 10 00:17:49.663814 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662869 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-etc-sysctl-conf\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.663814 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662885 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-etc-modprobe-d\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.663814 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662912 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-etc-sysctl-d\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.664631 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662937 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-host\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.664631 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662966 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slz7c\" (UniqueName: \"kubernetes.io/projected/d642e8f4-ca88-46f0-ba74-e17a26ec2f7f-kube-api-access-slz7c\") pod \"multus-additional-cni-plugins-8hp4r\" (UID: \"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f\") " pod="openshift-multus/multus-additional-cni-plugins-8hp4r" Jul 10 00:17:49.664631 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662993 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-sys\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.664631 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.663030 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-etc-openvswitch\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.664631 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.663053 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/613ac839-60fd-4f20-a127-a072d8a574bb-host\") pod \"node-ca-jdrfw\" (UID: \"613ac839-60fd-4f20-a127-a072d8a574bb\") " pod="openshift-image-registry/node-ca-jdrfw" Jul 10 00:17:49.664631 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.663095 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb7xv\" (UniqueName: \"kubernetes.io/projected/9de3defe-9135-40a5-994e-2a2b3c888c6d-kube-api-access-qb7xv\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.664631 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.663137 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.664631 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.663163 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-os-release\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.664631 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.663190 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d642e8f4-ca88-46f0-ba74-e17a26ec2f7f-cnibin\") pod \"multus-additional-cni-plugins-8hp4r\" (UID: \"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f\") " pod="openshift-multus/multus-additional-cni-plugins-8hp4r" Jul 10 00:17:49.664631 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.663216 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-var-lib-openvswitch\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.664631 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.663240 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-host-run-ovn-kubernetes\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.664631 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.663264 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-host-cni-netd\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.664631 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.663290 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/613ac839-60fd-4f20-a127-a072d8a574bb-serviceca\") pod \"node-ca-jdrfw\" (UID: \"613ac839-60fd-4f20-a127-a072d8a574bb\") " pod="openshift-image-registry/node-ca-jdrfw" Jul 10 00:17:49.664631 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.663389 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d642e8f4-ca88-46f0-ba74-e17a26ec2f7f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8hp4r\" (UID: \"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f\") " pod="openshift-multus/multus-additional-cni-plugins-8hp4r" Jul 10 00:17:49.664631 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.663450 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/de6e5b48-34f6-43bb-8b9d-d1e651860753-multus-daemon-config\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.664631 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:49.663487 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:17:49.664631 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.663531 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d642e8f4-ca88-46f0-ba74-e17a26ec2f7f-os-release\") pod \"multus-additional-cni-plugins-8hp4r\" (UID: \"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f\") " pod="openshift-multus/multus-additional-cni-plugins-8hp4r" Jul 10 00:17:49.665477 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:49.663548 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs podName:376630cb-0d38-46f3-a311-7cf9af6f2d10 nodeName:}" failed. No retries permitted until 2026-07-10 00:17:50.163532014 +0000 UTC m=+3.096616315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs") pod "network-metrics-daemon-ms9mf" (UID: "376630cb-0d38-46f3-a311-7cf9af6f2d10") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:17:49.665477 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.663586 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-node-log\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.665477 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.663612 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-etc-systemd\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.665477 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.663631 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f158539b-d568-492a-a0b6-0266deb4109e-host-slash\") pod \"iptables-alerter-n9prh\" (UID: \"f158539b-d568-492a-a0b6-0266deb4109e\") " pod="openshift-network-operator/iptables-alerter-n9prh" Jul 10 00:17:49.665477 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.663663 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-run-ovn\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.665477 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.663704 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-multus-cni-dir\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.665477 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.663765 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/613ac839-60fd-4f20-a127-a072d8a574bb-serviceca\") pod \"node-ca-jdrfw\" (UID: \"613ac839-60fd-4f20-a127-a072d8a574bb\") " pod="openshift-image-registry/node-ca-jdrfw" Jul 10 00:17:49.665477 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.663839 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-multus-socket-dir-parent\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.665477 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.663889 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-host-var-lib-cni-bin\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.665477 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.663915 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-host-var-lib-cni-multus\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.665477 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.663939 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-multus-conf-dir\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.665477 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.664086 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d642e8f4-ca88-46f0-ba74-e17a26ec2f7f-system-cni-dir\") pod \"multus-additional-cni-plugins-8hp4r\" (UID: \"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f\") " pod="openshift-multus/multus-additional-cni-plugins-8hp4r" Jul 10 00:17:49.665477 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.664176 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-host-run-netns\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.665477 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.664178 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-etc-sysconfig\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.665477 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.664220 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-host-kubelet\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.665477 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.664231 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-etc-kubernetes\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.665477 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.664324 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-etc-modprobe-d\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.666312 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.664404 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-etc-sysctl-d\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.666312 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.662913 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-host-slash\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.666312 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.664457 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-host\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.666312 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.664659 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-hostroot\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.666312 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.664689 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-sys\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.666312 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.664741 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-etc-openvswitch\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.666312 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.664762 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/de6e5b48-34f6-43bb-8b9d-d1e651860753-cni-binary-copy\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.666312 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.664786 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/613ac839-60fd-4f20-a127-a072d8a574bb-host\") pod \"node-ca-jdrfw\" (UID: \"613ac839-60fd-4f20-a127-a072d8a574bb\") " pod="openshift-image-registry/node-ca-jdrfw" Jul 10 00:17:49.666312 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.664809 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f59c94d4-93a7-4602-bf06-b2e0a4b2e312-konnectivity-ca\") pod \"konnectivity-agent-k2hs8\" (UID: \"f59c94d4-93a7-4602-bf06-b2e0a4b2e312\") " pod="kube-system/konnectivity-agent-k2hs8" Jul 10 00:17:49.666312 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.664911 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d642e8f4-ca88-46f0-ba74-e17a26ec2f7f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8hp4r\" (UID: \"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f\") " pod="openshift-multus/multus-additional-cni-plugins-8hp4r" Jul 10 00:17:49.666312 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.664942 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-lib-modules\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.666312 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.664962 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-host-var-lib-kubelet\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.666312 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.664988 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.666312 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.664992 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-run-systemd\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.666312 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.665067 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-os-release\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.666312 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.665094 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-ovnkube-config\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.666312 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.665104 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d642e8f4-ca88-46f0-ba74-e17a26ec2f7f-cnibin\") pod \"multus-additional-cni-plugins-8hp4r\" (UID: \"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f\") " pod="openshift-multus/multus-additional-cni-plugins-8hp4r" Jul 10 00:17:49.666312 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.665164 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-var-lib-openvswitch\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.667220 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.665204 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-host-run-ovn-kubernetes\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.667220 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.665242 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-host-cni-netd\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.667220 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.665453 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d642e8f4-ca88-46f0-ba74-e17a26ec2f7f-cni-binary-copy\") pod \"multus-additional-cni-plugins-8hp4r\" (UID: \"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f\") " pod="openshift-multus/multus-additional-cni-plugins-8hp4r" Jul 10 00:17:49.667220 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.665514 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-run-openvswitch\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.667220 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.665612 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-env-overrides\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.667220 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.665678 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-var-lib-kubelet\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.667220 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.665688 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-cnibin\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.667220 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.665728 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/de6e5b48-34f6-43bb-8b9d-d1e651860753-host-run-k8s-cni-cncf-io\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.667220 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.665748 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9de3defe-9135-40a5-994e-2a2b3c888c6d-run\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.667220 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.665939 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d642e8f4-ca88-46f0-ba74-e17a26ec2f7f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8hp4r\" (UID: \"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f\") " pod="openshift-multus/multus-additional-cni-plugins-8hp4r" Jul 10 00:17:49.667220 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.666172 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f158539b-d568-492a-a0b6-0266deb4109e-iptables-alerter-script\") pod \"iptables-alerter-n9prh\" (UID: \"f158539b-d568-492a-a0b6-0266deb4109e\") " pod="openshift-network-operator/iptables-alerter-n9prh" Jul 10 00:17:49.669978 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.669949 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f59c94d4-93a7-4602-bf06-b2e0a4b2e312-agent-certs\") pod \"konnectivity-agent-k2hs8\" (UID: \"f59c94d4-93a7-4602-bf06-b2e0a4b2e312\") " pod="kube-system/konnectivity-agent-k2hs8" Jul 10 00:17:49.670847 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.670587 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9de3defe-9135-40a5-994e-2a2b3c888c6d-etc-tuned\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.670847 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.670700 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5hb9\" (UniqueName: \"kubernetes.io/projected/f158539b-d568-492a-a0b6-0266deb4109e-kube-api-access-t5hb9\") pod \"iptables-alerter-n9prh\" (UID: \"f158539b-d568-492a-a0b6-0266deb4109e\") " pod="openshift-network-operator/iptables-alerter-n9prh" Jul 10 00:17:49.670847 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.670806 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-ovn-node-metrics-cert\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.671039 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.671004 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2m9w\" (UniqueName: \"kubernetes.io/projected/376630cb-0d38-46f3-a311-7cf9af6f2d10-kube-api-access-f2m9w\") pod \"network-metrics-daemon-ms9mf\" (UID: \"376630cb-0d38-46f3-a311-7cf9af6f2d10\") " pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:17:49.671653 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.671610 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv5vw\" (UniqueName: \"kubernetes.io/projected/613ac839-60fd-4f20-a127-a072d8a574bb-kube-api-access-mv5vw\") pod \"node-ca-jdrfw\" (UID: \"613ac839-60fd-4f20-a127-a072d8a574bb\") " pod="openshift-image-registry/node-ca-jdrfw" Jul 10 00:17:49.671751 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.671649 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9de3defe-9135-40a5-994e-2a2b3c888c6d-tmp\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.673333 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.672640 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nstk\" (UniqueName: \"kubernetes.io/projected/de6e5b48-34f6-43bb-8b9d-d1e651860753-kube-api-access-5nstk\") pod \"multus-hf6sh\" (UID: \"de6e5b48-34f6-43bb-8b9d-d1e651860753\") " pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.675606 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.675555 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxmgw\" (UniqueName: \"kubernetes.io/projected/20198eb1-6b3e-4e30-9adf-d4f13df4ee3b-kube-api-access-zxmgw\") pod \"ovnkube-node-p7zrr\" (UID: \"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b\") " pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.675732 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.675699 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slz7c\" (UniqueName: \"kubernetes.io/projected/d642e8f4-ca88-46f0-ba74-e17a26ec2f7f-kube-api-access-slz7c\") pod \"multus-additional-cni-plugins-8hp4r\" (UID: \"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f\") " pod="openshift-multus/multus-additional-cni-plugins-8hp4r" Jul 10 00:17:49.675806 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.675727 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb7xv\" (UniqueName: \"kubernetes.io/projected/9de3defe-9135-40a5-994e-2a2b3c888c6d-kube-api-access-qb7xv\") pod \"tuned-nsx5f\" (UID: \"9de3defe-9135-40a5-994e-2a2b3c888c6d\") " pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.754480 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.754447 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gzg2p" Jul 10 00:17:49.761304 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.761149 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" Jul 10 00:17:49.762205 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:49.762182 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733f8b73_939c_414d_9d90_7791508e5b84.slice/crio-73b7192be7416603ed2a6b9e6030a4c0811ac864972d751d6e035ce8f4b85f7b WatchSource:0}: Error finding container 73b7192be7416603ed2a6b9e6030a4c0811ac864972d751d6e035ce8f4b85f7b: Status 404 returned error can't find the container with id 73b7192be7416603ed2a6b9e6030a4c0811ac864972d751d6e035ce8f4b85f7b Jul 10 00:17:49.768815 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:49.768793 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57273bce_c5a5_4b08_86b5_8f69c2eed2de.slice/crio-a2f5707ee6fc5cdeec4477e59c1d52a029764a2598e97af33faf3317fdf62ffb WatchSource:0}: Error finding container a2f5707ee6fc5cdeec4477e59c1d52a029764a2598e97af33faf3317fdf62ffb: Status 404 returned error can't find the container with id a2f5707ee6fc5cdeec4477e59c1d52a029764a2598e97af33faf3317fdf62ffb Jul 10 00:17:49.773762 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.773727 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8hp4r" Jul 10 00:17:49.778527 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.778511 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hf6sh" Jul 10 00:17:49.781165 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:49.781042 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd642e8f4_ca88_46f0_ba74_e17a26ec2f7f.slice/crio-f1e7c6fb073218b0648256faabc04d2b3168b33fb9cdc58e825756a78c327842 WatchSource:0}: Error finding container f1e7c6fb073218b0648256faabc04d2b3168b33fb9cdc58e825756a78c327842: Status 404 returned error can't find the container with id f1e7c6fb073218b0648256faabc04d2b3168b33fb9cdc58e825756a78c327842 Jul 10 00:17:49.784932 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.784918 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jdrfw" Jul 10 00:17:49.786255 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:49.786236 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde6e5b48_34f6_43bb_8b9d_d1e651860753.slice/crio-429c4aa12a7bac704a1f905bd797d2649da4e91fcf8bbf990e0a04cbefc7701f WatchSource:0}: Error finding container 429c4aa12a7bac704a1f905bd797d2649da4e91fcf8bbf990e0a04cbefc7701f: Status 404 returned error can't find the container with id 429c4aa12a7bac704a1f905bd797d2649da4e91fcf8bbf990e0a04cbefc7701f Jul 10 00:17:49.795716 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:49.795668 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613ac839_60fd_4f20_a127_a072d8a574bb.slice/crio-8a4fa423b629f71ac658bf875407992c7876b5f7c9ca2ee737f5a2b0a322df26 WatchSource:0}: Error finding container 8a4fa423b629f71ac658bf875407992c7876b5f7c9ca2ee737f5a2b0a322df26: Status 404 returned error can't find the container with id 8a4fa423b629f71ac658bf875407992c7876b5f7c9ca2ee737f5a2b0a322df26 Jul 10 00:17:49.803607 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.803578 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-n9prh" Jul 10 00:17:49.811840 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.811817 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k2hs8" Jul 10 00:17:49.813208 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:49.813185 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf158539b_d568_492a_a0b6_0266deb4109e.slice/crio-b82f02461a05d913a445e845687535dffa665f8a7ef047b2a0eb8a5d024b7b6e WatchSource:0}: Error finding container b82f02461a05d913a445e845687535dffa665f8a7ef047b2a0eb8a5d024b7b6e: Status 404 returned error can't find the container with id b82f02461a05d913a445e845687535dffa665f8a7ef047b2a0eb8a5d024b7b6e Jul 10 00:17:49.820330 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.820306 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" Jul 10 00:17:49.820668 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:49.820640 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59c94d4_93a7_4602_bf06_b2e0a4b2e312.slice/crio-dc69e5b2f4a7ad922957f31d3c5e8c434d26336b3ec74547f439f4c55e74079d WatchSource:0}: Error finding container dc69e5b2f4a7ad922957f31d3c5e8c434d26336b3ec74547f439f4c55e74079d: Status 404 returned error can't find the container with id dc69e5b2f4a7ad922957f31d3c5e8c434d26336b3ec74547f439f4c55e74079d Jul 10 00:17:49.827045 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.827023 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:17:49.829878 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:49.829856 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de3defe_9135_40a5_994e_2a2b3c888c6d.slice/crio-471ac1fe99548c0ec389ff6b21751c5f1cd71e91cca66284a131134c1b4f9ac1 WatchSource:0}: Error finding container 471ac1fe99548c0ec389ff6b21751c5f1cd71e91cca66284a131134c1b4f9ac1: Status 404 returned error can't find the container with id 471ac1fe99548c0ec389ff6b21751c5f1cd71e91cca66284a131134c1b4f9ac1 Jul 10 00:17:49.838139 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:17:49.838100 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20198eb1_6b3e_4e30_9adf_d4f13df4ee3b.slice/crio-afb0a1a52189b0d3d7456fd52486554ddb1349bc87babeec1896c9ab8024aba1 WatchSource:0}: Error finding container afb0a1a52189b0d3d7456fd52486554ddb1349bc87babeec1896c9ab8024aba1: Status 404 returned error can't find the container with id afb0a1a52189b0d3d7456fd52486554ddb1349bc87babeec1896c9ab8024aba1 Jul 10 00:17:49.852527 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.852507 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:17:49.909678 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:49.909653 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:17:50.065845 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:50.065742 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1c3645d-2300-4b32-b0ef-e373064e5aad-original-pull-secret\") pod \"global-pull-secret-syncer-zqzc6\" (UID: \"b1c3645d-2300-4b32-b0ef-e373064e5aad\") " pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:17:50.065986 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:50.065882 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:17:50.065986 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:50.065950 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1c3645d-2300-4b32-b0ef-e373064e5aad-original-pull-secret podName:b1c3645d-2300-4b32-b0ef-e373064e5aad nodeName:}" failed. No retries permitted until 2026-07-10 00:17:51.065931696 +0000 UTC m=+3.999015983 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1c3645d-2300-4b32-b0ef-e373064e5aad-original-pull-secret") pod "global-pull-secret-syncer-zqzc6" (UID: "b1c3645d-2300-4b32-b0ef-e373064e5aad") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:17:50.166192 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:50.166159 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs\") pod \"network-metrics-daemon-ms9mf\" (UID: \"376630cb-0d38-46f3-a311-7cf9af6f2d10\") " pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:17:50.166363 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:50.166199 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdbw5\" (UniqueName: \"kubernetes.io/projected/4c396c5d-d578-4bc2-b02e-0d1f129d8247-kube-api-access-kdbw5\") pod \"network-check-target-wrzt4\" (UID: \"4c396c5d-d578-4bc2-b02e-0d1f129d8247\") " pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:17:50.166363 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:50.166298 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:17:50.166363 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:50.166328 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:17:50.166363 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:50.166350 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:17:50.166363 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:50.166363 2559 projected.go:196] Error preparing data for projected volume kube-api-access-kdbw5 for pod openshift-network-diagnostics/network-check-target-wrzt4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:17:50.166645 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:50.166375 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs podName:376630cb-0d38-46f3-a311-7cf9af6f2d10 nodeName:}" failed. No retries permitted until 2026-07-10 00:17:51.16635322 +0000 UTC m=+4.099437511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs") pod "network-metrics-daemon-ms9mf" (UID: "376630cb-0d38-46f3-a311-7cf9af6f2d10") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:17:50.166645 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:50.166442 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c396c5d-d578-4bc2-b02e-0d1f129d8247-kube-api-access-kdbw5 podName:4c396c5d-d578-4bc2-b02e-0d1f129d8247 nodeName:}" failed. No retries permitted until 2026-07-10 00:17:51.166396589 +0000 UTC m=+4.099480898 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kdbw5" (UniqueName: "kubernetes.io/projected/4c396c5d-d578-4bc2-b02e-0d1f129d8247-kube-api-access-kdbw5") pod "network-check-target-wrzt4" (UID: "4c396c5d-d578-4bc2-b02e-0d1f129d8247") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:17:50.487944 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:50.487859 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 00:12:49 +0000 UTC" deadline="2028-03-17 00:36:46.822296092 +0000 UTC" Jul 10 00:17:50.487944 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:50.487893 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14784h18m56.334408229s" Jul 10 00:17:50.644780 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:50.644575 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:17:50.644780 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:50.644731 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9mf" podUID="376630cb-0d38-46f3-a311-7cf9af6f2d10" Jul 10 00:17:50.654202 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:50.654167 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n9prh" event={"ID":"f158539b-d568-492a-a0b6-0266deb4109e","Type":"ContainerStarted","Data":"b82f02461a05d913a445e845687535dffa665f8a7ef047b2a0eb8a5d024b7b6e"} Jul 10 00:17:50.655919 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:50.655892 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gzg2p" event={"ID":"733f8b73-939c-414d-9d90-7791508e5b84","Type":"ContainerStarted","Data":"73b7192be7416603ed2a6b9e6030a4c0811ac864972d751d6e035ce8f4b85f7b"} Jul 10 00:17:50.658139 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:50.658103 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" event={"ID":"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b","Type":"ContainerStarted","Data":"afb0a1a52189b0d3d7456fd52486554ddb1349bc87babeec1896c9ab8024aba1"} Jul 10 00:17:50.659961 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:50.659936 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" event={"ID":"9de3defe-9135-40a5-994e-2a2b3c888c6d","Type":"ContainerStarted","Data":"471ac1fe99548c0ec389ff6b21751c5f1cd71e91cca66284a131134c1b4f9ac1"} Jul 10 00:17:50.661473 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:50.661451 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k2hs8" event={"ID":"f59c94d4-93a7-4602-bf06-b2e0a4b2e312","Type":"ContainerStarted","Data":"dc69e5b2f4a7ad922957f31d3c5e8c434d26336b3ec74547f439f4c55e74079d"} Jul 10 00:17:50.662850 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:50.662818 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jdrfw" event={"ID":"613ac839-60fd-4f20-a127-a072d8a574bb","Type":"ContainerStarted","Data":"8a4fa423b629f71ac658bf875407992c7876b5f7c9ca2ee737f5a2b0a322df26"} Jul 10 00:17:50.664328 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:50.664285 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hf6sh" event={"ID":"de6e5b48-34f6-43bb-8b9d-d1e651860753","Type":"ContainerStarted","Data":"429c4aa12a7bac704a1f905bd797d2649da4e91fcf8bbf990e0a04cbefc7701f"} Jul 10 00:17:50.665670 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:50.665645 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8hp4r" event={"ID":"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f","Type":"ContainerStarted","Data":"f1e7c6fb073218b0648256faabc04d2b3168b33fb9cdc58e825756a78c327842"} Jul 10 00:17:50.667419 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:50.667393 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" event={"ID":"57273bce-c5a5-4b08-86b5-8f69c2eed2de","Type":"ContainerStarted","Data":"a2f5707ee6fc5cdeec4477e59c1d52a029764a2598e97af33faf3317fdf62ffb"} Jul 10 00:17:51.073435 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:51.072767 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1c3645d-2300-4b32-b0ef-e373064e5aad-original-pull-secret\") pod \"global-pull-secret-syncer-zqzc6\" (UID: \"b1c3645d-2300-4b32-b0ef-e373064e5aad\") " pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:17:51.073435 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:51.072982 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:17:51.073435 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:51.073048 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1c3645d-2300-4b32-b0ef-e373064e5aad-original-pull-secret podName:b1c3645d-2300-4b32-b0ef-e373064e5aad nodeName:}" failed. No retries permitted until 2026-07-10 00:17:53.073029454 +0000 UTC m=+6.006113743 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1c3645d-2300-4b32-b0ef-e373064e5aad-original-pull-secret") pod "global-pull-secret-syncer-zqzc6" (UID: "b1c3645d-2300-4b32-b0ef-e373064e5aad") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:17:51.173984 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:51.173195 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs\") pod \"network-metrics-daemon-ms9mf\" (UID: \"376630cb-0d38-46f3-a311-7cf9af6f2d10\") " pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:17:51.173984 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:51.173250 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdbw5\" (UniqueName: \"kubernetes.io/projected/4c396c5d-d578-4bc2-b02e-0d1f129d8247-kube-api-access-kdbw5\") pod \"network-check-target-wrzt4\" (UID: \"4c396c5d-d578-4bc2-b02e-0d1f129d8247\") " pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:17:51.173984 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:51.173440 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:17:51.173984 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:51.173459 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:17:51.173984 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:51.173472 2559 projected.go:196] Error preparing data for projected volume kube-api-access-kdbw5 for pod openshift-network-diagnostics/network-check-target-wrzt4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:17:51.173984 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:51.173530 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c396c5d-d578-4bc2-b02e-0d1f129d8247-kube-api-access-kdbw5 podName:4c396c5d-d578-4bc2-b02e-0d1f129d8247 nodeName:}" failed. No retries permitted until 2026-07-10 00:17:53.173512675 +0000 UTC m=+6.106596964 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kdbw5" (UniqueName: "kubernetes.io/projected/4c396c5d-d578-4bc2-b02e-0d1f129d8247-kube-api-access-kdbw5") pod "network-check-target-wrzt4" (UID: "4c396c5d-d578-4bc2-b02e-0d1f129d8247") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:17:51.173984 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:51.173900 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:17:51.173984 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:51.173948 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs podName:376630cb-0d38-46f3-a311-7cf9af6f2d10 nodeName:}" failed. No retries permitted until 2026-07-10 00:17:53.173933636 +0000 UTC m=+6.107017946 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs") pod "network-metrics-daemon-ms9mf" (UID: "376630cb-0d38-46f3-a311-7cf9af6f2d10") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:17:51.489641 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:51.489505 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 00:12:49 +0000 UTC" deadline="2028-04-02 15:00:13.339463862 +0000 UTC" Jul 10 00:17:51.489641 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:51.489562 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15182h42m21.849905151s" Jul 10 00:17:51.644307 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:51.644223 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:17:51.644451 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:51.644355 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrzt4" podUID="4c396c5d-d578-4bc2-b02e-0d1f129d8247" Jul 10 00:17:51.644722 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:51.644561 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:17:51.644722 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:51.644683 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqzc6" podUID="b1c3645d-2300-4b32-b0ef-e373064e5aad" Jul 10 00:17:52.644457 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:52.644419 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:17:52.644908 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:52.644608 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9mf" podUID="376630cb-0d38-46f3-a311-7cf9af6f2d10" Jul 10 00:17:53.093913 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:53.093281 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1c3645d-2300-4b32-b0ef-e373064e5aad-original-pull-secret\") pod \"global-pull-secret-syncer-zqzc6\" (UID: \"b1c3645d-2300-4b32-b0ef-e373064e5aad\") " pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:17:53.093913 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:53.093480 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:17:53.093913 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:53.093544 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1c3645d-2300-4b32-b0ef-e373064e5aad-original-pull-secret podName:b1c3645d-2300-4b32-b0ef-e373064e5aad nodeName:}" failed. No retries permitted until 2026-07-10 00:17:57.093525474 +0000 UTC m=+10.026609783 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1c3645d-2300-4b32-b0ef-e373064e5aad-original-pull-secret") pod "global-pull-secret-syncer-zqzc6" (UID: "b1c3645d-2300-4b32-b0ef-e373064e5aad") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:17:53.194332 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:53.194294 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs\") pod \"network-metrics-daemon-ms9mf\" (UID: \"376630cb-0d38-46f3-a311-7cf9af6f2d10\") " pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:17:53.194514 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:53.194345 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdbw5\" (UniqueName: \"kubernetes.io/projected/4c396c5d-d578-4bc2-b02e-0d1f129d8247-kube-api-access-kdbw5\") pod \"network-check-target-wrzt4\" (UID: \"4c396c5d-d578-4bc2-b02e-0d1f129d8247\") " pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:17:53.194581 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:53.194517 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:17:53.194581 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:53.194538 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:17:53.194581 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:53.194552 2559 projected.go:196] Error preparing data for projected volume kube-api-access-kdbw5 for pod openshift-network-diagnostics/network-check-target-wrzt4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:17:53.194725 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:53.194608 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c396c5d-d578-4bc2-b02e-0d1f129d8247-kube-api-access-kdbw5 podName:4c396c5d-d578-4bc2-b02e-0d1f129d8247 nodeName:}" failed. No retries permitted until 2026-07-10 00:17:57.194589777 +0000 UTC m=+10.127674070 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kdbw5" (UniqueName: "kubernetes.io/projected/4c396c5d-d578-4bc2-b02e-0d1f129d8247-kube-api-access-kdbw5") pod "network-check-target-wrzt4" (UID: "4c396c5d-d578-4bc2-b02e-0d1f129d8247") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:17:53.195010 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:53.194994 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:17:53.195078 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:53.195039 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs podName:376630cb-0d38-46f3-a311-7cf9af6f2d10 nodeName:}" failed. No retries permitted until 2026-07-10 00:17:57.195027548 +0000 UTC m=+10.128111835 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs") pod "network-metrics-daemon-ms9mf" (UID: "376630cb-0d38-46f3-a311-7cf9af6f2d10") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:17:53.645666 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:53.644954 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:17:53.645666 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:53.645103 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqzc6" podUID="b1c3645d-2300-4b32-b0ef-e373064e5aad" Jul 10 00:17:53.645666 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:53.645527 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:17:53.645666 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:53.645618 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrzt4" podUID="4c396c5d-d578-4bc2-b02e-0d1f129d8247" Jul 10 00:17:54.644296 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:54.643744 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:17:54.644296 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:54.643897 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9mf" podUID="376630cb-0d38-46f3-a311-7cf9af6f2d10" Jul 10 00:17:55.644814 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:55.644771 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:17:55.645258 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:55.644927 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrzt4" podUID="4c396c5d-d578-4bc2-b02e-0d1f129d8247" Jul 10 00:17:55.645415 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:55.645397 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:17:55.645526 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:55.645503 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqzc6" podUID="b1c3645d-2300-4b32-b0ef-e373064e5aad" Jul 10 00:17:56.644088 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:56.644050 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:17:56.644292 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:56.644184 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9mf" podUID="376630cb-0d38-46f3-a311-7cf9af6f2d10" Jul 10 00:17:57.128608 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:57.128574 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1c3645d-2300-4b32-b0ef-e373064e5aad-original-pull-secret\") pod \"global-pull-secret-syncer-zqzc6\" (UID: \"b1c3645d-2300-4b32-b0ef-e373064e5aad\") " pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:17:57.129050 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:57.128689 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:17:57.129050 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:57.128748 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1c3645d-2300-4b32-b0ef-e373064e5aad-original-pull-secret podName:b1c3645d-2300-4b32-b0ef-e373064e5aad nodeName:}" failed. No retries permitted until 2026-07-10 00:18:05.128731131 +0000 UTC m=+18.061815417 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1c3645d-2300-4b32-b0ef-e373064e5aad-original-pull-secret") pod "global-pull-secret-syncer-zqzc6" (UID: "b1c3645d-2300-4b32-b0ef-e373064e5aad") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:17:57.229505 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:57.229449 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdbw5\" (UniqueName: \"kubernetes.io/projected/4c396c5d-d578-4bc2-b02e-0d1f129d8247-kube-api-access-kdbw5\") pod \"network-check-target-wrzt4\" (UID: \"4c396c5d-d578-4bc2-b02e-0d1f129d8247\") " pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:17:57.229693 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:57.229560 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs\") pod \"network-metrics-daemon-ms9mf\" (UID: \"376630cb-0d38-46f3-a311-7cf9af6f2d10\") " pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:17:57.229693 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:57.229632 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:17:57.229693 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:57.229656 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:17:57.229828 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:57.229723 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs podName:376630cb-0d38-46f3-a311-7cf9af6f2d10 nodeName:}" failed. No retries permitted until 2026-07-10 00:18:05.229700504 +0000 UTC m=+18.162784803 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs") pod "network-metrics-daemon-ms9mf" (UID: "376630cb-0d38-46f3-a311-7cf9af6f2d10") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:17:57.229828 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:57.229658 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:17:57.229828 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:57.229756 2559 projected.go:196] Error preparing data for projected volume kube-api-access-kdbw5 for pod openshift-network-diagnostics/network-check-target-wrzt4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:17:57.229828 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:57.229798 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c396c5d-d578-4bc2-b02e-0d1f129d8247-kube-api-access-kdbw5 podName:4c396c5d-d578-4bc2-b02e-0d1f129d8247 nodeName:}" failed. No retries permitted until 2026-07-10 00:18:05.229786896 +0000 UTC m=+18.162871200 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kdbw5" (UniqueName: "kubernetes.io/projected/4c396c5d-d578-4bc2-b02e-0d1f129d8247-kube-api-access-kdbw5") pod "network-check-target-wrzt4" (UID: "4c396c5d-d578-4bc2-b02e-0d1f129d8247") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:17:57.644779 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:57.644748 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:17:57.644946 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:57.644858 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqzc6" podUID="b1c3645d-2300-4b32-b0ef-e373064e5aad" Jul 10 00:17:57.644946 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:57.644920 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:17:57.645065 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:57.645034 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrzt4" podUID="4c396c5d-d578-4bc2-b02e-0d1f129d8247" Jul 10 00:17:58.643842 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:58.643746 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:17:58.644345 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:58.643901 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9mf" podUID="376630cb-0d38-46f3-a311-7cf9af6f2d10" Jul 10 00:17:59.644385 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:59.644339 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:17:59.644816 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:59.644472 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrzt4" podUID="4c396c5d-d578-4bc2-b02e-0d1f129d8247" Jul 10 00:17:59.644816 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:17:59.644558 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:17:59.644816 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:17:59.644669 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqzc6" podUID="b1c3645d-2300-4b32-b0ef-e373064e5aad" Jul 10 00:18:00.643974 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:00.643936 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:18:00.644165 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:00.644085 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9mf" podUID="376630cb-0d38-46f3-a311-7cf9af6f2d10" Jul 10 00:18:01.644298 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:01.644261 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:18:01.644706 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:01.644306 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:18:01.644706 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:01.644396 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrzt4" podUID="4c396c5d-d578-4bc2-b02e-0d1f129d8247" Jul 10 00:18:01.644706 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:01.644572 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqzc6" podUID="b1c3645d-2300-4b32-b0ef-e373064e5aad" Jul 10 00:18:02.643726 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:02.643696 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:18:02.643907 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:02.643825 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9mf" podUID="376630cb-0d38-46f3-a311-7cf9af6f2d10" Jul 10 00:18:03.643849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:03.643813 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:18:03.643849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:03.643832 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:18:03.644368 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:03.643935 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrzt4" podUID="4c396c5d-d578-4bc2-b02e-0d1f129d8247" Jul 10 00:18:03.644368 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:03.644088 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqzc6" podUID="b1c3645d-2300-4b32-b0ef-e373064e5aad" Jul 10 00:18:04.644487 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:04.644448 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:18:04.644914 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:04.644583 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9mf" podUID="376630cb-0d38-46f3-a311-7cf9af6f2d10" Jul 10 00:18:05.186276 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:05.186241 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1c3645d-2300-4b32-b0ef-e373064e5aad-original-pull-secret\") pod \"global-pull-secret-syncer-zqzc6\" (UID: \"b1c3645d-2300-4b32-b0ef-e373064e5aad\") " pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:18:05.186402 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:05.186378 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:18:05.186453 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:05.186438 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1c3645d-2300-4b32-b0ef-e373064e5aad-original-pull-secret podName:b1c3645d-2300-4b32-b0ef-e373064e5aad nodeName:}" failed. No retries permitted until 2026-07-10 00:18:21.18642132 +0000 UTC m=+34.119505617 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1c3645d-2300-4b32-b0ef-e373064e5aad-original-pull-secret") pod "global-pull-secret-syncer-zqzc6" (UID: "b1c3645d-2300-4b32-b0ef-e373064e5aad") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:18:05.286740 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:05.286712 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs\") pod \"network-metrics-daemon-ms9mf\" (UID: \"376630cb-0d38-46f3-a311-7cf9af6f2d10\") " pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:18:05.286839 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:05.286754 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdbw5\" (UniqueName: \"kubernetes.io/projected/4c396c5d-d578-4bc2-b02e-0d1f129d8247-kube-api-access-kdbw5\") pod \"network-check-target-wrzt4\" (UID: \"4c396c5d-d578-4bc2-b02e-0d1f129d8247\") " pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:18:05.286903 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:05.286880 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:18:05.286972 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:05.286962 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs podName:376630cb-0d38-46f3-a311-7cf9af6f2d10 nodeName:}" failed. No retries permitted until 2026-07-10 00:18:21.286938681 +0000 UTC m=+34.220022985 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs") pod "network-metrics-daemon-ms9mf" (UID: "376630cb-0d38-46f3-a311-7cf9af6f2d10") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:18:05.287012 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:05.286893 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:18:05.287041 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:05.287013 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:18:05.287041 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:05.287025 2559 projected.go:196] Error preparing data for projected volume kube-api-access-kdbw5 for pod openshift-network-diagnostics/network-check-target-wrzt4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:18:05.287111 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:05.287071 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c396c5d-d578-4bc2-b02e-0d1f129d8247-kube-api-access-kdbw5 podName:4c396c5d-d578-4bc2-b02e-0d1f129d8247 nodeName:}" failed. No retries permitted until 2026-07-10 00:18:21.28705664 +0000 UTC m=+34.220140926 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kdbw5" (UniqueName: "kubernetes.io/projected/4c396c5d-d578-4bc2-b02e-0d1f129d8247-kube-api-access-kdbw5") pod "network-check-target-wrzt4" (UID: "4c396c5d-d578-4bc2-b02e-0d1f129d8247") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:18:05.646933 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:05.646902 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:18:05.647346 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:05.647028 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqzc6" podUID="b1c3645d-2300-4b32-b0ef-e373064e5aad" Jul 10 00:18:05.647346 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:05.647081 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:18:05.647346 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:05.647213 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrzt4" podUID="4c396c5d-d578-4bc2-b02e-0d1f129d8247" Jul 10 00:18:05.698889 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:05.698842 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-30.ec2.internal" event={"ID":"71ce3f1111f97924efae8ea5aa51d9e5","Type":"ContainerStarted","Data":"daacdcb76b93e61f3ae6a99743a4a5dc6676ecacd24822ad0f8d1031b3030b86"} Jul 10 00:18:06.643894 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:06.643864 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:18:06.644049 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:06.643986 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9mf" podUID="376630cb-0d38-46f3-a311-7cf9af6f2d10" Jul 10 00:18:07.646539 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:07.646511 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:18:07.646539 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:07.646530 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:18:07.647038 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:07.646619 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrzt4" podUID="4c396c5d-d578-4bc2-b02e-0d1f129d8247" Jul 10 00:18:07.647038 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:07.646749 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqzc6" podUID="b1c3645d-2300-4b32-b0ef-e373064e5aad" Jul 10 00:18:08.644427 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:08.644395 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:18:08.644563 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:08.644506 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9mf" podUID="376630cb-0d38-46f3-a311-7cf9af6f2d10" Jul 10 00:18:09.646856 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:09.646705 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:18:09.647567 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:09.646707 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:18:09.647567 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:09.646966 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrzt4" podUID="4c396c5d-d578-4bc2-b02e-0d1f129d8247" Jul 10 00:18:09.647567 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:09.647014 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqzc6" podUID="b1c3645d-2300-4b32-b0ef-e373064e5aad" Jul 10 00:18:09.706649 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:09.706616 2559 generic.go:356] "Generic (PLEG): container finished" podID="d642e8f4-ca88-46f0-ba74-e17a26ec2f7f" containerID="d0ad2fd38f4a4101ae1f518ce9e371b721465fbb2652db8cd6a26cefa5a88bde" exitCode=0 Jul 10 00:18:09.706763 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:09.706679 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8hp4r" event={"ID":"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f","Type":"ContainerDied","Data":"d0ad2fd38f4a4101ae1f518ce9e371b721465fbb2652db8cd6a26cefa5a88bde"} Jul 10 00:18:09.708269 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:09.708253 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" event={"ID":"57273bce-c5a5-4b08-86b5-8f69c2eed2de","Type":"ContainerStarted","Data":"b7df0c4ab739380bc32e41a2b475fdf04e183bfe7b4ed72f1fbf7fb1500d36af"} Jul 10 00:18:09.709696 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:09.709671 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-30.ec2.internal" event={"ID":"fea6ac5a21d75add74f7c2411fe7019d","Type":"ContainerStarted","Data":"22d8a0118155e3ff43b5973c65dce91b8331c2b63bc8a3b7a389e2f6a06e9c27"} Jul 10 00:18:09.711030 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:09.711007 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gzg2p" event={"ID":"733f8b73-939c-414d-9d90-7791508e5b84","Type":"ContainerStarted","Data":"a906dd16bc03bf7e7d758e6b2dec750009ad72f12663d40307f7d343dad3a688"} Jul 10 00:18:09.712342 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:09.712321 2559 generic.go:356] "Generic (PLEG): container finished" podID="71ce3f1111f97924efae8ea5aa51d9e5" containerID="daacdcb76b93e61f3ae6a99743a4a5dc6676ecacd24822ad0f8d1031b3030b86" exitCode=0 Jul 10 00:18:09.712436 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:09.712356 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-30.ec2.internal" event={"ID":"71ce3f1111f97924efae8ea5aa51d9e5","Type":"ContainerDied","Data":"daacdcb76b93e61f3ae6a99743a4a5dc6676ecacd24822ad0f8d1031b3030b86"} Jul 10 00:18:09.713841 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:09.713819 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" event={"ID":"9de3defe-9135-40a5-994e-2a2b3c888c6d","Type":"ContainerStarted","Data":"a46dd39df30f6b6be101de3d9c2c4b1f429aaa4e9dce98f31c7754fefd5fbcb9"} Jul 10 00:18:09.715038 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:09.715017 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k2hs8" event={"ID":"f59c94d4-93a7-4602-bf06-b2e0a4b2e312","Type":"ContainerStarted","Data":"d4f65e9ea75d882bd4bcc401d3ad0753e1629c53ff78109b444611c6043c82ef"} Jul 10 00:18:09.716224 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:09.716203 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jdrfw" event={"ID":"613ac839-60fd-4f20-a127-a072d8a574bb","Type":"ContainerStarted","Data":"2e1fb7dc822e10f15897408004b2a63b2fa68bfebba634685b05396232ce3543"} Jul 10 00:18:09.717427 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:09.717410 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hf6sh" event={"ID":"de6e5b48-34f6-43bb-8b9d-d1e651860753","Type":"ContainerStarted","Data":"86def5f44c06793b04fad2ce980c7a968470b59bc9ba8526e48eb257d96cdef7"} Jul 10 00:18:09.754921 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:09.754760 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-30.ec2.internal" podStartSLOduration=21.754741167 podStartE2EDuration="21.754741167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:18:09.754384243 +0000 UTC m=+22.687468553" watchObservedRunningTime="2026-07-10 00:18:09.754741167 +0000 UTC m=+22.687825479" Jul 10 00:18:09.754921 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:09.754865 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nsx5f" podStartSLOduration=3.50372018 podStartE2EDuration="22.754857801s" totalImagesPullingTime="19.251137621s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:18:09.74257678 +0000 UTC m=+22.675661116" watchObservedRunningTime="2026-07-10 00:18:09.754857801 +0000 UTC m=+22.687942112" Jul 10 00:18:09.770687 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:09.770645 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gzg2p" podStartSLOduration=3.49090624 podStartE2EDuration="22.770629753s" totalImagesPullingTime="19.279723513s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:18:09.770249455 +0000 UTC m=+22.703333788" watchObservedRunningTime="2026-07-10 00:18:09.770629753 +0000 UTC m=+22.703714061" Jul 10 00:18:09.835830 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:09.835658 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hf6sh" podStartSLOduration=3.525780241 podStartE2EDuration="22.835641311s" totalImagesPullingTime="19.30986107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:18:09.810771131 +0000 UTC m=+22.743855442" watchObservedRunningTime="2026-07-10 00:18:09.835641311 +0000 UTC m=+22.768725619" Jul 10 00:18:09.847531 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:09.847477 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jdrfw" podStartSLOduration=3.977090123 podStartE2EDuration="22.847457496s" totalImagesPullingTime="18.870367373s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:18:09.8471037 +0000 UTC m=+22.780188016" watchObservedRunningTime="2026-07-10 00:18:09.847457496 +0000 UTC m=+22.780541806" Jul 10 00:18:09.859845 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:09.859800 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-k2hs8" podStartSLOduration=3.639174939 podStartE2EDuration="22.859785957s" totalImagesPullingTime="19.220611018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:18:09.859293714 +0000 UTC m=+22.792378050" watchObservedRunningTime="2026-07-10 00:18:09.859785957 +0000 UTC m=+22.792870266" Jul 10 00:18:10.644140 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:10.644060 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:18:10.644309 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:10.644206 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9mf" podUID="376630cb-0d38-46f3-a311-7cf9af6f2d10" Jul 10 00:18:10.722404 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:10.722355 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-30.ec2.internal" event={"ID":"71ce3f1111f97924efae8ea5aa51d9e5","Type":"ContainerStarted","Data":"167bc23c3dd249630ca4661b040ad2c84f645dd50f271ec05914744bbe30cd88"} Jul 10 00:18:10.726858 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:10.726832 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7zrr_20198eb1-6b3e-4e30-9adf-d4f13df4ee3b/ovn-acl-logging/0.log" Jul 10 00:18:10.727682 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:10.727568 2559 generic.go:356] "Generic (PLEG): container finished" podID="20198eb1-6b3e-4e30-9adf-d4f13df4ee3b" containerID="ab27eca5f342e402dbf7cc6bdd4fa34d16765c907b4362582a029cfe8b93f11e" exitCode=1 Jul 10 00:18:10.727682 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:10.727665 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" event={"ID":"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b","Type":"ContainerStarted","Data":"af3ea68c6d8d3a9038896c08bbc4801a87d72ac04253244591dd2ff381a4c461"} Jul 10 00:18:10.728013 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:10.727734 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" event={"ID":"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b","Type":"ContainerStarted","Data":"076b0784c4837ca2a6838a074ae7f229229fab062db25b0589a43eb2444aaf2d"} Jul 10 00:18:10.728013 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:10.727781 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" event={"ID":"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b","Type":"ContainerStarted","Data":"a5a54b9696ee3aeb5c8ecc690fdafb2ac137b29366459952cd9fd939851992fa"} Jul 10 00:18:10.728013 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:10.727819 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" event={"ID":"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b","Type":"ContainerStarted","Data":"f3dc083c894b53dc256c243f7db429e96d93ebd8b03b5c41991df63284b988d2"} Jul 10 00:18:10.728013 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:10.727854 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" event={"ID":"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b","Type":"ContainerDied","Data":"ab27eca5f342e402dbf7cc6bdd4fa34d16765c907b4362582a029cfe8b93f11e"} Jul 10 00:18:10.728013 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:10.727905 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" event={"ID":"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b","Type":"ContainerStarted","Data":"a1b12c9365922fac3d8ad477fca0fae41655ddf4f3bdf7a96bb105e929cb4c45"} Jul 10 00:18:10.729995 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:10.729970 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-n9prh" event={"ID":"f158539b-d568-492a-a0b6-0266deb4109e","Type":"ContainerStarted","Data":"9a860bd7a8efc878473c25fb5015c023b3b45bde844842d813a091d6670e494b"} Jul 10 00:18:10.736611 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:10.736569 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-30.ec2.internal" podStartSLOduration=22.736555615 podStartE2EDuration="22.736555615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:18:10.736224841 +0000 UTC m=+23.669309164" watchObservedRunningTime="2026-07-10 00:18:10.736555615 +0000 UTC m=+23.669639923" Jul 10 00:18:10.748325 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:10.748286 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-n9prh" podStartSLOduration=4.481475507 podStartE2EDuration="23.748273961s" totalImagesPullingTime="19.266798454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:18:10.748098645 +0000 UTC m=+23.681182953" watchObservedRunningTime="2026-07-10 00:18:10.748273961 +0000 UTC m=+23.681358269" Jul 10 00:18:10.801172 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:10.801085 2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 00:18:11.508998 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:11.508707 2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T00:18:10.801102664Z","UUID":"18040b2a-96e4-44cc-a7fd-bcf1d4ae4932","Handler":null,"Name":"","Endpoint":""} Jul 10 00:18:11.510434 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:11.510406 2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 00:18:11.510594 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:11.510449 2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 00:18:11.643578 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:11.643538 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:18:11.643578 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:11.643555 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:18:11.643813 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:11.643671 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqzc6" podUID="b1c3645d-2300-4b32-b0ef-e373064e5aad" Jul 10 00:18:11.643813 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:11.643784 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrzt4" podUID="4c396c5d-d578-4bc2-b02e-0d1f129d8247" Jul 10 00:18:11.733514 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:11.733483 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" event={"ID":"57273bce-c5a5-4b08-86b5-8f69c2eed2de","Type":"ContainerStarted","Data":"576a4048ebe65d9289e59ed7a75f6724753360f582e90295cb864d122572120d"} Jul 10 00:18:12.644352 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:12.644280 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:18:12.644485 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:12.644385 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9mf" podUID="376630cb-0d38-46f3-a311-7cf9af6f2d10" Jul 10 00:18:12.739099 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:12.739065 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" event={"ID":"57273bce-c5a5-4b08-86b5-8f69c2eed2de","Type":"ContainerStarted","Data":"bab770e532ec2f1dc9904ee96354e521806095cbdecc7de31ec272a874facb28"} Jul 10 00:18:12.755287 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:12.755239 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2hkpp" podStartSLOduration=3.9453714829999997 podStartE2EDuration="25.75521974s" totalImagesPullingTime="21.809848257s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:17:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:18:12.75486867 +0000 UTC m=+25.687952999" watchObservedRunningTime="2026-07-10 00:18:12.75521974 +0000 UTC m=+25.688304063" Jul 10 00:18:12.896887 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:12.896813 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-k2hs8" Jul 10 00:18:13.195472 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:13.195395 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-k2hs8" Jul 10 00:18:13.196202 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:13.196181 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-k2hs8" Jul 10 00:18:13.644174 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:13.644140 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:18:13.644357 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:13.644146 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:18:13.644357 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:13.644266 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrzt4" podUID="4c396c5d-d578-4bc2-b02e-0d1f129d8247" Jul 10 00:18:13.644465 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:13.644354 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqzc6" podUID="b1c3645d-2300-4b32-b0ef-e373064e5aad" Jul 10 00:18:13.744592 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:13.744569 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7zrr_20198eb1-6b3e-4e30-9adf-d4f13df4ee3b/ovn-acl-logging/0.log" Jul 10 00:18:13.745211 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:13.744972 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" event={"ID":"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b","Type":"ContainerStarted","Data":"7ae10198e38f8cec8217d9f42187961bbe7b9475affb5ea4ab957a581ef663a7"} Jul 10 00:18:13.745637 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:13.745623 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k2hs8" Jul 10 00:18:14.643819 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:14.643784 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:18:14.643979 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:14.643902 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9mf" podUID="376630cb-0d38-46f3-a311-7cf9af6f2d10" Jul 10 00:18:15.647314 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:15.647284 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:18:15.647830 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:15.647324 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:18:15.647830 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:15.647399 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqzc6" podUID="b1c3645d-2300-4b32-b0ef-e373064e5aad" Jul 10 00:18:15.647830 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:15.647561 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrzt4" podUID="4c396c5d-d578-4bc2-b02e-0d1f129d8247" Jul 10 00:18:16.644405 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:16.644237 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:18:16.644567 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:16.644491 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9mf" podUID="376630cb-0d38-46f3-a311-7cf9af6f2d10" Jul 10 00:18:16.756038 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:16.756013 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7zrr_20198eb1-6b3e-4e30-9adf-d4f13df4ee3b/ovn-acl-logging/0.log" Jul 10 00:18:16.756496 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:16.756410 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" event={"ID":"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b","Type":"ContainerStarted","Data":"8f310a28977e23bcaead4503ff93e1e01addb4ae0177d245ec8a2f03bf6e1af3"} Jul 10 00:18:16.756722 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:16.756677 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:18:16.756857 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:16.756841 2559 scope.go:117] "RemoveContainer" containerID="ab27eca5f342e402dbf7cc6bdd4fa34d16765c907b4362582a029cfe8b93f11e" Jul 10 00:18:16.758376 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:16.758355 2559 generic.go:356] "Generic (PLEG): container finished" podID="d642e8f4-ca88-46f0-ba74-e17a26ec2f7f" containerID="748440c1924814cc37912a680db5f7bbdd2948994e2b9d76c11bf1d92c30115a" exitCode=0 Jul 10 00:18:16.758449 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:16.758393 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8hp4r" event={"ID":"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f","Type":"ContainerDied","Data":"748440c1924814cc37912a680db5f7bbdd2948994e2b9d76c11bf1d92c30115a"} Jul 10 00:18:16.773491 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:16.773468 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:18:17.644342 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:17.644308 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:18:17.644509 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:17.644404 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqzc6" podUID="b1c3645d-2300-4b32-b0ef-e373064e5aad" Jul 10 00:18:17.644509 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:17.644455 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:18:17.644631 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:17.644561 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrzt4" podUID="4c396c5d-d578-4bc2-b02e-0d1f129d8247" Jul 10 00:18:17.765107 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:17.765083 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7zrr_20198eb1-6b3e-4e30-9adf-d4f13df4ee3b/ovn-acl-logging/0.log" Jul 10 00:18:17.765568 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:17.765532 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" event={"ID":"20198eb1-6b3e-4e30-9adf-d4f13df4ee3b","Type":"ContainerStarted","Data":"5cc1e02ce8e317d218b65c7680770b4b56ed29e4c4f994453b94691f372b4ad1"} Jul 10 00:18:17.765825 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:17.765797 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:18:17.765935 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:17.765881 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:18:17.780388 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:17.780367 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:18:17.792231 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:17.792193 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" podStartSLOduration=9.928688448 podStartE2EDuration="29.792162369s" totalImagesPullingTime="19.863473921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:17:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:18:17.791689742 +0000 UTC m=+30.724774071" watchObservedRunningTime="2026-07-10 00:18:17.792162369 +0000 UTC m=+30.725246677" Jul 10 00:18:18.133585 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:18.133313 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wrzt4"] Jul 10 00:18:18.133585 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:18.133568 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:18:18.133771 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:18.133665 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrzt4" podUID="4c396c5d-d578-4bc2-b02e-0d1f129d8247" Jul 10 00:18:18.136869 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:18.136844 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zqzc6"] Jul 10 00:18:18.137011 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:18.136929 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:18:18.137071 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:18.137024 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqzc6" podUID="b1c3645d-2300-4b32-b0ef-e373064e5aad" Jul 10 00:18:18.137491 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:18.137467 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ms9mf"] Jul 10 00:18:18.137592 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:18.137577 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:18:18.137699 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:18.137681 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9mf" podUID="376630cb-0d38-46f3-a311-7cf9af6f2d10" Jul 10 00:18:18.770622 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:18.770592 2559 generic.go:356] "Generic (PLEG): container finished" podID="d642e8f4-ca88-46f0-ba74-e17a26ec2f7f" containerID="2c01604ec537fed913d9bb6577306a208942ca9959f5216ecb80fd9d55ca0621" exitCode=0 Jul 10 00:18:18.771265 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:18.770688 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8hp4r" event={"ID":"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f","Type":"ContainerDied","Data":"2c01604ec537fed913d9bb6577306a208942ca9959f5216ecb80fd9d55ca0621"} Jul 10 00:18:19.643837 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:19.643807 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:18:19.643966 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:19.643810 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:18:19.643966 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:19.643931 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:18:19.643966 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:19.643926 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrzt4" podUID="4c396c5d-d578-4bc2-b02e-0d1f129d8247" Jul 10 00:18:19.644138 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:19.644019 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqzc6" podUID="b1c3645d-2300-4b32-b0ef-e373064e5aad" Jul 10 00:18:19.644189 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:19.644134 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9mf" podUID="376630cb-0d38-46f3-a311-7cf9af6f2d10" Jul 10 00:18:20.779030 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:20.778998 2559 generic.go:356] "Generic (PLEG): container finished" podID="d642e8f4-ca88-46f0-ba74-e17a26ec2f7f" containerID="dcb0f21f2bce63d1458e27d41e8a7039f2feb30d784d9cf3c77a6bb6e50f0465" exitCode=0 Jul 10 00:18:20.779557 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:20.779054 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8hp4r" event={"ID":"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f","Type":"ContainerDied","Data":"dcb0f21f2bce63d1458e27d41e8a7039f2feb30d784d9cf3c77a6bb6e50f0465"} Jul 10 00:18:21.209255 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:21.209175 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1c3645d-2300-4b32-b0ef-e373064e5aad-original-pull-secret\") pod \"global-pull-secret-syncer-zqzc6\" (UID: \"b1c3645d-2300-4b32-b0ef-e373064e5aad\") " pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:18:21.209255 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:21.209217 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:18:21.209422 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:21.209270 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1c3645d-2300-4b32-b0ef-e373064e5aad-original-pull-secret podName:b1c3645d-2300-4b32-b0ef-e373064e5aad nodeName:}" failed. No retries permitted until 2026-07-10 00:18:53.2092567 +0000 UTC m=+66.142340987 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1c3645d-2300-4b32-b0ef-e373064e5aad-original-pull-secret") pod "global-pull-secret-syncer-zqzc6" (UID: "b1c3645d-2300-4b32-b0ef-e373064e5aad") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:18:21.310029 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:21.310001 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs\") pod \"network-metrics-daemon-ms9mf\" (UID: \"376630cb-0d38-46f3-a311-7cf9af6f2d10\") " pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:18:21.310172 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:21.310036 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdbw5\" (UniqueName: \"kubernetes.io/projected/4c396c5d-d578-4bc2-b02e-0d1f129d8247-kube-api-access-kdbw5\") pod \"network-check-target-wrzt4\" (UID: \"4c396c5d-d578-4bc2-b02e-0d1f129d8247\") " pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:18:21.310229 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:21.310166 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:18:21.310262 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:21.310230 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs podName:376630cb-0d38-46f3-a311-7cf9af6f2d10 nodeName:}" failed. No retries permitted until 2026-07-10 00:18:53.310212707 +0000 UTC m=+66.243296994 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs") pod "network-metrics-daemon-ms9mf" (UID: "376630cb-0d38-46f3-a311-7cf9af6f2d10") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:18:21.310262 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:21.310170 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:18:21.310334 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:21.310266 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:18:21.310334 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:21.310279 2559 projected.go:196] Error preparing data for projected volume kube-api-access-kdbw5 for pod openshift-network-diagnostics/network-check-target-wrzt4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:18:21.310334 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:21.310323 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c396c5d-d578-4bc2-b02e-0d1f129d8247-kube-api-access-kdbw5 podName:4c396c5d-d578-4bc2-b02e-0d1f129d8247 nodeName:}" failed. No retries permitted until 2026-07-10 00:18:53.310309638 +0000 UTC m=+66.243393926 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kdbw5" (UniqueName: "kubernetes.io/projected/4c396c5d-d578-4bc2-b02e-0d1f129d8247-kube-api-access-kdbw5") pod "network-check-target-wrzt4" (UID: "4c396c5d-d578-4bc2-b02e-0d1f129d8247") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:18:21.643855 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:21.643822 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:18:21.644032 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:21.643834 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:18:21.644032 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:21.643957 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqzc6" podUID="b1c3645d-2300-4b32-b0ef-e373064e5aad" Jul 10 00:18:21.644032 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:21.644024 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9mf" podUID="376630cb-0d38-46f3-a311-7cf9af6f2d10" Jul 10 00:18:21.644209 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:21.644071 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:18:21.644209 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:21.644152 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wrzt4" podUID="4c396c5d-d578-4bc2-b02e-0d1f129d8247" Jul 10 00:18:22.879691 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:22.879662 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-30.ec2.internal" event="NodeReady" Jul 10 00:18:22.880139 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:22.879779 2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 00:18:22.922550 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:22.922520 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ds66q"] Jul 10 00:18:22.924932 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:22.924908 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ds66q" Jul 10 00:18:22.925533 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:22.925513 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-q8pzq"] Jul 10 00:18:22.927058 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:22.927032 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 00:18:22.927188 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:22.927104 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 00:18:22.927251 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:22.927129 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p75jv\"" Jul 10 00:18:22.928101 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:22.928083 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q8pzq" Jul 10 00:18:22.929876 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:22.929844 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 00:18:22.930038 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:22.930027 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ffdj8\"" Jul 10 00:18:22.930192 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:22.930177 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 00:18:22.930407 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:22.930389 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 00:18:22.934608 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:22.934590 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ds66q"] Jul 10 00:18:22.945750 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:22.945730 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q8pzq"] Jul 10 00:18:23.020131 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.020090 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgc9f\" (UniqueName: \"kubernetes.io/projected/5918e467-096f-431c-a997-d13e6ddd04e0-kube-api-access-sgc9f\") pod \"ingress-canary-q8pzq\" (UID: \"5918e467-096f-431c-a997-d13e6ddd04e0\") " pod="openshift-ingress-canary/ingress-canary-q8pzq" Jul 10 00:18:23.020273 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.020137 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0654b471-0fff-4b5c-a6fc-d3e972163d20-tmp-dir\") pod \"dns-default-ds66q\" (UID: \"0654b471-0fff-4b5c-a6fc-d3e972163d20\") " pod="openshift-dns/dns-default-ds66q" Jul 10 00:18:23.020273 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.020161 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz48h\" (UniqueName: \"kubernetes.io/projected/0654b471-0fff-4b5c-a6fc-d3e972163d20-kube-api-access-pz48h\") pod \"dns-default-ds66q\" (UID: \"0654b471-0fff-4b5c-a6fc-d3e972163d20\") " pod="openshift-dns/dns-default-ds66q" Jul 10 00:18:23.020273 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.020210 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0654b471-0fff-4b5c-a6fc-d3e972163d20-config-volume\") pod \"dns-default-ds66q\" (UID: \"0654b471-0fff-4b5c-a6fc-d3e972163d20\") " pod="openshift-dns/dns-default-ds66q" Jul 10 00:18:23.020273 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.020245 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5918e467-096f-431c-a997-d13e6ddd04e0-cert\") pod \"ingress-canary-q8pzq\" (UID: \"5918e467-096f-431c-a997-d13e6ddd04e0\") " pod="openshift-ingress-canary/ingress-canary-q8pzq" Jul 10 00:18:23.020401 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.020280 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls\") pod \"dns-default-ds66q\" (UID: \"0654b471-0fff-4b5c-a6fc-d3e972163d20\") " pod="openshift-dns/dns-default-ds66q" Jul 10 00:18:23.121385 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.121354 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgc9f\" (UniqueName: \"kubernetes.io/projected/5918e467-096f-431c-a997-d13e6ddd04e0-kube-api-access-sgc9f\") pod \"ingress-canary-q8pzq\" (UID: \"5918e467-096f-431c-a997-d13e6ddd04e0\") " pod="openshift-ingress-canary/ingress-canary-q8pzq" Jul 10 00:18:23.121549 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.121390 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0654b471-0fff-4b5c-a6fc-d3e972163d20-tmp-dir\") pod \"dns-default-ds66q\" (UID: \"0654b471-0fff-4b5c-a6fc-d3e972163d20\") " pod="openshift-dns/dns-default-ds66q" Jul 10 00:18:23.121549 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.121416 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz48h\" (UniqueName: \"kubernetes.io/projected/0654b471-0fff-4b5c-a6fc-d3e972163d20-kube-api-access-pz48h\") pod \"dns-default-ds66q\" (UID: \"0654b471-0fff-4b5c-a6fc-d3e972163d20\") " pod="openshift-dns/dns-default-ds66q" Jul 10 00:18:23.121549 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.121456 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0654b471-0fff-4b5c-a6fc-d3e972163d20-config-volume\") pod \"dns-default-ds66q\" (UID: \"0654b471-0fff-4b5c-a6fc-d3e972163d20\") " pod="openshift-dns/dns-default-ds66q" Jul 10 00:18:23.121549 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.121508 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5918e467-096f-431c-a997-d13e6ddd04e0-cert\") pod \"ingress-canary-q8pzq\" (UID: \"5918e467-096f-431c-a997-d13e6ddd04e0\") " pod="openshift-ingress-canary/ingress-canary-q8pzq" Jul 10 00:18:23.121754 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.121631 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls\") pod \"dns-default-ds66q\" (UID: \"0654b471-0fff-4b5c-a6fc-d3e972163d20\") " pod="openshift-dns/dns-default-ds66q" Jul 10 00:18:23.121804 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:23.121758 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:18:23.121804 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.121773 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0654b471-0fff-4b5c-a6fc-d3e972163d20-tmp-dir\") pod \"dns-default-ds66q\" (UID: \"0654b471-0fff-4b5c-a6fc-d3e972163d20\") " pod="openshift-dns/dns-default-ds66q" Jul 10 00:18:23.121898 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:23.121819 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls podName:0654b471-0fff-4b5c-a6fc-d3e972163d20 nodeName:}" failed. No retries permitted until 2026-07-10 00:18:23.621796866 +0000 UTC m=+36.554881172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls") pod "dns-default-ds66q" (UID: "0654b471-0fff-4b5c-a6fc-d3e972163d20") : secret "dns-default-metrics-tls" not found Jul 10 00:18:23.122363 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.122154 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0654b471-0fff-4b5c-a6fc-d3e972163d20-config-volume\") pod \"dns-default-ds66q\" (UID: \"0654b471-0fff-4b5c-a6fc-d3e972163d20\") " pod="openshift-dns/dns-default-ds66q" Jul 10 00:18:23.125844 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.125826 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5918e467-096f-431c-a997-d13e6ddd04e0-cert\") pod \"ingress-canary-q8pzq\" (UID: \"5918e467-096f-431c-a997-d13e6ddd04e0\") " pod="openshift-ingress-canary/ingress-canary-q8pzq" Jul 10 00:18:23.131985 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.131934 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgc9f\" (UniqueName: \"kubernetes.io/projected/5918e467-096f-431c-a997-d13e6ddd04e0-kube-api-access-sgc9f\") pod \"ingress-canary-q8pzq\" (UID: \"5918e467-096f-431c-a997-d13e6ddd04e0\") " pod="openshift-ingress-canary/ingress-canary-q8pzq" Jul 10 00:18:23.132088 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.132044 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz48h\" (UniqueName: \"kubernetes.io/projected/0654b471-0fff-4b5c-a6fc-d3e972163d20-kube-api-access-pz48h\") pod \"dns-default-ds66q\" (UID: \"0654b471-0fff-4b5c-a6fc-d3e972163d20\") " pod="openshift-dns/dns-default-ds66q" Jul 10 00:18:23.240777 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.240748 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-q8pzq" Jul 10 00:18:23.379999 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.379970 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-q8pzq"] Jul 10 00:18:23.384190 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:18:23.384166 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5918e467_096f_431c_a997_d13e6ddd04e0.slice/crio-5ab14b3d7e794fe2cce37443eb30d93edd7f5ac4a892596900654c1c490999af WatchSource:0}: Error finding container 5ab14b3d7e794fe2cce37443eb30d93edd7f5ac4a892596900654c1c490999af: Status 404 returned error can't find the container with id 5ab14b3d7e794fe2cce37443eb30d93edd7f5ac4a892596900654c1c490999af Jul 10 00:18:23.625135 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.625049 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls\") pod \"dns-default-ds66q\" (UID: \"0654b471-0fff-4b5c-a6fc-d3e972163d20\") " pod="openshift-dns/dns-default-ds66q" Jul 10 00:18:23.625268 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:23.625182 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:18:23.625268 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:23.625233 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls podName:0654b471-0fff-4b5c-a6fc-d3e972163d20 nodeName:}" failed. No retries permitted until 2026-07-10 00:18:24.625218376 +0000 UTC m=+37.558302663 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls") pod "dns-default-ds66q" (UID: "0654b471-0fff-4b5c-a6fc-d3e972163d20") : secret "dns-default-metrics-tls" not found Jul 10 00:18:23.644468 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.644439 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:18:23.644605 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.644499 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:18:23.644605 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.644500 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:18:23.646874 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.646853 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 00:18:23.647008 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.646888 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 00:18:23.647008 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.646893 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l7wp6\"" Jul 10 00:18:23.647474 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.647455 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 00:18:23.647618 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.647451 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7p4r9\"" Jul 10 00:18:23.647618 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.647458 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 00:18:23.787775 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:23.787738 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q8pzq" event={"ID":"5918e467-096f-431c-a997-d13e6ddd04e0","Type":"ContainerStarted","Data":"5ab14b3d7e794fe2cce37443eb30d93edd7f5ac4a892596900654c1c490999af"} Jul 10 00:18:24.633262 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:24.633226 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls\") pod \"dns-default-ds66q\" (UID: \"0654b471-0fff-4b5c-a6fc-d3e972163d20\") " pod="openshift-dns/dns-default-ds66q" Jul 10 00:18:24.633810 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:24.633353 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:18:24.633810 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:24.633413 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls podName:0654b471-0fff-4b5c-a6fc-d3e972163d20 nodeName:}" failed. No retries permitted until 2026-07-10 00:18:26.633398329 +0000 UTC m=+39.566482616 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls") pod "dns-default-ds66q" (UID: "0654b471-0fff-4b5c-a6fc-d3e972163d20") : secret "dns-default-metrics-tls" not found Jul 10 00:18:26.647722 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:26.647490 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls\") pod \"dns-default-ds66q\" (UID: \"0654b471-0fff-4b5c-a6fc-d3e972163d20\") " pod="openshift-dns/dns-default-ds66q" Jul 10 00:18:26.648083 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:26.647872 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:18:26.648083 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:26.647941 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls podName:0654b471-0fff-4b5c-a6fc-d3e972163d20 nodeName:}" failed. No retries permitted until 2026-07-10 00:18:30.647921074 +0000 UTC m=+43.581005363 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls") pod "dns-default-ds66q" (UID: "0654b471-0fff-4b5c-a6fc-d3e972163d20") : secret "dns-default-metrics-tls" not found Jul 10 00:18:26.797007 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:26.796961 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-q8pzq" event={"ID":"5918e467-096f-431c-a997-d13e6ddd04e0","Type":"ContainerStarted","Data":"abb8c1b18dce046721f3b72c7e7571d868e276bea576542f34a9ce7cad816cac"} Jul 10 00:18:26.811846 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:26.811802 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-q8pzq" podStartSLOduration=1.874588741 podStartE2EDuration="4.811786678s" totalImagesPullingTime="2.937197937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:18:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:18:26.810602809 +0000 UTC m=+39.743687118" watchObservedRunningTime="2026-07-10 00:18:26.811786678 +0000 UTC m=+39.744870988" Jul 10 00:18:29.807524 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:29.807484 2559 generic.go:356] "Generic (PLEG): container finished" podID="d642e8f4-ca88-46f0-ba74-e17a26ec2f7f" containerID="b1b336e266ce4181e2891b959aa9a3b916c102321d43a915bb07ea72a9e2f5b7" exitCode=0 Jul 10 00:18:29.807895 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:29.807543 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8hp4r" event={"ID":"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f","Type":"ContainerDied","Data":"b1b336e266ce4181e2891b959aa9a3b916c102321d43a915bb07ea72a9e2f5b7"} Jul 10 00:18:30.677662 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:30.677621 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls\") pod \"dns-default-ds66q\" (UID: \"0654b471-0fff-4b5c-a6fc-d3e972163d20\") " pod="openshift-dns/dns-default-ds66q" Jul 10 00:18:30.677857 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:30.677788 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:18:30.677902 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:30.677858 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls podName:0654b471-0fff-4b5c-a6fc-d3e972163d20 nodeName:}" failed. No retries permitted until 2026-07-10 00:18:38.677842511 +0000 UTC m=+51.610926800 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls") pod "dns-default-ds66q" (UID: "0654b471-0fff-4b5c-a6fc-d3e972163d20") : secret "dns-default-metrics-tls" not found Jul 10 00:18:30.813030 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:30.812991 2559 generic.go:356] "Generic (PLEG): container finished" podID="d642e8f4-ca88-46f0-ba74-e17a26ec2f7f" containerID="124b65b822205547d66045f9dd6bc83a70531a57520c16ed1d2718dc267ed070" exitCode=0 Jul 10 00:18:30.813424 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:30.813075 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8hp4r" event={"ID":"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f","Type":"ContainerDied","Data":"124b65b822205547d66045f9dd6bc83a70531a57520c16ed1d2718dc267ed070"} Jul 10 00:18:31.819251 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:31.819217 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8hp4r" event={"ID":"d642e8f4-ca88-46f0-ba74-e17a26ec2f7f","Type":"ContainerStarted","Data":"8247951b7a97ef6354e2e4d1a907a2d554bb7fc2866a864907843fa0cf4e5f5b"} Jul 10 00:18:31.840715 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:31.840668 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8hp4r" podStartSLOduration=8.372247448 podStartE2EDuration="44.840652714s" totalImagesPullingTime="36.468405266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:17:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:18:31.839466417 +0000 UTC m=+44.772550726" watchObservedRunningTime="2026-07-10 00:18:31.840652714 +0000 UTC m=+44.773737023" Jul 10 00:18:38.731912 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:38.731867 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls\") pod \"dns-default-ds66q\" (UID: \"0654b471-0fff-4b5c-a6fc-d3e972163d20\") " pod="openshift-dns/dns-default-ds66q" Jul 10 00:18:38.732456 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:38.732027 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:18:38.732456 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:38.732110 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls podName:0654b471-0fff-4b5c-a6fc-d3e972163d20 nodeName:}" failed. No retries permitted until 2026-07-10 00:18:54.732088962 +0000 UTC m=+67.665173267 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls") pod "dns-default-ds66q" (UID: "0654b471-0fff-4b5c-a6fc-d3e972163d20") : secret "dns-default-metrics-tls" not found Jul 10 00:18:49.785006 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:49.784967 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-p7zrr" Jul 10 00:18:53.234271 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:53.234238 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1c3645d-2300-4b32-b0ef-e373064e5aad-original-pull-secret\") pod \"global-pull-secret-syncer-zqzc6\" (UID: \"b1c3645d-2300-4b32-b0ef-e373064e5aad\") " pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:18:53.236384 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:53.236367 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 00:18:53.248320 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:53.248302 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1c3645d-2300-4b32-b0ef-e373064e5aad-original-pull-secret\") pod \"global-pull-secret-syncer-zqzc6\" (UID: \"b1c3645d-2300-4b32-b0ef-e373064e5aad\") " pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:18:53.334624 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:53.334598 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs\") pod \"network-metrics-daemon-ms9mf\" (UID: \"376630cb-0d38-46f3-a311-7cf9af6f2d10\") " pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:18:53.334734 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:53.334629 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdbw5\" (UniqueName: \"kubernetes.io/projected/4c396c5d-d578-4bc2-b02e-0d1f129d8247-kube-api-access-kdbw5\") pod \"network-check-target-wrzt4\" (UID: \"4c396c5d-d578-4bc2-b02e-0d1f129d8247\") " pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:18:53.336832 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:53.336810 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 00:18:53.336984 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:53.336967 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 00:18:53.345490 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:53.345469 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 00:18:53.345577 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:53.345539 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs podName:376630cb-0d38-46f3-a311-7cf9af6f2d10 nodeName:}" failed. No retries permitted until 2026-07-10 00:19:57.345516602 +0000 UTC m=+130.278600889 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs") pod "network-metrics-daemon-ms9mf" (UID: "376630cb-0d38-46f3-a311-7cf9af6f2d10") : secret "metrics-daemon-secret" not found Jul 10 00:18:53.347109 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:53.347094 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 00:18:53.357978 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:53.357960 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdbw5\" (UniqueName: \"kubernetes.io/projected/4c396c5d-d578-4bc2-b02e-0d1f129d8247-kube-api-access-kdbw5\") pod \"network-check-target-wrzt4\" (UID: \"4c396c5d-d578-4bc2-b02e-0d1f129d8247\") " pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:18:53.360734 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:53.360711 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqzc6" Jul 10 00:18:53.367849 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:53.367833 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7p4r9\"" Jul 10 00:18:53.376378 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:53.376360 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:18:53.484881 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:53.484812 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zqzc6"] Jul 10 00:18:53.487951 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:18:53.487923 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c3645d_2300_4b32_b0ef_e373064e5aad.slice/crio-df0289d71e9b2dc3344c79b672bbf57762bbfee39e882a50e13efec3e2130377 WatchSource:0}: Error finding container df0289d71e9b2dc3344c79b672bbf57762bbfee39e882a50e13efec3e2130377: Status 404 returned error can't find the container with id df0289d71e9b2dc3344c79b672bbf57762bbfee39e882a50e13efec3e2130377 Jul 10 00:18:53.499641 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:53.499617 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wrzt4"] Jul 10 00:18:53.502633 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:18:53.502612 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c396c5d_d578_4bc2_b02e_0d1f129d8247.slice/crio-cae88cd471a31f96b072e916840de981a05019f890c2f3779a2f81f5126bf202 WatchSource:0}: Error finding container cae88cd471a31f96b072e916840de981a05019f890c2f3779a2f81f5126bf202: Status 404 returned error can't find the container with id cae88cd471a31f96b072e916840de981a05019f890c2f3779a2f81f5126bf202 Jul 10 00:18:53.868742 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:53.868706 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wrzt4" event={"ID":"4c396c5d-d578-4bc2-b02e-0d1f129d8247","Type":"ContainerStarted","Data":"cae88cd471a31f96b072e916840de981a05019f890c2f3779a2f81f5126bf202"} Jul 10 00:18:53.869665 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:53.869643 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zqzc6" event={"ID":"b1c3645d-2300-4b32-b0ef-e373064e5aad","Type":"ContainerStarted","Data":"df0289d71e9b2dc3344c79b672bbf57762bbfee39e882a50e13efec3e2130377"} Jul 10 00:18:54.744533 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:54.744502 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls\") pod \"dns-default-ds66q\" (UID: \"0654b471-0fff-4b5c-a6fc-d3e972163d20\") " pod="openshift-dns/dns-default-ds66q" Jul 10 00:18:54.744960 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:54.744637 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:18:54.744960 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:18:54.744695 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls podName:0654b471-0fff-4b5c-a6fc-d3e972163d20 nodeName:}" failed. No retries permitted until 2026-07-10 00:19:26.744679409 +0000 UTC m=+99.677763695 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls") pod "dns-default-ds66q" (UID: "0654b471-0fff-4b5c-a6fc-d3e972163d20") : secret "dns-default-metrics-tls" not found Jul 10 00:18:59.885897 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:59.885861 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wrzt4" event={"ID":"4c396c5d-d578-4bc2-b02e-0d1f129d8247","Type":"ContainerStarted","Data":"8542de5eb990b830c18044859ecd053365c13cf2189b2c72f38411dbf0093c3d"} Jul 10 00:18:59.886382 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:59.885982 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:18:59.887539 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:59.887509 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zqzc6" event={"ID":"b1c3645d-2300-4b32-b0ef-e373064e5aad","Type":"ContainerStarted","Data":"a8f6db0c24d55e46421b8d2878741696087b2d4239c39226cee3ad505c13102a"} Jul 10 00:18:59.901349 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:18:59.901305 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wrzt4" podStartSLOduration=67.255661948 podStartE2EDuration="1m12.901289261s" totalImagesPullingTime="5.645627313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:18:59.900718719 +0000 UTC m=+72.833803048" watchObservedRunningTime="2026-07-10 00:18:59.901289261 +0000 UTC m=+72.834373571" Jul 10 00:19:08.922203 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:08.922156 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zqzc6" podStartSLOduration=76.249373088 podStartE2EDuration="1m21.922139877s" totalImagesPullingTime="5.672766789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:18:59.913470026 +0000 UTC m=+72.846554336" watchObservedRunningTime="2026-07-10 00:19:08.922139877 +0000 UTC m=+81.855224164" Jul 10 00:19:08.922600 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:08.922386 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b654b8d4-rwmhc"] Jul 10 00:19:08.924986 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:08.924949 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b654b8d4-rwmhc" Jul 10 00:19:08.925843 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:08.925823 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk"] Jul 10 00:19:08.927608 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:08.927576 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-9p75l\"" Jul 10 00:19:08.927608 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:08.927593 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 10 00:19:08.927760 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:08.927659 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 10 00:19:08.927760 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:08.927601 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 10 00:19:08.927760 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:08.927599 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 10 00:19:08.928378 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:08.928365 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" Jul 10 00:19:08.930177 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:08.930160 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 10 00:19:08.930369 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:08.930345 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 10 00:19:08.930485 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:08.930421 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 10 00:19:08.930485 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:08.930458 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 10 00:19:08.934842 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:08.934826 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b654b8d4-rwmhc"] Jul 10 00:19:08.941163 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:08.941104 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk"] Jul 10 00:19:09.056370 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.056341 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a93bcd7-0c6b-45bc-ac54-b7c6873f7748-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6f85bff9fb-6qmhk\" (UID: \"4a93bcd7-0c6b-45bc-ac54-b7c6873f7748\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" Jul 10 00:19:09.056494 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.056387 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4a93bcd7-0c6b-45bc-ac54-b7c6873f7748-ca\") pod \"cluster-proxy-proxy-agent-6f85bff9fb-6qmhk\" (UID: \"4a93bcd7-0c6b-45bc-ac54-b7c6873f7748\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" Jul 10 00:19:09.056494 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.056411 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4a93bcd7-0c6b-45bc-ac54-b7c6873f7748-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f85bff9fb-6qmhk\" (UID: \"4a93bcd7-0c6b-45bc-ac54-b7c6873f7748\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" Jul 10 00:19:09.056494 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.056432 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00ae2e20-33bc-4ed1-bfab-ef18b5bef277-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75b654b8d4-rwmhc\" (UID: \"00ae2e20-33bc-4ed1-bfab-ef18b5bef277\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b654b8d4-rwmhc" Jul 10 00:19:09.056494 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.056471 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6xsg\" (UniqueName: \"kubernetes.io/projected/4a93bcd7-0c6b-45bc-ac54-b7c6873f7748-kube-api-access-d6xsg\") pod \"cluster-proxy-proxy-agent-6f85bff9fb-6qmhk\" (UID: \"4a93bcd7-0c6b-45bc-ac54-b7c6873f7748\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" Jul 10 00:19:09.056494 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.056491 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4a93bcd7-0c6b-45bc-ac54-b7c6873f7748-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f85bff9fb-6qmhk\" (UID: \"4a93bcd7-0c6b-45bc-ac54-b7c6873f7748\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" Jul 10 00:19:09.056662 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.056543 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4a93bcd7-0c6b-45bc-ac54-b7c6873f7748-hub\") pod \"cluster-proxy-proxy-agent-6f85bff9fb-6qmhk\" (UID: \"4a93bcd7-0c6b-45bc-ac54-b7c6873f7748\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" Jul 10 00:19:09.056662 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.056575 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq2hc\" (UniqueName: \"kubernetes.io/projected/00ae2e20-33bc-4ed1-bfab-ef18b5bef277-kube-api-access-jq2hc\") pod \"managed-serviceaccount-addon-agent-75b654b8d4-rwmhc\" (UID: \"00ae2e20-33bc-4ed1-bfab-ef18b5bef277\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b654b8d4-rwmhc" Jul 10 00:19:09.157260 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.157182 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq2hc\" (UniqueName: \"kubernetes.io/projected/00ae2e20-33bc-4ed1-bfab-ef18b5bef277-kube-api-access-jq2hc\") pod \"managed-serviceaccount-addon-agent-75b654b8d4-rwmhc\" (UID: \"00ae2e20-33bc-4ed1-bfab-ef18b5bef277\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b654b8d4-rwmhc" Jul 10 00:19:09.157260 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.157215 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a93bcd7-0c6b-45bc-ac54-b7c6873f7748-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6f85bff9fb-6qmhk\" (UID: \"4a93bcd7-0c6b-45bc-ac54-b7c6873f7748\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" Jul 10 00:19:09.157260 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.157239 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4a93bcd7-0c6b-45bc-ac54-b7c6873f7748-ca\") pod \"cluster-proxy-proxy-agent-6f85bff9fb-6qmhk\" (UID: \"4a93bcd7-0c6b-45bc-ac54-b7c6873f7748\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" Jul 10 00:19:09.157403 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.157269 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4a93bcd7-0c6b-45bc-ac54-b7c6873f7748-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f85bff9fb-6qmhk\" (UID: \"4a93bcd7-0c6b-45bc-ac54-b7c6873f7748\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" Jul 10 00:19:09.157403 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.157298 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00ae2e20-33bc-4ed1-bfab-ef18b5bef277-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75b654b8d4-rwmhc\" (UID: \"00ae2e20-33bc-4ed1-bfab-ef18b5bef277\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b654b8d4-rwmhc" Jul 10 00:19:09.157403 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.157340 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6xsg\" (UniqueName: \"kubernetes.io/projected/4a93bcd7-0c6b-45bc-ac54-b7c6873f7748-kube-api-access-d6xsg\") pod \"cluster-proxy-proxy-agent-6f85bff9fb-6qmhk\" (UID: \"4a93bcd7-0c6b-45bc-ac54-b7c6873f7748\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" Jul 10 00:19:09.157403 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.157370 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4a93bcd7-0c6b-45bc-ac54-b7c6873f7748-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f85bff9fb-6qmhk\" (UID: \"4a93bcd7-0c6b-45bc-ac54-b7c6873f7748\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" Jul 10 00:19:09.157403 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.157393 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4a93bcd7-0c6b-45bc-ac54-b7c6873f7748-hub\") pod \"cluster-proxy-proxy-agent-6f85bff9fb-6qmhk\" (UID: \"4a93bcd7-0c6b-45bc-ac54-b7c6873f7748\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" Jul 10 00:19:09.158029 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.158003 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a93bcd7-0c6b-45bc-ac54-b7c6873f7748-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6f85bff9fb-6qmhk\" (UID: \"4a93bcd7-0c6b-45bc-ac54-b7c6873f7748\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" Jul 10 00:19:09.159839 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.159816 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/4a93bcd7-0c6b-45bc-ac54-b7c6873f7748-hub\") pod \"cluster-proxy-proxy-agent-6f85bff9fb-6qmhk\" (UID: \"4a93bcd7-0c6b-45bc-ac54-b7c6873f7748\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" Jul 10 00:19:09.159974 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.159944 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/4a93bcd7-0c6b-45bc-ac54-b7c6873f7748-ca\") pod \"cluster-proxy-proxy-agent-6f85bff9fb-6qmhk\" (UID: \"4a93bcd7-0c6b-45bc-ac54-b7c6873f7748\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" Jul 10 00:19:09.160061 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.159983 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/4a93bcd7-0c6b-45bc-ac54-b7c6873f7748-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6f85bff9fb-6qmhk\" (UID: \"4a93bcd7-0c6b-45bc-ac54-b7c6873f7748\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" Jul 10 00:19:09.160269 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.160255 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4a93bcd7-0c6b-45bc-ac54-b7c6873f7748-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6f85bff9fb-6qmhk\" (UID: \"4a93bcd7-0c6b-45bc-ac54-b7c6873f7748\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" Jul 10 00:19:09.160543 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.160524 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/00ae2e20-33bc-4ed1-bfab-ef18b5bef277-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-75b654b8d4-rwmhc\" (UID: \"00ae2e20-33bc-4ed1-bfab-ef18b5bef277\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b654b8d4-rwmhc" Jul 10 00:19:09.164370 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.164348 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6xsg\" (UniqueName: \"kubernetes.io/projected/4a93bcd7-0c6b-45bc-ac54-b7c6873f7748-kube-api-access-d6xsg\") pod \"cluster-proxy-proxy-agent-6f85bff9fb-6qmhk\" (UID: \"4a93bcd7-0c6b-45bc-ac54-b7c6873f7748\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" Jul 10 00:19:09.164631 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.164609 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq2hc\" (UniqueName: \"kubernetes.io/projected/00ae2e20-33bc-4ed1-bfab-ef18b5bef277-kube-api-access-jq2hc\") pod \"managed-serviceaccount-addon-agent-75b654b8d4-rwmhc\" (UID: \"00ae2e20-33bc-4ed1-bfab-ef18b5bef277\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b654b8d4-rwmhc" Jul 10 00:19:09.253547 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.253485 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b654b8d4-rwmhc" Jul 10 00:19:09.258487 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.258465 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" Jul 10 00:19:09.587379 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.587276 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b654b8d4-rwmhc"] Jul 10 00:19:09.589890 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:19:09.589863 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00ae2e20_33bc_4ed1_bfab_ef18b5bef277.slice/crio-436f262b855d21d0e446ea0e34b44595c81d09af991899dc0baabae3cfd69f7d WatchSource:0}: Error finding container 436f262b855d21d0e446ea0e34b44595c81d09af991899dc0baabae3cfd69f7d: Status 404 returned error can't find the container with id 436f262b855d21d0e446ea0e34b44595c81d09af991899dc0baabae3cfd69f7d Jul 10 00:19:09.591932 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.591882 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk"] Jul 10 00:19:09.594934 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:19:09.594911 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a93bcd7_0c6b_45bc_ac54_b7c6873f7748.slice/crio-f7e7522b62afa454fbae15a447736534a2e005bdc35edab1081e9ff3390aab06 WatchSource:0}: Error finding container f7e7522b62afa454fbae15a447736534a2e005bdc35edab1081e9ff3390aab06: Status 404 returned error can't find the container with id f7e7522b62afa454fbae15a447736534a2e005bdc35edab1081e9ff3390aab06 Jul 10 00:19:09.911754 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.911670 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b654b8d4-rwmhc" event={"ID":"00ae2e20-33bc-4ed1-bfab-ef18b5bef277","Type":"ContainerStarted","Data":"436f262b855d21d0e446ea0e34b44595c81d09af991899dc0baabae3cfd69f7d"} Jul 10 00:19:09.912834 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:09.912804 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" event={"ID":"4a93bcd7-0c6b-45bc-ac54-b7c6873f7748","Type":"ContainerStarted","Data":"f7e7522b62afa454fbae15a447736534a2e005bdc35edab1081e9ff3390aab06"} Jul 10 00:19:12.921642 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:12.921556 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b654b8d4-rwmhc" event={"ID":"00ae2e20-33bc-4ed1-bfab-ef18b5bef277","Type":"ContainerStarted","Data":"3a8e22001479dcce4aab432282320f327de3b90e017119e7c87e2113725f9ddb"} Jul 10 00:19:12.936752 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:12.936707 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b654b8d4-rwmhc" podStartSLOduration=1.9329678399999999 podStartE2EDuration="4.936692367s" totalImagesPullingTime="3.003724527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:19:12.935826402 +0000 UTC m=+85.868910712" watchObservedRunningTime="2026-07-10 00:19:12.936692367 +0000 UTC m=+85.869776654" Jul 10 00:19:14.929514 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:14.929485 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" event={"ID":"4a93bcd7-0c6b-45bc-ac54-b7c6873f7748","Type":"ContainerStarted","Data":"adc0c70d761505323a0c7d1c1b7bd86dfb2b7c9f9a47249c732a5b5138633475"} Jul 10 00:19:14.929914 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:14.929523 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" event={"ID":"4a93bcd7-0c6b-45bc-ac54-b7c6873f7748","Type":"ContainerStarted","Data":"1fc3afec75c70564be513f23571fe3cbde9bf14b4fdd0c6779089c62111408e7"} Jul 10 00:19:14.929914 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:14.929534 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" event={"ID":"4a93bcd7-0c6b-45bc-ac54-b7c6873f7748","Type":"ContainerStarted","Data":"d77cb69356736666b939cc5477350841f67f6a411edba361434d83d9747ab3d9"} Jul 10 00:19:14.947021 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:14.946981 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" podStartSLOduration=2.12512612 podStartE2EDuration="6.946967782s" totalImagesPullingTime="4.821841662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:19:14.945916082 +0000 UTC m=+87.879000404" watchObservedRunningTime="2026-07-10 00:19:14.946967782 +0000 UTC m=+87.880052090" Jul 10 00:19:26.779774 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:26.779740 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls\") pod \"dns-default-ds66q\" (UID: \"0654b471-0fff-4b5c-a6fc-d3e972163d20\") " pod="openshift-dns/dns-default-ds66q" Jul 10 00:19:26.780211 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:19:26.779856 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:19:26.780211 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:19:26.779920 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls podName:0654b471-0fff-4b5c-a6fc-d3e972163d20 nodeName:}" failed. No retries permitted until 2026-07-10 00:20:30.779906341 +0000 UTC m=+163.712990628 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls") pod "dns-default-ds66q" (UID: "0654b471-0fff-4b5c-a6fc-d3e972163d20") : secret "dns-default-metrics-tls" not found Jul 10 00:19:30.892889 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:30.892861 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wrzt4" Jul 10 00:19:57.380010 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:19:57.379963 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs\") pod \"network-metrics-daemon-ms9mf\" (UID: \"376630cb-0d38-46f3-a311-7cf9af6f2d10\") " pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:19:57.380442 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:19:57.380107 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 00:19:57.380442 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:19:57.380195 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs podName:376630cb-0d38-46f3-a311-7cf9af6f2d10 nodeName:}" failed. No retries permitted until 2026-07-10 00:21:59.380177745 +0000 UTC m=+252.313262032 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs") pod "network-metrics-daemon-ms9mf" (UID: "376630cb-0d38-46f3-a311-7cf9af6f2d10") : secret "metrics-daemon-secret" not found Jul 10 00:20:25.935620 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:20:25.935569 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-ds66q" podUID="0654b471-0fff-4b5c-a6fc-d3e972163d20" Jul 10 00:20:26.121746 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:20:26.121719 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ds66q" Jul 10 00:20:26.654937 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:20:26.654900 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-ms9mf" podUID="376630cb-0d38-46f3-a311-7cf9af6f2d10" Jul 10 00:20:30.808098 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:20:30.808051 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls\") pod \"dns-default-ds66q\" (UID: \"0654b471-0fff-4b5c-a6fc-d3e972163d20\") " pod="openshift-dns/dns-default-ds66q" Jul 10 00:20:30.808506 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:20:30.808190 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:20:30.808506 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:20:30.808246 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls podName:0654b471-0fff-4b5c-a6fc-d3e972163d20 nodeName:}" failed. No retries permitted until 2026-07-10 00:22:32.808231629 +0000 UTC m=+285.741315917 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls") pod "dns-default-ds66q" (UID: "0654b471-0fff-4b5c-a6fc-d3e972163d20") : secret "dns-default-metrics-tls" not found Jul 10 00:20:38.644092 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:20:38.644062 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:21:13.244521 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:13.244488 2559 generic.go:356] "Generic (PLEG): container finished" podID="00ae2e20-33bc-4ed1-bfab-ef18b5bef277" containerID="3a8e22001479dcce4aab432282320f327de3b90e017119e7c87e2113725f9ddb" exitCode=255 Jul 10 00:21:13.244913 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:13.244536 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b654b8d4-rwmhc" event={"ID":"00ae2e20-33bc-4ed1-bfab-ef18b5bef277","Type":"ContainerDied","Data":"3a8e22001479dcce4aab432282320f327de3b90e017119e7c87e2113725f9ddb"} Jul 10 00:21:13.244913 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:13.244811 2559 scope.go:117] "RemoveContainer" containerID="3a8e22001479dcce4aab432282320f327de3b90e017119e7c87e2113725f9ddb" Jul 10 00:21:14.249083 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:14.249054 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-75b654b8d4-rwmhc" event={"ID":"00ae2e20-33bc-4ed1-bfab-ef18b5bef277","Type":"ContainerStarted","Data":"e4e46af2010f4049aedfe9af172d9ace86001bbe65fe4316db84d086311345ec"} Jul 10 00:21:32.208409 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:32.208374 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-khzbs"] Jul 10 00:21:32.211413 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:32.211397 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-khzbs" Jul 10 00:21:32.213515 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:32.213495 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 00:21:32.213632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:32.213549 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 00:21:32.213632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:32.213580 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6fk5l\"" Jul 10 00:21:32.214207 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:32.214183 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 00:21:32.214285 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:32.214229 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 00:21:32.220197 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:32.220177 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-khzbs"] Jul 10 00:21:32.301339 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:32.301309 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/373f4170-5007-4ba6-8bc9-f109677f4f49-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-khzbs\" (UID: \"373f4170-5007-4ba6-8bc9-f109677f4f49\") " pod="openshift-insights/insights-runtime-extractor-khzbs" Jul 10 00:21:32.301498 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:32.301367 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg6tg\" (UniqueName: \"kubernetes.io/projected/373f4170-5007-4ba6-8bc9-f109677f4f49-kube-api-access-xg6tg\") pod \"insights-runtime-extractor-khzbs\" (UID: \"373f4170-5007-4ba6-8bc9-f109677f4f49\") " pod="openshift-insights/insights-runtime-extractor-khzbs" Jul 10 00:21:32.301498 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:32.301458 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/373f4170-5007-4ba6-8bc9-f109677f4f49-crio-socket\") pod \"insights-runtime-extractor-khzbs\" (UID: \"373f4170-5007-4ba6-8bc9-f109677f4f49\") " pod="openshift-insights/insights-runtime-extractor-khzbs" Jul 10 00:21:32.301498 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:32.301487 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/373f4170-5007-4ba6-8bc9-f109677f4f49-data-volume\") pod \"insights-runtime-extractor-khzbs\" (UID: \"373f4170-5007-4ba6-8bc9-f109677f4f49\") " pod="openshift-insights/insights-runtime-extractor-khzbs" Jul 10 00:21:32.301642 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:32.301520 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/373f4170-5007-4ba6-8bc9-f109677f4f49-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-khzbs\" (UID: \"373f4170-5007-4ba6-8bc9-f109677f4f49\") " pod="openshift-insights/insights-runtime-extractor-khzbs" Jul 10 00:21:32.402295 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:32.402257 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/373f4170-5007-4ba6-8bc9-f109677f4f49-crio-socket\") pod \"insights-runtime-extractor-khzbs\" (UID: \"373f4170-5007-4ba6-8bc9-f109677f4f49\") " pod="openshift-insights/insights-runtime-extractor-khzbs" Jul 10 00:21:32.402295 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:32.402297 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/373f4170-5007-4ba6-8bc9-f109677f4f49-data-volume\") pod \"insights-runtime-extractor-khzbs\" (UID: \"373f4170-5007-4ba6-8bc9-f109677f4f49\") " pod="openshift-insights/insights-runtime-extractor-khzbs" Jul 10 00:21:32.402496 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:32.402316 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/373f4170-5007-4ba6-8bc9-f109677f4f49-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-khzbs\" (UID: \"373f4170-5007-4ba6-8bc9-f109677f4f49\") " pod="openshift-insights/insights-runtime-extractor-khzbs" Jul 10 00:21:32.402496 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:32.402372 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/373f4170-5007-4ba6-8bc9-f109677f4f49-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-khzbs\" (UID: \"373f4170-5007-4ba6-8bc9-f109677f4f49\") " pod="openshift-insights/insights-runtime-extractor-khzbs" Jul 10 00:21:32.402496 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:32.402379 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/373f4170-5007-4ba6-8bc9-f109677f4f49-crio-socket\") pod \"insights-runtime-extractor-khzbs\" (UID: \"373f4170-5007-4ba6-8bc9-f109677f4f49\") " pod="openshift-insights/insights-runtime-extractor-khzbs" Jul 10 00:21:32.402496 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:32.402421 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg6tg\" (UniqueName: \"kubernetes.io/projected/373f4170-5007-4ba6-8bc9-f109677f4f49-kube-api-access-xg6tg\") pod \"insights-runtime-extractor-khzbs\" (UID: \"373f4170-5007-4ba6-8bc9-f109677f4f49\") " pod="openshift-insights/insights-runtime-extractor-khzbs" Jul 10 00:21:32.402496 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:21:32.402477 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 00:21:32.402699 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:21:32.402521 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 00:21:32.402699 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:21:32.402540 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/373f4170-5007-4ba6-8bc9-f109677f4f49-insights-runtime-extractor-tls podName:373f4170-5007-4ba6-8bc9-f109677f4f49 nodeName:}" failed. No retries permitted until 2026-07-10 00:21:32.902524437 +0000 UTC m=+225.835608724 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/373f4170-5007-4ba6-8bc9-f109677f4f49-insights-runtime-extractor-tls") pod "insights-runtime-extractor-khzbs" (UID: "373f4170-5007-4ba6-8bc9-f109677f4f49") : secret "insights-runtime-extractor-tls" not found Jul 10 00:21:32.402699 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:21:32.402577 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/373f4170-5007-4ba6-8bc9-f109677f4f49-kube-rbac-proxy-cm podName:373f4170-5007-4ba6-8bc9-f109677f4f49 nodeName:}" failed. No retries permitted until 2026-07-10 00:21:32.902560689 +0000 UTC m=+225.835644979 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/373f4170-5007-4ba6-8bc9-f109677f4f49-kube-rbac-proxy-cm") pod "insights-runtime-extractor-khzbs" (UID: "373f4170-5007-4ba6-8bc9-f109677f4f49") : configmap "kube-rbac-proxy" not found Jul 10 00:21:32.402699 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:32.402643 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/373f4170-5007-4ba6-8bc9-f109677f4f49-data-volume\") pod \"insights-runtime-extractor-khzbs\" (UID: \"373f4170-5007-4ba6-8bc9-f109677f4f49\") " pod="openshift-insights/insights-runtime-extractor-khzbs" Jul 10 00:21:32.410427 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:32.410407 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg6tg\" (UniqueName: \"kubernetes.io/projected/373f4170-5007-4ba6-8bc9-f109677f4f49-kube-api-access-xg6tg\") pod \"insights-runtime-extractor-khzbs\" (UID: \"373f4170-5007-4ba6-8bc9-f109677f4f49\") " pod="openshift-insights/insights-runtime-extractor-khzbs" Jul 10 00:21:32.906583 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:32.906554 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/373f4170-5007-4ba6-8bc9-f109677f4f49-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-khzbs\" (UID: \"373f4170-5007-4ba6-8bc9-f109677f4f49\") " pod="openshift-insights/insights-runtime-extractor-khzbs" Jul 10 00:21:32.906751 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:32.906603 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/373f4170-5007-4ba6-8bc9-f109677f4f49-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-khzbs\" (UID: \"373f4170-5007-4ba6-8bc9-f109677f4f49\") " pod="openshift-insights/insights-runtime-extractor-khzbs" Jul 10 00:21:32.906751 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:21:32.906683 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 00:21:32.906751 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:21:32.906709 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 00:21:32.906751 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:21:32.906741 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/373f4170-5007-4ba6-8bc9-f109677f4f49-kube-rbac-proxy-cm podName:373f4170-5007-4ba6-8bc9-f109677f4f49 nodeName:}" failed. No retries permitted until 2026-07-10 00:21:33.906726268 +0000 UTC m=+226.839810554 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/373f4170-5007-4ba6-8bc9-f109677f4f49-kube-rbac-proxy-cm") pod "insights-runtime-extractor-khzbs" (UID: "373f4170-5007-4ba6-8bc9-f109677f4f49") : configmap "kube-rbac-proxy" not found Jul 10 00:21:32.906941 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:21:32.906767 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/373f4170-5007-4ba6-8bc9-f109677f4f49-insights-runtime-extractor-tls podName:373f4170-5007-4ba6-8bc9-f109677f4f49 nodeName:}" failed. No retries permitted until 2026-07-10 00:21:33.906752951 +0000 UTC m=+226.839837237 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/373f4170-5007-4ba6-8bc9-f109677f4f49-insights-runtime-extractor-tls") pod "insights-runtime-extractor-khzbs" (UID: "373f4170-5007-4ba6-8bc9-f109677f4f49") : secret "insights-runtime-extractor-tls" not found Jul 10 00:21:33.914166 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:33.914131 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/373f4170-5007-4ba6-8bc9-f109677f4f49-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-khzbs\" (UID: \"373f4170-5007-4ba6-8bc9-f109677f4f49\") " pod="openshift-insights/insights-runtime-extractor-khzbs" Jul 10 00:21:33.914514 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:33.914186 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/373f4170-5007-4ba6-8bc9-f109677f4f49-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-khzbs\" (UID: \"373f4170-5007-4ba6-8bc9-f109677f4f49\") " pod="openshift-insights/insights-runtime-extractor-khzbs" Jul 10 00:21:33.914514 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:21:33.914247 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 00:21:33.914514 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:21:33.914261 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 00:21:33.914514 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:21:33.914301 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/373f4170-5007-4ba6-8bc9-f109677f4f49-insights-runtime-extractor-tls podName:373f4170-5007-4ba6-8bc9-f109677f4f49 nodeName:}" failed. No retries permitted until 2026-07-10 00:21:35.91428815 +0000 UTC m=+228.847372436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/373f4170-5007-4ba6-8bc9-f109677f4f49-insights-runtime-extractor-tls") pod "insights-runtime-extractor-khzbs" (UID: "373f4170-5007-4ba6-8bc9-f109677f4f49") : secret "insights-runtime-extractor-tls" not found Jul 10 00:21:33.914514 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:21:33.914315 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/373f4170-5007-4ba6-8bc9-f109677f4f49-kube-rbac-proxy-cm podName:373f4170-5007-4ba6-8bc9-f109677f4f49 nodeName:}" failed. No retries permitted until 2026-07-10 00:21:35.914308595 +0000 UTC m=+228.847392881 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/373f4170-5007-4ba6-8bc9-f109677f4f49-kube-rbac-proxy-cm") pod "insights-runtime-extractor-khzbs" (UID: "373f4170-5007-4ba6-8bc9-f109677f4f49") : configmap "kube-rbac-proxy" not found Jul 10 00:21:33.982266 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:33.982244 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gzg2p_733f8b73-939c-414d-9d90-7791508e5b84/dns-node-resolver/0.log" Jul 10 00:21:34.343437 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:34.343410 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-7nz67"] Jul 10 00:21:34.346290 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:34.346276 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-7nz67" Jul 10 00:21:34.348196 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:34.348176 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 10 00:21:34.348889 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:34.348867 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-x7vlq\"" Jul 10 00:21:34.348970 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:34.348868 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 10 00:21:34.348970 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:34.348881 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 10 00:21:34.349073 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:34.348908 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 10 00:21:34.354148 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:34.354111 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-7nz67"] Jul 10 00:21:34.417956 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:34.417931 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bb76a854-340b-4632-8afa-abe0eaf0d90f-signing-key\") pod \"service-ca-79f68b9f4f-7nz67\" (UID: \"bb76a854-340b-4632-8afa-abe0eaf0d90f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-7nz67" Jul 10 00:21:34.418046 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:34.417959 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p294\" (UniqueName: \"kubernetes.io/projected/bb76a854-340b-4632-8afa-abe0eaf0d90f-kube-api-access-2p294\") pod \"service-ca-79f68b9f4f-7nz67\" (UID: \"bb76a854-340b-4632-8afa-abe0eaf0d90f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-7nz67" Jul 10 00:21:34.418046 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:34.418023 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bb76a854-340b-4632-8afa-abe0eaf0d90f-signing-cabundle\") pod \"service-ca-79f68b9f4f-7nz67\" (UID: \"bb76a854-340b-4632-8afa-abe0eaf0d90f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-7nz67" Jul 10 00:21:34.520204 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:34.520174 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bb76a854-340b-4632-8afa-abe0eaf0d90f-signing-cabundle\") pod \"service-ca-79f68b9f4f-7nz67\" (UID: \"bb76a854-340b-4632-8afa-abe0eaf0d90f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-7nz67" Jul 10 00:21:34.520307 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:34.520237 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bb76a854-340b-4632-8afa-abe0eaf0d90f-signing-key\") pod \"service-ca-79f68b9f4f-7nz67\" (UID: \"bb76a854-340b-4632-8afa-abe0eaf0d90f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-7nz67" Jul 10 00:21:34.520307 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:34.520263 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p294\" (UniqueName: \"kubernetes.io/projected/bb76a854-340b-4632-8afa-abe0eaf0d90f-kube-api-access-2p294\") pod \"service-ca-79f68b9f4f-7nz67\" (UID: \"bb76a854-340b-4632-8afa-abe0eaf0d90f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-7nz67" Jul 10 00:21:34.521365 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:34.521341 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/bb76a854-340b-4632-8afa-abe0eaf0d90f-signing-cabundle\") pod \"service-ca-79f68b9f4f-7nz67\" (UID: \"bb76a854-340b-4632-8afa-abe0eaf0d90f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-7nz67" Jul 10 00:21:34.523954 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:34.523926 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/bb76a854-340b-4632-8afa-abe0eaf0d90f-signing-key\") pod \"service-ca-79f68b9f4f-7nz67\" (UID: \"bb76a854-340b-4632-8afa-abe0eaf0d90f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-7nz67" Jul 10 00:21:34.527418 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:34.527400 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p294\" (UniqueName: \"kubernetes.io/projected/bb76a854-340b-4632-8afa-abe0eaf0d90f-kube-api-access-2p294\") pod \"service-ca-79f68b9f4f-7nz67\" (UID: \"bb76a854-340b-4632-8afa-abe0eaf0d90f\") " pod="openshift-service-ca/service-ca-79f68b9f4f-7nz67" Jul 10 00:21:34.655449 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:34.655399 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-7nz67" Jul 10 00:21:34.769676 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:34.769648 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-7nz67"] Jul 10 00:21:34.772474 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:21:34.772440 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb76a854_340b_4632_8afa_abe0eaf0d90f.slice/crio-6ea2bf8140934e2f8ddc5dc575b94b7b77af21a86fbf0770e25b4bb0de8dfed8 WatchSource:0}: Error finding container 6ea2bf8140934e2f8ddc5dc575b94b7b77af21a86fbf0770e25b4bb0de8dfed8: Status 404 returned error can't find the container with id 6ea2bf8140934e2f8ddc5dc575b94b7b77af21a86fbf0770e25b4bb0de8dfed8 Jul 10 00:21:35.181647 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:35.181625 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jdrfw_613ac839-60fd-4f20-a127-a072d8a574bb/node-ca/0.log" Jul 10 00:21:35.307838 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:35.307810 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-7nz67" event={"ID":"bb76a854-340b-4632-8afa-abe0eaf0d90f","Type":"ContainerStarted","Data":"6ea2bf8140934e2f8ddc5dc575b94b7b77af21a86fbf0770e25b4bb0de8dfed8"} Jul 10 00:21:35.929818 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:35.929775 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/373f4170-5007-4ba6-8bc9-f109677f4f49-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-khzbs\" (UID: \"373f4170-5007-4ba6-8bc9-f109677f4f49\") " pod="openshift-insights/insights-runtime-extractor-khzbs" Jul 10 00:21:35.930007 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:35.929860 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/373f4170-5007-4ba6-8bc9-f109677f4f49-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-khzbs\" (UID: \"373f4170-5007-4ba6-8bc9-f109677f4f49\") " pod="openshift-insights/insights-runtime-extractor-khzbs" Jul 10 00:21:35.930007 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:21:35.929903 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 00:21:35.930007 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:21:35.929967 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 00:21:35.930007 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:21:35.929988 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/373f4170-5007-4ba6-8bc9-f109677f4f49-kube-rbac-proxy-cm podName:373f4170-5007-4ba6-8bc9-f109677f4f49 nodeName:}" failed. No retries permitted until 2026-07-10 00:21:39.929967409 +0000 UTC m=+232.863051698 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/373f4170-5007-4ba6-8bc9-f109677f4f49-kube-rbac-proxy-cm") pod "insights-runtime-extractor-khzbs" (UID: "373f4170-5007-4ba6-8bc9-f109677f4f49") : configmap "kube-rbac-proxy" not found Jul 10 00:21:35.930190 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:21:35.930020 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/373f4170-5007-4ba6-8bc9-f109677f4f49-insights-runtime-extractor-tls podName:373f4170-5007-4ba6-8bc9-f109677f4f49 nodeName:}" failed. No retries permitted until 2026-07-10 00:21:39.930007897 +0000 UTC m=+232.863092184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/373f4170-5007-4ba6-8bc9-f109677f4f49-insights-runtime-extractor-tls") pod "insights-runtime-extractor-khzbs" (UID: "373f4170-5007-4ba6-8bc9-f109677f4f49") : secret "insights-runtime-extractor-tls" not found Jul 10 00:21:36.181692 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:36.181610 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-q8pzq_5918e467-096f-431c-a997-d13e6ddd04e0/serve-healthcheck-canary/0.log" Jul 10 00:21:37.314835 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:37.314795 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-7nz67" event={"ID":"bb76a854-340b-4632-8afa-abe0eaf0d90f","Type":"ContainerStarted","Data":"6673312f241849bfd3b6921ad7544a9579701ac330425eec14a97b6b9721b286"} Jul 10 00:21:37.329044 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:37.329000 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-7nz67" podStartSLOduration=1.254448354 podStartE2EDuration="3.328984293s" totalImagesPullingTime="2.074535939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:21:37.328386641 +0000 UTC m=+230.261470971" watchObservedRunningTime="2026-07-10 00:21:37.328984293 +0000 UTC m=+230.262068629" Jul 10 00:21:39.958055 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:39.958022 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/373f4170-5007-4ba6-8bc9-f109677f4f49-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-khzbs\" (UID: \"373f4170-5007-4ba6-8bc9-f109677f4f49\") " pod="openshift-insights/insights-runtime-extractor-khzbs" Jul 10 00:21:39.958440 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:39.958085 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/373f4170-5007-4ba6-8bc9-f109677f4f49-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-khzbs\" (UID: \"373f4170-5007-4ba6-8bc9-f109677f4f49\") " pod="openshift-insights/insights-runtime-extractor-khzbs" Jul 10 00:21:39.958440 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:21:39.958166 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 00:21:39.958440 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:21:39.958179 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 00:21:39.958440 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:21:39.958228 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/373f4170-5007-4ba6-8bc9-f109677f4f49-kube-rbac-proxy-cm podName:373f4170-5007-4ba6-8bc9-f109677f4f49 nodeName:}" failed. No retries permitted until 2026-07-10 00:21:47.958212213 +0000 UTC m=+240.891296500 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/373f4170-5007-4ba6-8bc9-f109677f4f49-kube-rbac-proxy-cm") pod "insights-runtime-extractor-khzbs" (UID: "373f4170-5007-4ba6-8bc9-f109677f4f49") : configmap "kube-rbac-proxy" not found Jul 10 00:21:39.958440 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:21:39.958244 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/373f4170-5007-4ba6-8bc9-f109677f4f49-insights-runtime-extractor-tls podName:373f4170-5007-4ba6-8bc9-f109677f4f49 nodeName:}" failed. No retries permitted until 2026-07-10 00:21:47.958237333 +0000 UTC m=+240.891321620 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/373f4170-5007-4ba6-8bc9-f109677f4f49-insights-runtime-extractor-tls") pod "insights-runtime-extractor-khzbs" (UID: "373f4170-5007-4ba6-8bc9-f109677f4f49") : secret "insights-runtime-extractor-tls" not found Jul 10 00:21:48.017463 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:48.017426 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/373f4170-5007-4ba6-8bc9-f109677f4f49-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-khzbs\" (UID: \"373f4170-5007-4ba6-8bc9-f109677f4f49\") " pod="openshift-insights/insights-runtime-extractor-khzbs" Jul 10 00:21:48.017918 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:48.017495 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/373f4170-5007-4ba6-8bc9-f109677f4f49-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-khzbs\" (UID: \"373f4170-5007-4ba6-8bc9-f109677f4f49\") " pod="openshift-insights/insights-runtime-extractor-khzbs" Jul 10 00:21:48.017963 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:48.017936 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/373f4170-5007-4ba6-8bc9-f109677f4f49-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-khzbs\" (UID: \"373f4170-5007-4ba6-8bc9-f109677f4f49\") " pod="openshift-insights/insights-runtime-extractor-khzbs" Jul 10 00:21:48.019690 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:48.019672 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/373f4170-5007-4ba6-8bc9-f109677f4f49-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-khzbs\" (UID: \"373f4170-5007-4ba6-8bc9-f109677f4f49\") " pod="openshift-insights/insights-runtime-extractor-khzbs" Jul 10 00:21:48.121218 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:48.121187 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6fk5l\"" Jul 10 00:21:48.129940 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:48.129923 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-khzbs" Jul 10 00:21:48.248267 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:48.248236 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-khzbs"] Jul 10 00:21:48.250954 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:21:48.250923 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373f4170_5007_4ba6_8bc9_f109677f4f49.slice/crio-404fe2ebf9447b6690dc307c5bb32a798821e129ee7f02c893d0df491cb83e72 WatchSource:0}: Error finding container 404fe2ebf9447b6690dc307c5bb32a798821e129ee7f02c893d0df491cb83e72: Status 404 returned error can't find the container with id 404fe2ebf9447b6690dc307c5bb32a798821e129ee7f02c893d0df491cb83e72 Jul 10 00:21:48.345918 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:48.345835 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-khzbs" event={"ID":"373f4170-5007-4ba6-8bc9-f109677f4f49","Type":"ContainerStarted","Data":"404fe2ebf9447b6690dc307c5bb32a798821e129ee7f02c893d0df491cb83e72"} Jul 10 00:21:52.377004 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:52.376964 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-khzbs" event={"ID":"373f4170-5007-4ba6-8bc9-f109677f4f49","Type":"ContainerStarted","Data":"2ba7b9b8e9c609bed24d1becb5985f4d051e11116c8537d7d226e5092bad54cf"} Jul 10 00:21:53.382682 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:53.382646 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-khzbs" event={"ID":"373f4170-5007-4ba6-8bc9-f109677f4f49","Type":"ContainerStarted","Data":"96925fb3215b812996d6f2b859765ffec2874655bf9eff517588ce3b6932f43d"} Jul 10 00:21:55.390778 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:55.390735 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-khzbs" event={"ID":"373f4170-5007-4ba6-8bc9-f109677f4f49","Type":"ContainerStarted","Data":"936b68fd6c426d23295da6f10e4c63bc48c335fcce02777b821286024a8072b0"} Jul 10 00:21:55.412201 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:55.412081 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-khzbs" podStartSLOduration=16.646650448 podStartE2EDuration="23.412066997s" totalImagesPullingTime="6.765416549s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:21:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:21:55.411006701 +0000 UTC m=+248.344091024" watchObservedRunningTime="2026-07-10 00:21:55.412066997 +0000 UTC m=+248.345151305" Jul 10 00:21:59.260316 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:59.260265 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" podUID="4a93bcd7-0c6b-45bc-ac54-b7c6873f7748" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 10 00:21:59.402505 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:59.402471 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs\") pod \"network-metrics-daemon-ms9mf\" (UID: \"376630cb-0d38-46f3-a311-7cf9af6f2d10\") " pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:21:59.404852 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:59.404831 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/376630cb-0d38-46f3-a311-7cf9af6f2d10-metrics-certs\") pod \"network-metrics-daemon-ms9mf\" (UID: \"376630cb-0d38-46f3-a311-7cf9af6f2d10\") " pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:21:59.646795 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:59.646715 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l7wp6\"" Jul 10 00:21:59.655390 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:59.655370 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9mf" Jul 10 00:21:59.780825 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:21:59.780784 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ms9mf"] Jul 10 00:21:59.783964 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:21:59.783941 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod376630cb_0d38_46f3_a311_7cf9af6f2d10.slice/crio-43f88f9ae062b452584d79f72f803ddbfb3cd79e573d14a1e4287bb525996ed7 WatchSource:0}: Error finding container 43f88f9ae062b452584d79f72f803ddbfb3cd79e573d14a1e4287bb525996ed7: Status 404 returned error can't find the container with id 43f88f9ae062b452584d79f72f803ddbfb3cd79e573d14a1e4287bb525996ed7 Jul 10 00:22:00.407040 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:00.406999 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ms9mf" event={"ID":"376630cb-0d38-46f3-a311-7cf9af6f2d10","Type":"ContainerStarted","Data":"43f88f9ae062b452584d79f72f803ddbfb3cd79e573d14a1e4287bb525996ed7"} Jul 10 00:22:01.412385 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:01.412345 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ms9mf" event={"ID":"376630cb-0d38-46f3-a311-7cf9af6f2d10","Type":"ContainerStarted","Data":"d2b4739b2a52f3fae9f7ee88fca74f0898e3f373209233b1ecf15b67cf12c686"} Jul 10 00:22:01.412730 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:01.412408 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ms9mf" event={"ID":"376630cb-0d38-46f3-a311-7cf9af6f2d10","Type":"ContainerStarted","Data":"01db1634392f1868adb0a9a1c6b0741dc7ed3f14b08d7b4d1d79692dad0824d6"} Jul 10 00:22:01.427265 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:01.427221 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ms9mf" podStartSLOduration=253.35724823 podStartE2EDuration="4m14.427205396s" totalImagesPullingTime="1.069957166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:22:01.426014955 +0000 UTC m=+254.359099263" watchObservedRunningTime="2026-07-10 00:22:01.427205396 +0000 UTC m=+254.360289705" Jul 10 00:22:09.259838 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:09.259759 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" podUID="4a93bcd7-0c6b-45bc-ac54-b7c6873f7748" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 10 00:22:09.873878 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:09.873844 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4szx5"] Jul 10 00:22:09.878481 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:09.878461 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:09.880869 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:09.880843 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 00:22:09.880988 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:09.880887 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 00:22:09.880988 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:09.880843 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 00:22:09.881348 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:09.881330 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 00:22:09.881648 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:09.881631 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 00:22:09.881758 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:09.881740 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f7qsh\"" Jul 10 00:22:09.881895 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:09.881880 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 00:22:10.072372 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.072327 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04dd1d7f-86d8-49a0-89aa-e15bc09ac11f-metrics-client-ca\") pod \"node-exporter-4szx5\" (UID: \"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f\") " pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.072542 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.072374 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2tvr\" (UniqueName: \"kubernetes.io/projected/04dd1d7f-86d8-49a0-89aa-e15bc09ac11f-kube-api-access-t2tvr\") pod \"node-exporter-4szx5\" (UID: \"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f\") " pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.072542 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.072408 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/04dd1d7f-86d8-49a0-89aa-e15bc09ac11f-node-exporter-tls\") pod \"node-exporter-4szx5\" (UID: \"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f\") " pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.072542 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.072435 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/04dd1d7f-86d8-49a0-89aa-e15bc09ac11f-node-exporter-textfile\") pod \"node-exporter-4szx5\" (UID: \"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f\") " pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.072542 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.072450 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/04dd1d7f-86d8-49a0-89aa-e15bc09ac11f-root\") pod \"node-exporter-4szx5\" (UID: \"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f\") " pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.072542 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.072479 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04dd1d7f-86d8-49a0-89aa-e15bc09ac11f-sys\") pod \"node-exporter-4szx5\" (UID: \"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f\") " pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.072542 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.072505 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/04dd1d7f-86d8-49a0-89aa-e15bc09ac11f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4szx5\" (UID: \"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f\") " pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.072859 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.072572 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/04dd1d7f-86d8-49a0-89aa-e15bc09ac11f-node-exporter-accelerators-collector-config\") pod \"node-exporter-4szx5\" (UID: \"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f\") " pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.072859 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.072679 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/04dd1d7f-86d8-49a0-89aa-e15bc09ac11f-node-exporter-wtmp\") pod \"node-exporter-4szx5\" (UID: \"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f\") " pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.173292 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.173216 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/04dd1d7f-86d8-49a0-89aa-e15bc09ac11f-node-exporter-accelerators-collector-config\") pod \"node-exporter-4szx5\" (UID: \"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f\") " pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.173292 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.173277 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/04dd1d7f-86d8-49a0-89aa-e15bc09ac11f-node-exporter-wtmp\") pod \"node-exporter-4szx5\" (UID: \"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f\") " pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.173469 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.173313 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04dd1d7f-86d8-49a0-89aa-e15bc09ac11f-metrics-client-ca\") pod \"node-exporter-4szx5\" (UID: \"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f\") " pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.173469 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.173330 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2tvr\" (UniqueName: \"kubernetes.io/projected/04dd1d7f-86d8-49a0-89aa-e15bc09ac11f-kube-api-access-t2tvr\") pod \"node-exporter-4szx5\" (UID: \"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f\") " pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.173469 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.173355 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/04dd1d7f-86d8-49a0-89aa-e15bc09ac11f-node-exporter-tls\") pod \"node-exporter-4szx5\" (UID: \"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f\") " pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.173469 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.173381 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/04dd1d7f-86d8-49a0-89aa-e15bc09ac11f-node-exporter-textfile\") pod \"node-exporter-4szx5\" (UID: \"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f\") " pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.173469 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.173452 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/04dd1d7f-86d8-49a0-89aa-e15bc09ac11f-root\") pod \"node-exporter-4szx5\" (UID: \"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f\") " pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.173702 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.173481 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04dd1d7f-86d8-49a0-89aa-e15bc09ac11f-sys\") pod \"node-exporter-4szx5\" (UID: \"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f\") " pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.173702 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.173504 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/04dd1d7f-86d8-49a0-89aa-e15bc09ac11f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4szx5\" (UID: \"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f\") " pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.173702 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.173523 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/04dd1d7f-86d8-49a0-89aa-e15bc09ac11f-node-exporter-wtmp\") pod \"node-exporter-4szx5\" (UID: \"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f\") " pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.173702 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.173584 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/04dd1d7f-86d8-49a0-89aa-e15bc09ac11f-root\") pod \"node-exporter-4szx5\" (UID: \"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f\") " pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.173702 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.173629 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/04dd1d7f-86d8-49a0-89aa-e15bc09ac11f-sys\") pod \"node-exporter-4szx5\" (UID: \"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f\") " pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.173702 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.173697 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/04dd1d7f-86d8-49a0-89aa-e15bc09ac11f-node-exporter-textfile\") pod \"node-exporter-4szx5\" (UID: \"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f\") " pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.173978 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.173920 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04dd1d7f-86d8-49a0-89aa-e15bc09ac11f-metrics-client-ca\") pod \"node-exporter-4szx5\" (UID: \"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f\") " pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.174106 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.174082 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/04dd1d7f-86d8-49a0-89aa-e15bc09ac11f-node-exporter-accelerators-collector-config\") pod \"node-exporter-4szx5\" (UID: \"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f\") " pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.175777 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.175745 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/04dd1d7f-86d8-49a0-89aa-e15bc09ac11f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4szx5\" (UID: \"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f\") " pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.175999 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.175982 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/04dd1d7f-86d8-49a0-89aa-e15bc09ac11f-node-exporter-tls\") pod \"node-exporter-4szx5\" (UID: \"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f\") " pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.180846 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.180821 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2tvr\" (UniqueName: \"kubernetes.io/projected/04dd1d7f-86d8-49a0-89aa-e15bc09ac11f-kube-api-access-t2tvr\") pod \"node-exporter-4szx5\" (UID: \"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f\") " pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.189645 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.189628 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4szx5" Jul 10 00:22:10.200552 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:22:10.200529 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04dd1d7f_86d8_49a0_89aa_e15bc09ac11f.slice/crio-f9821c0e879dd2e2f7221a0aa11619f0277a0fe5a4584f34e3fdf332f6f86f56 WatchSource:0}: Error finding container f9821c0e879dd2e2f7221a0aa11619f0277a0fe5a4584f34e3fdf332f6f86f56: Status 404 returned error can't find the container with id f9821c0e879dd2e2f7221a0aa11619f0277a0fe5a4584f34e3fdf332f6f86f56 Jul 10 00:22:10.443291 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.443215 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4szx5" event={"ID":"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f","Type":"ContainerStarted","Data":"f9821c0e879dd2e2f7221a0aa11619f0277a0fe5a4584f34e3fdf332f6f86f56"} Jul 10 00:22:10.891227 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.891192 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 00:22:10.896693 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.896669 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:10.898782 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.898751 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 10 00:22:10.898897 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.898763 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 10 00:22:10.898897 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.898866 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 10 00:22:10.898897 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.898867 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 10 00:22:10.898897 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.898879 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zw4zw\"" Jul 10 00:22:10.899058 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.899035 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 10 00:22:10.899190 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.899175 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 10 00:22:10.899285 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.899267 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 10 00:22:10.899342 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.899324 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 10 00:22:10.905599 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.905580 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 10 00:22:10.911975 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:10.911956 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 00:22:11.080762 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.080720 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7380182e-1b4d-427c-8c17-537c70315c91-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.080907 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.080767 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.080907 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.080830 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7380182e-1b4d-427c-8c17-537c70315c91-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.080907 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.080871 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7380182e-1b4d-427c-8c17-537c70315c91-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.080907 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.080895 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.081030 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.080923 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7380182e-1b4d-427c-8c17-537c70315c91-config-out\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.081030 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.080957 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7380182e-1b4d-427c-8c17-537c70315c91-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.081030 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.080975 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-web-config\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.081030 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.080995 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.081030 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.081023 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.081189 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.081041 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9zkk\" (UniqueName: \"kubernetes.io/projected/7380182e-1b4d-427c-8c17-537c70315c91-kube-api-access-r9zkk\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.081189 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.081068 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.081189 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.081137 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-config-volume\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.181620 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.181539 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7380182e-1b4d-427c-8c17-537c70315c91-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.181620 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.181582 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-web-config\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.181620 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.181615 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.181887 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.181638 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.181887 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.181662 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9zkk\" (UniqueName: \"kubernetes.io/projected/7380182e-1b4d-427c-8c17-537c70315c91-kube-api-access-r9zkk\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.181887 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.181696 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.181887 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.181739 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-config-volume\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.181887 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.181769 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7380182e-1b4d-427c-8c17-537c70315c91-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.181887 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.181797 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.181887 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.181831 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7380182e-1b4d-427c-8c17-537c70315c91-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.181887 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.181873 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7380182e-1b4d-427c-8c17-537c70315c91-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.182296 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.181902 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.182296 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.181931 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7380182e-1b4d-427c-8c17-537c70315c91-config-out\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.182938 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.182456 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7380182e-1b4d-427c-8c17-537c70315c91-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.182938 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.182744 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7380182e-1b4d-427c-8c17-537c70315c91-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.182938 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:22:11.182884 2559 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 10 00:22:11.183217 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:22:11.182961 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-secret-alertmanager-main-tls podName:7380182e-1b4d-427c-8c17-537c70315c91 nodeName:}" failed. No retries permitted until 2026-07-10 00:22:11.682933355 +0000 UTC m=+264.616017642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "7380182e-1b4d-427c-8c17-537c70315c91") : secret "alertmanager-main-tls" not found Jul 10 00:22:11.183758 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.183730 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7380182e-1b4d-427c-8c17-537c70315c91-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.185350 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.185301 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7380182e-1b4d-427c-8c17-537c70315c91-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.185459 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.185379 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-web-config\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.185459 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.185445 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.185602 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.185577 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.186092 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.186047 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-config-volume\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.186719 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.186696 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.186719 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.186716 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7380182e-1b4d-427c-8c17-537c70315c91-config-out\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.187430 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.187405 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.190416 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.190370 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9zkk\" (UniqueName: \"kubernetes.io/projected/7380182e-1b4d-427c-8c17-537c70315c91-kube-api-access-r9zkk\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.686564 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.686532 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.688956 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.688924 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.806383 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.806356 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:22:11.934140 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:11.934103 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 00:22:11.935124 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:22:11.935090 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7380182e_1b4d_427c_8c17_537c70315c91.slice/crio-60f2db231c3f3967c221e52e36f3395c666696de2d44f7cf9d4f6d8b617a3d08 WatchSource:0}: Error finding container 60f2db231c3f3967c221e52e36f3395c666696de2d44f7cf9d4f6d8b617a3d08: Status 404 returned error can't find the container with id 60f2db231c3f3967c221e52e36f3395c666696de2d44f7cf9d4f6d8b617a3d08 Jul 10 00:22:12.450852 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:12.450812 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7380182e-1b4d-427c-8c17-537c70315c91","Type":"ContainerStarted","Data":"60f2db231c3f3967c221e52e36f3395c666696de2d44f7cf9d4f6d8b617a3d08"} Jul 10 00:22:12.452541 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:12.452472 2559 generic.go:356] "Generic (PLEG): container finished" podID="04dd1d7f-86d8-49a0-89aa-e15bc09ac11f" containerID="37a24dadd004f641131ad479b8a19fe83dfb5c283b075831c03ffa1db7903ce7" exitCode=0 Jul 10 00:22:12.452676 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:12.452546 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4szx5" event={"ID":"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f","Type":"ContainerDied","Data":"37a24dadd004f641131ad479b8a19fe83dfb5c283b075831c03ffa1db7903ce7"} Jul 10 00:22:13.458419 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:13.458322 2559 generic.go:356] "Generic (PLEG): container finished" podID="7380182e-1b4d-427c-8c17-537c70315c91" containerID="7310bab68e3901cf14cc25ac4624db1ec4f37f71336c44b8e2b3d81477b186ea" exitCode=0 Jul 10 00:22:13.458840 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:13.458415 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7380182e-1b4d-427c-8c17-537c70315c91","Type":"ContainerDied","Data":"7310bab68e3901cf14cc25ac4624db1ec4f37f71336c44b8e2b3d81477b186ea"} Jul 10 00:22:13.460516 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:13.460497 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4szx5" event={"ID":"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f","Type":"ContainerStarted","Data":"094d56795c84af8af235e309f99a20364a191c61b370bce40dd5b4c8d8c4fe9d"} Jul 10 00:22:13.460612 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:13.460543 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4szx5" event={"ID":"04dd1d7f-86d8-49a0-89aa-e15bc09ac11f","Type":"ContainerStarted","Data":"a71f95958e975f05c9d448343870bbce5fc0ed4d2bcfb6e3824186b724743e0b"} Jul 10 00:22:13.497207 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:13.497165 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4szx5" podStartSLOduration=3.154477474 podStartE2EDuration="4.497147066s" totalImagesPullingTime="1.280087078s" totalInitContainerRuntime="62.582514ms" isStatefulPod=true podCreationTimestamp="2026-07-10 00:22:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:22:13.496969923 +0000 UTC m=+266.430054246" watchObservedRunningTime="2026-07-10 00:22:13.497147066 +0000 UTC m=+266.430231377" Jul 10 00:22:14.609543 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:14.609510 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-dx5j6"] Jul 10 00:22:14.612649 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:14.612632 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dx5j6" Jul 10 00:22:14.614529 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:14.614510 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-tw4r8\"" Jul 10 00:22:14.614717 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:14.614703 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 10 00:22:14.619975 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:14.619949 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-dx5j6"] Jul 10 00:22:14.708034 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:14.708008 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rggzv\" (UniqueName: \"kubernetes.io/projected/f089d09e-5f9a-4818-b10c-b1926646d89e-kube-api-access-rggzv\") pod \"monitoring-plugin-5f68df7f58-dx5j6\" (UID: \"f089d09e-5f9a-4818-b10c-b1926646d89e\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dx5j6" Jul 10 00:22:14.708144 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:14.708073 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f089d09e-5f9a-4818-b10c-b1926646d89e-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-dx5j6\" (UID: \"f089d09e-5f9a-4818-b10c-b1926646d89e\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dx5j6" Jul 10 00:22:14.808408 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:14.808377 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f089d09e-5f9a-4818-b10c-b1926646d89e-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-dx5j6\" (UID: \"f089d09e-5f9a-4818-b10c-b1926646d89e\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dx5j6" Jul 10 00:22:14.808561 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:14.808437 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rggzv\" (UniqueName: \"kubernetes.io/projected/f089d09e-5f9a-4818-b10c-b1926646d89e-kube-api-access-rggzv\") pod \"monitoring-plugin-5f68df7f58-dx5j6\" (UID: \"f089d09e-5f9a-4818-b10c-b1926646d89e\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dx5j6" Jul 10 00:22:14.811038 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:14.811018 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f089d09e-5f9a-4818-b10c-b1926646d89e-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-dx5j6\" (UID: \"f089d09e-5f9a-4818-b10c-b1926646d89e\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dx5j6" Jul 10 00:22:14.816682 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:14.816655 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rggzv\" (UniqueName: \"kubernetes.io/projected/f089d09e-5f9a-4818-b10c-b1926646d89e-kube-api-access-rggzv\") pod \"monitoring-plugin-5f68df7f58-dx5j6\" (UID: \"f089d09e-5f9a-4818-b10c-b1926646d89e\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dx5j6" Jul 10 00:22:14.922515 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:14.922450 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dx5j6" Jul 10 00:22:15.052203 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:15.052168 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-dx5j6"] Jul 10 00:22:15.283286 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:22:15.283244 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf089d09e_5f9a_4818_b10c_b1926646d89e.slice/crio-2879a4c0e3960cfd4e01ce567fbe43bfede0017e1b29b66472bde5970e00a9e6 WatchSource:0}: Error finding container 2879a4c0e3960cfd4e01ce567fbe43bfede0017e1b29b66472bde5970e00a9e6: Status 404 returned error can't find the container with id 2879a4c0e3960cfd4e01ce567fbe43bfede0017e1b29b66472bde5970e00a9e6 Jul 10 00:22:15.470863 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:15.470816 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7380182e-1b4d-427c-8c17-537c70315c91","Type":"ContainerStarted","Data":"1519faa7931802dee8cecdb128e41e0930e5c280115ae8439a97e4ef23c524f1"} Jul 10 00:22:15.470960 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:15.470906 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7380182e-1b4d-427c-8c17-537c70315c91","Type":"ContainerStarted","Data":"0c47199081163da248d911f32c0881a9e7dae2597dab7f1a286954800b61ad3b"} Jul 10 00:22:15.472267 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:15.472237 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dx5j6" event={"ID":"f089d09e-5f9a-4818-b10c-b1926646d89e","Type":"ContainerStarted","Data":"2879a4c0e3960cfd4e01ce567fbe43bfede0017e1b29b66472bde5970e00a9e6"} Jul 10 00:22:16.479102 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:16.479064 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7380182e-1b4d-427c-8c17-537c70315c91","Type":"ContainerStarted","Data":"a58641b576bb66b2179ebb67a1508a4654a18a37eb3543ce5ae460d94eb37b4e"} Jul 10 00:22:16.479445 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:16.479168 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7380182e-1b4d-427c-8c17-537c70315c91","Type":"ContainerStarted","Data":"7fda78e5bf2b315745a8af8d48695971fa58a8e91bff5a40c00dfa7ea95e1bcd"} Jul 10 00:22:16.479445 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:16.479194 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7380182e-1b4d-427c-8c17-537c70315c91","Type":"ContainerStarted","Data":"9f24ae24e134e07a6fadd9fa4b0277418a05e90c376327e2df1b34bbe7396748"} Jul 10 00:22:17.484997 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:17.484963 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7380182e-1b4d-427c-8c17-537c70315c91","Type":"ContainerStarted","Data":"310b8ffdc94d0f1f0f98cca47884444ca72c09942d71686d70764cf9b72f3514"} Jul 10 00:22:17.486380 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:17.486354 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dx5j6" event={"ID":"f089d09e-5f9a-4818-b10c-b1926646d89e","Type":"ContainerStarted","Data":"fac699dff201477862215251fbd85c06c7f6ff9cb0663ff46d56be00f9e08259"} Jul 10 00:22:17.486608 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:17.486590 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dx5j6" Jul 10 00:22:17.491130 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:17.491100 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dx5j6" Jul 10 00:22:17.509733 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:17.509687 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.141597013 podStartE2EDuration="7.509674276s" totalImagesPullingTime="4.350900239s" totalInitContainerRuntime="17.177024ms" isStatefulPod=false podCreationTimestamp="2026-07-10 00:22:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:22:17.507696012 +0000 UTC m=+270.440780339" watchObservedRunningTime="2026-07-10 00:22:17.509674276 +0000 UTC m=+270.442758585" Jul 10 00:22:17.525195 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:17.525150 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-dx5j6" podStartSLOduration=1.825311653 podStartE2EDuration="3.525135464s" totalImagesPullingTime="1.699823811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:22:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:22:17.52414429 +0000 UTC m=+270.457228629" watchObservedRunningTime="2026-07-10 00:22:17.525135464 +0000 UTC m=+270.458219765" Jul 10 00:22:19.259939 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:19.259902 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" podUID="4a93bcd7-0c6b-45bc-ac54-b7c6873f7748" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 10 00:22:19.260400 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:19.259970 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" Jul 10 00:22:19.260474 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:19.260443 2559 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"adc0c70d761505323a0c7d1c1b7bd86dfb2b7c9f9a47249c732a5b5138633475"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 10 00:22:19.260510 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:19.260495 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" podUID="4a93bcd7-0c6b-45bc-ac54-b7c6873f7748" containerName="service-proxy" containerID="cri-o://adc0c70d761505323a0c7d1c1b7bd86dfb2b7c9f9a47249c732a5b5138633475" gracePeriod=30 Jul 10 00:22:19.496095 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:19.496064 2559 generic.go:356] "Generic (PLEG): container finished" podID="4a93bcd7-0c6b-45bc-ac54-b7c6873f7748" containerID="adc0c70d761505323a0c7d1c1b7bd86dfb2b7c9f9a47249c732a5b5138633475" exitCode=2 Jul 10 00:22:19.496252 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:19.496148 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" event={"ID":"4a93bcd7-0c6b-45bc-ac54-b7c6873f7748","Type":"ContainerDied","Data":"adc0c70d761505323a0c7d1c1b7bd86dfb2b7c9f9a47249c732a5b5138633475"} Jul 10 00:22:19.496252 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:19.496189 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6f85bff9fb-6qmhk" event={"ID":"4a93bcd7-0c6b-45bc-ac54-b7c6873f7748","Type":"ContainerStarted","Data":"6091a8ad8698023a32cd5d144c8a7e3a35d3778540484104dd6fe53c197edc65"} Jul 10 00:22:24.979790 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:24.979760 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-nxhzz"] Jul 10 00:22:24.986457 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:24.986435 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-nxhzz" Jul 10 00:22:24.988493 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:24.988473 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-lgm2z\"" Jul 10 00:22:24.988493 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:24.988478 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 00:22:24.988666 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:24.988512 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 00:22:24.993956 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:24.993934 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-nxhzz"] Jul 10 00:22:25.086271 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:25.086245 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw4gw\" (UniqueName: \"kubernetes.io/projected/f699436c-1006-49b2-8dc4-061253a6b2b2-kube-api-access-mw4gw\") pod \"downloads-68c9cbb497-nxhzz\" (UID: \"f699436c-1006-49b2-8dc4-061253a6b2b2\") " pod="openshift-console/downloads-68c9cbb497-nxhzz" Jul 10 00:22:25.187600 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:25.187572 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw4gw\" (UniqueName: \"kubernetes.io/projected/f699436c-1006-49b2-8dc4-061253a6b2b2-kube-api-access-mw4gw\") pod \"downloads-68c9cbb497-nxhzz\" (UID: \"f699436c-1006-49b2-8dc4-061253a6b2b2\") " pod="openshift-console/downloads-68c9cbb497-nxhzz" Jul 10 00:22:25.194765 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:25.194743 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw4gw\" (UniqueName: \"kubernetes.io/projected/f699436c-1006-49b2-8dc4-061253a6b2b2-kube-api-access-mw4gw\") pod \"downloads-68c9cbb497-nxhzz\" (UID: \"f699436c-1006-49b2-8dc4-061253a6b2b2\") " pod="openshift-console/downloads-68c9cbb497-nxhzz" Jul 10 00:22:25.295860 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:25.295836 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-nxhzz" Jul 10 00:22:25.415154 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:25.415107 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-nxhzz"] Jul 10 00:22:25.417824 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:22:25.417797 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf699436c_1006_49b2_8dc4_061253a6b2b2.slice/crio-dad0f6301c80ec2fe2403aa9d334b224e1d04da64e89c788f302ff8c20024a30 WatchSource:0}: Error finding container dad0f6301c80ec2fe2403aa9d334b224e1d04da64e89c788f302ff8c20024a30: Status 404 returned error can't find the container with id dad0f6301c80ec2fe2403aa9d334b224e1d04da64e89c788f302ff8c20024a30 Jul 10 00:22:25.520469 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:25.520439 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-nxhzz" event={"ID":"f699436c-1006-49b2-8dc4-061253a6b2b2","Type":"ContainerStarted","Data":"dad0f6301c80ec2fe2403aa9d334b224e1d04da64e89c788f302ff8c20024a30"} Jul 10 00:22:29.122393 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:22:29.122345 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-ds66q" podUID="0654b471-0fff-4b5c-a6fc-d3e972163d20" Jul 10 00:22:29.534574 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:29.534545 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ds66q" Jul 10 00:22:32.854446 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:32.854408 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls\") pod \"dns-default-ds66q\" (UID: \"0654b471-0fff-4b5c-a6fc-d3e972163d20\") " pod="openshift-dns/dns-default-ds66q" Jul 10 00:22:32.856929 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:32.856900 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0654b471-0fff-4b5c-a6fc-d3e972163d20-metrics-tls\") pod \"dns-default-ds66q\" (UID: \"0654b471-0fff-4b5c-a6fc-d3e972163d20\") " pod="openshift-dns/dns-default-ds66q" Jul 10 00:22:33.137856 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:33.137766 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-p75jv\"" Jul 10 00:22:33.145766 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:33.145743 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ds66q" Jul 10 00:22:33.301135 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:33.301088 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ds66q"] Jul 10 00:22:33.304346 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:22:33.304315 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0654b471_0fff_4b5c_a6fc_d3e972163d20.slice/crio-e4ddbd640ccadddb27348881549c8e5f8091dfda15f2caf36c402d118b22883e WatchSource:0}: Error finding container e4ddbd640ccadddb27348881549c8e5f8091dfda15f2caf36c402d118b22883e: Status 404 returned error can't find the container with id e4ddbd640ccadddb27348881549c8e5f8091dfda15f2caf36c402d118b22883e Jul 10 00:22:33.551545 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:33.551492 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ds66q" event={"ID":"0654b471-0fff-4b5c-a6fc-d3e972163d20","Type":"ContainerStarted","Data":"e4ddbd640ccadddb27348881549c8e5f8091dfda15f2caf36c402d118b22883e"} Jul 10 00:22:42.590351 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:42.590311 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ds66q" event={"ID":"0654b471-0fff-4b5c-a6fc-d3e972163d20","Type":"ContainerStarted","Data":"071366b2c0d4784b2a1bc6bd62723104a4f2c5c938f3e15e4aa2fc3b53ea1d3b"} Jul 10 00:22:42.590820 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:42.590380 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ds66q" event={"ID":"0654b471-0fff-4b5c-a6fc-d3e972163d20","Type":"ContainerStarted","Data":"4fcd05a71bfa26a93fa56a48690b5f4f5446da15d5e1710057396b631b4d879a"} Jul 10 00:22:42.590820 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:42.590420 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ds66q" Jul 10 00:22:42.592249 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:42.592220 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-nxhzz" event={"ID":"f699436c-1006-49b2-8dc4-061253a6b2b2","Type":"ContainerStarted","Data":"2e71fa6cfeb4b88cb962c71b273572a360c3a6a879f562ceea0ca3d2497f5355"} Jul 10 00:22:42.592375 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:42.592337 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-nxhzz" Jul 10 00:22:42.607032 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:42.606983 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ds66q" podStartSLOduration=252.227967593 podStartE2EDuration="4m20.60696616s" totalImagesPullingTime="8.378998567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:18:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:22:42.605340711 +0000 UTC m=+295.538425039" watchObservedRunningTime="2026-07-10 00:22:42.60696616 +0000 UTC m=+295.540050483" Jul 10 00:22:42.610339 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:42.610318 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-nxhzz" Jul 10 00:22:42.620201 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:42.620159 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-nxhzz" podStartSLOduration=2.305685135 podStartE2EDuration="18.620145759s" totalImagesPullingTime="16.314460624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:22:42.619310359 +0000 UTC m=+295.552394669" watchObservedRunningTime="2026-07-10 00:22:42.620145759 +0000 UTC m=+295.553230066" Jul 10 00:22:47.463322 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:47.463292 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7zrr_20198eb1-6b3e-4e30-9adf-d4f13df4ee3b/ovn-acl-logging/0.log" Jul 10 00:22:47.466285 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:47.466264 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7zrr_20198eb1-6b3e-4e30-9adf-d4f13df4ee3b/ovn-acl-logging/0.log" Jul 10 00:22:47.472736 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:47.472694 2559 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 00:22:52.598972 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:22:52.598931 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ds66q" Jul 10 00:23:30.146703 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:30.146668 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 00:23:30.147181 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:30.146935 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7380182e-1b4d-427c-8c17-537c70315c91" containerName="alertmanager" containerID="cri-o://0c47199081163da248d911f32c0881a9e7dae2597dab7f1a286954800b61ad3b" gracePeriod=120 Jul 10 00:23:30.147181 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:30.146993 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7380182e-1b4d-427c-8c17-537c70315c91" containerName="kube-rbac-proxy-metric" containerID="cri-o://a58641b576bb66b2179ebb67a1508a4654a18a37eb3543ce5ae460d94eb37b4e" gracePeriod=120 Jul 10 00:23:30.147181 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:30.147043 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7380182e-1b4d-427c-8c17-537c70315c91" containerName="kube-rbac-proxy" containerID="cri-o://7fda78e5bf2b315745a8af8d48695971fa58a8e91bff5a40c00dfa7ea95e1bcd" gracePeriod=120 Jul 10 00:23:30.147181 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:30.147047 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7380182e-1b4d-427c-8c17-537c70315c91" containerName="kube-rbac-proxy-web" containerID="cri-o://9f24ae24e134e07a6fadd9fa4b0277418a05e90c376327e2df1b34bbe7396748" gracePeriod=120 Jul 10 00:23:30.147181 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:30.147047 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7380182e-1b4d-427c-8c17-537c70315c91" containerName="config-reloader" containerID="cri-o://1519faa7931802dee8cecdb128e41e0930e5c280115ae8439a97e4ef23c524f1" gracePeriod=120 Jul 10 00:23:30.147784 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:30.147132 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7380182e-1b4d-427c-8c17-537c70315c91" containerName="prom-label-proxy" containerID="cri-o://310b8ffdc94d0f1f0f98cca47884444ca72c09942d71686d70764cf9b72f3514" gracePeriod=120 Jul 10 00:23:30.765966 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:30.765931 2559 generic.go:356] "Generic (PLEG): container finished" podID="7380182e-1b4d-427c-8c17-537c70315c91" containerID="310b8ffdc94d0f1f0f98cca47884444ca72c09942d71686d70764cf9b72f3514" exitCode=0 Jul 10 00:23:30.765966 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:30.765957 2559 generic.go:356] "Generic (PLEG): container finished" podID="7380182e-1b4d-427c-8c17-537c70315c91" containerID="7fda78e5bf2b315745a8af8d48695971fa58a8e91bff5a40c00dfa7ea95e1bcd" exitCode=0 Jul 10 00:23:30.765966 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:30.765964 2559 generic.go:356] "Generic (PLEG): container finished" podID="7380182e-1b4d-427c-8c17-537c70315c91" containerID="1519faa7931802dee8cecdb128e41e0930e5c280115ae8439a97e4ef23c524f1" exitCode=0 Jul 10 00:23:30.765966 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:30.765970 2559 generic.go:356] "Generic (PLEG): container finished" podID="7380182e-1b4d-427c-8c17-537c70315c91" containerID="0c47199081163da248d911f32c0881a9e7dae2597dab7f1a286954800b61ad3b" exitCode=0 Jul 10 00:23:30.766241 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:30.765999 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7380182e-1b4d-427c-8c17-537c70315c91","Type":"ContainerDied","Data":"310b8ffdc94d0f1f0f98cca47884444ca72c09942d71686d70764cf9b72f3514"} Jul 10 00:23:30.766241 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:30.766056 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7380182e-1b4d-427c-8c17-537c70315c91","Type":"ContainerDied","Data":"7fda78e5bf2b315745a8af8d48695971fa58a8e91bff5a40c00dfa7ea95e1bcd"} Jul 10 00:23:30.766241 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:30.766083 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7380182e-1b4d-427c-8c17-537c70315c91","Type":"ContainerDied","Data":"1519faa7931802dee8cecdb128e41e0930e5c280115ae8439a97e4ef23c524f1"} Jul 10 00:23:30.766241 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:30.766107 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7380182e-1b4d-427c-8c17-537c70315c91","Type":"ContainerDied","Data":"0c47199081163da248d911f32c0881a9e7dae2597dab7f1a286954800b61ad3b"} Jul 10 00:23:31.395047 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.395026 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.472277 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.472249 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-secret-alertmanager-main-tls\") pod \"7380182e-1b4d-427c-8c17-537c70315c91\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " Jul 10 00:23:31.472277 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.472278 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9zkk\" (UniqueName: \"kubernetes.io/projected/7380182e-1b4d-427c-8c17-537c70315c91-kube-api-access-r9zkk\") pod \"7380182e-1b4d-427c-8c17-537c70315c91\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " Jul 10 00:23:31.472469 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.472301 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-config-volume\") pod \"7380182e-1b4d-427c-8c17-537c70315c91\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " Jul 10 00:23:31.472469 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.472326 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7380182e-1b4d-427c-8c17-537c70315c91-alertmanager-trusted-ca-bundle\") pod \"7380182e-1b4d-427c-8c17-537c70315c91\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " Jul 10 00:23:31.472469 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.472366 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7380182e-1b4d-427c-8c17-537c70315c91-alertmanager-main-db\") pod \"7380182e-1b4d-427c-8c17-537c70315c91\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " Jul 10 00:23:31.472469 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.472383 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-secret-alertmanager-kube-rbac-proxy-metric\") pod \"7380182e-1b4d-427c-8c17-537c70315c91\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " Jul 10 00:23:31.472469 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.472408 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7380182e-1b4d-427c-8c17-537c70315c91-tls-assets\") pod \"7380182e-1b4d-427c-8c17-537c70315c91\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " Jul 10 00:23:31.472469 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.472431 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-cluster-tls-config\") pod \"7380182e-1b4d-427c-8c17-537c70315c91\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " Jul 10 00:23:31.472469 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.472465 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-secret-alertmanager-kube-rbac-proxy\") pod \"7380182e-1b4d-427c-8c17-537c70315c91\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " Jul 10 00:23:31.472803 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.472492 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-secret-alertmanager-kube-rbac-proxy-web\") pod \"7380182e-1b4d-427c-8c17-537c70315c91\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " Jul 10 00:23:31.472803 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.472521 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7380182e-1b4d-427c-8c17-537c70315c91-metrics-client-ca\") pod \"7380182e-1b4d-427c-8c17-537c70315c91\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " Jul 10 00:23:31.472803 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.472744 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7380182e-1b4d-427c-8c17-537c70315c91-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "7380182e-1b4d-427c-8c17-537c70315c91" (UID: "7380182e-1b4d-427c-8c17-537c70315c91"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:23:31.472944 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.472890 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7380182e-1b4d-427c-8c17-537c70315c91-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "7380182e-1b4d-427c-8c17-537c70315c91" (UID: "7380182e-1b4d-427c-8c17-537c70315c91"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:23:31.472944 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.472904 2559 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7380182e-1b4d-427c-8c17-537c70315c91-alertmanager-main-db\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:23:31.472944 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.472935 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7380182e-1b4d-427c-8c17-537c70315c91-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "7380182e-1b4d-427c-8c17-537c70315c91" (UID: "7380182e-1b4d-427c-8c17-537c70315c91"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:23:31.475109 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.475082 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-config-volume" (OuterVolumeSpecName: "config-volume") pod "7380182e-1b4d-427c-8c17-537c70315c91" (UID: "7380182e-1b4d-427c-8c17-537c70315c91"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:23:31.475231 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.475175 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "7380182e-1b4d-427c-8c17-537c70315c91" (UID: "7380182e-1b4d-427c-8c17-537c70315c91"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:23:31.475298 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.475272 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7380182e-1b4d-427c-8c17-537c70315c91-kube-api-access-r9zkk" (OuterVolumeSpecName: "kube-api-access-r9zkk") pod "7380182e-1b4d-427c-8c17-537c70315c91" (UID: "7380182e-1b4d-427c-8c17-537c70315c91"). InnerVolumeSpecName "kube-api-access-r9zkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:23:31.475378 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.475359 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "7380182e-1b4d-427c-8c17-537c70315c91" (UID: "7380182e-1b4d-427c-8c17-537c70315c91"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:23:31.475704 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.475666 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "7380182e-1b4d-427c-8c17-537c70315c91" (UID: "7380182e-1b4d-427c-8c17-537c70315c91"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:23:31.475704 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.475695 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "7380182e-1b4d-427c-8c17-537c70315c91" (UID: "7380182e-1b4d-427c-8c17-537c70315c91"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:23:31.475878 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.475683 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7380182e-1b4d-427c-8c17-537c70315c91-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "7380182e-1b4d-427c-8c17-537c70315c91" (UID: "7380182e-1b4d-427c-8c17-537c70315c91"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:23:31.478855 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.478827 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "7380182e-1b4d-427c-8c17-537c70315c91" (UID: "7380182e-1b4d-427c-8c17-537c70315c91"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:23:31.573632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.573614 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-web-config\") pod \"7380182e-1b4d-427c-8c17-537c70315c91\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " Jul 10 00:23:31.573720 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.573640 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7380182e-1b4d-427c-8c17-537c70315c91-config-out\") pod \"7380182e-1b4d-427c-8c17-537c70315c91\" (UID: \"7380182e-1b4d-427c-8c17-537c70315c91\") " Jul 10 00:23:31.573758 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.573722 2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-secret-alertmanager-main-tls\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:23:31.573758 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.573733 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9zkk\" (UniqueName: \"kubernetes.io/projected/7380182e-1b4d-427c-8c17-537c70315c91-kube-api-access-r9zkk\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:23:31.573758 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.573742 2559 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-config-volume\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:23:31.573758 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.573751 2559 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7380182e-1b4d-427c-8c17-537c70315c91-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:23:31.573874 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.573764 2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:23:31.573874 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.573792 2559 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7380182e-1b4d-427c-8c17-537c70315c91-tls-assets\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:23:31.573874 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.573800 2559 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-cluster-tls-config\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:23:31.573874 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.573808 2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:23:31.573874 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.573817 2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:23:31.573874 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.573826 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7380182e-1b4d-427c-8c17-537c70315c91-metrics-client-ca\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:23:31.575331 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.575312 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7380182e-1b4d-427c-8c17-537c70315c91-config-out" (OuterVolumeSpecName: "config-out") pod "7380182e-1b4d-427c-8c17-537c70315c91" (UID: "7380182e-1b4d-427c-8c17-537c70315c91"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:23:31.583361 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.583341 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-web-config" (OuterVolumeSpecName: "web-config") pod "7380182e-1b4d-427c-8c17-537c70315c91" (UID: "7380182e-1b4d-427c-8c17-537c70315c91"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:23:31.675000 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.674981 2559 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7380182e-1b4d-427c-8c17-537c70315c91-web-config\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:23:31.675000 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.675002 2559 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7380182e-1b4d-427c-8c17-537c70315c91-config-out\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:23:31.774994 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.774965 2559 generic.go:356] "Generic (PLEG): container finished" podID="7380182e-1b4d-427c-8c17-537c70315c91" containerID="a58641b576bb66b2179ebb67a1508a4654a18a37eb3543ce5ae460d94eb37b4e" exitCode=0 Jul 10 00:23:31.774994 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.774991 2559 generic.go:356] "Generic (PLEG): container finished" podID="7380182e-1b4d-427c-8c17-537c70315c91" containerID="9f24ae24e134e07a6fadd9fa4b0277418a05e90c376327e2df1b34bbe7396748" exitCode=0 Jul 10 00:23:31.775217 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.775036 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.775217 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.775043 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7380182e-1b4d-427c-8c17-537c70315c91","Type":"ContainerDied","Data":"a58641b576bb66b2179ebb67a1508a4654a18a37eb3543ce5ae460d94eb37b4e"} Jul 10 00:23:31.775217 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.775108 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7380182e-1b4d-427c-8c17-537c70315c91","Type":"ContainerDied","Data":"9f24ae24e134e07a6fadd9fa4b0277418a05e90c376327e2df1b34bbe7396748"} Jul 10 00:23:31.775217 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.775148 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7380182e-1b4d-427c-8c17-537c70315c91","Type":"ContainerDied","Data":"60f2db231c3f3967c221e52e36f3395c666696de2d44f7cf9d4f6d8b617a3d08"} Jul 10 00:23:31.775788 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.775772 2559 scope.go:117] "RemoveContainer" containerID="310b8ffdc94d0f1f0f98cca47884444ca72c09942d71686d70764cf9b72f3514" Jul 10 00:23:31.783351 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.783337 2559 scope.go:117] "RemoveContainer" containerID="a58641b576bb66b2179ebb67a1508a4654a18a37eb3543ce5ae460d94eb37b4e" Jul 10 00:23:31.789748 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.789733 2559 scope.go:117] "RemoveContainer" containerID="7fda78e5bf2b315745a8af8d48695971fa58a8e91bff5a40c00dfa7ea95e1bcd" Jul 10 00:23:31.792511 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.792492 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 00:23:31.796399 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.796380 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 00:23:31.797035 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.797018 2559 scope.go:117] "RemoveContainer" containerID="9f24ae24e134e07a6fadd9fa4b0277418a05e90c376327e2df1b34bbe7396748" Jul 10 00:23:31.803465 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.803449 2559 scope.go:117] "RemoveContainer" containerID="1519faa7931802dee8cecdb128e41e0930e5c280115ae8439a97e4ef23c524f1" Jul 10 00:23:31.809954 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.809939 2559 scope.go:117] "RemoveContainer" containerID="0c47199081163da248d911f32c0881a9e7dae2597dab7f1a286954800b61ad3b" Jul 10 00:23:31.816607 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.816591 2559 scope.go:117] "RemoveContainer" containerID="7310bab68e3901cf14cc25ac4624db1ec4f37f71336c44b8e2b3d81477b186ea" Jul 10 00:23:31.823261 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.823246 2559 scope.go:117] "RemoveContainer" containerID="310b8ffdc94d0f1f0f98cca47884444ca72c09942d71686d70764cf9b72f3514" Jul 10 00:23:31.823536 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:23:31.823517 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"310b8ffdc94d0f1f0f98cca47884444ca72c09942d71686d70764cf9b72f3514\": container with ID starting with 310b8ffdc94d0f1f0f98cca47884444ca72c09942d71686d70764cf9b72f3514 not found: ID does not exist" containerID="310b8ffdc94d0f1f0f98cca47884444ca72c09942d71686d70764cf9b72f3514" Jul 10 00:23:31.823587 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.823543 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"310b8ffdc94d0f1f0f98cca47884444ca72c09942d71686d70764cf9b72f3514"} err="failed to get container status \"310b8ffdc94d0f1f0f98cca47884444ca72c09942d71686d70764cf9b72f3514\": rpc error: code = NotFound desc = could not find container \"310b8ffdc94d0f1f0f98cca47884444ca72c09942d71686d70764cf9b72f3514\": container with ID starting with 310b8ffdc94d0f1f0f98cca47884444ca72c09942d71686d70764cf9b72f3514 not found: ID does not exist" Jul 10 00:23:31.823587 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.823563 2559 scope.go:117] "RemoveContainer" containerID="a58641b576bb66b2179ebb67a1508a4654a18a37eb3543ce5ae460d94eb37b4e" Jul 10 00:23:31.823794 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:23:31.823753 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a58641b576bb66b2179ebb67a1508a4654a18a37eb3543ce5ae460d94eb37b4e\": container with ID starting with a58641b576bb66b2179ebb67a1508a4654a18a37eb3543ce5ae460d94eb37b4e not found: ID does not exist" containerID="a58641b576bb66b2179ebb67a1508a4654a18a37eb3543ce5ae460d94eb37b4e" Jul 10 00:23:31.823794 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.823776 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a58641b576bb66b2179ebb67a1508a4654a18a37eb3543ce5ae460d94eb37b4e"} err="failed to get container status \"a58641b576bb66b2179ebb67a1508a4654a18a37eb3543ce5ae460d94eb37b4e\": rpc error: code = NotFound desc = could not find container \"a58641b576bb66b2179ebb67a1508a4654a18a37eb3543ce5ae460d94eb37b4e\": container with ID starting with a58641b576bb66b2179ebb67a1508a4654a18a37eb3543ce5ae460d94eb37b4e not found: ID does not exist" Jul 10 00:23:31.823794 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.823790 2559 scope.go:117] "RemoveContainer" containerID="7fda78e5bf2b315745a8af8d48695971fa58a8e91bff5a40c00dfa7ea95e1bcd" Jul 10 00:23:31.824014 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:23:31.823995 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fda78e5bf2b315745a8af8d48695971fa58a8e91bff5a40c00dfa7ea95e1bcd\": container with ID starting with 7fda78e5bf2b315745a8af8d48695971fa58a8e91bff5a40c00dfa7ea95e1bcd not found: ID does not exist" containerID="7fda78e5bf2b315745a8af8d48695971fa58a8e91bff5a40c00dfa7ea95e1bcd" Jul 10 00:23:31.824055 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.824020 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fda78e5bf2b315745a8af8d48695971fa58a8e91bff5a40c00dfa7ea95e1bcd"} err="failed to get container status \"7fda78e5bf2b315745a8af8d48695971fa58a8e91bff5a40c00dfa7ea95e1bcd\": rpc error: code = NotFound desc = could not find container \"7fda78e5bf2b315745a8af8d48695971fa58a8e91bff5a40c00dfa7ea95e1bcd\": container with ID starting with 7fda78e5bf2b315745a8af8d48695971fa58a8e91bff5a40c00dfa7ea95e1bcd not found: ID does not exist" Jul 10 00:23:31.824055 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.824037 2559 scope.go:117] "RemoveContainer" containerID="9f24ae24e134e07a6fadd9fa4b0277418a05e90c376327e2df1b34bbe7396748" Jul 10 00:23:31.824249 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:23:31.824234 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f24ae24e134e07a6fadd9fa4b0277418a05e90c376327e2df1b34bbe7396748\": container with ID starting with 9f24ae24e134e07a6fadd9fa4b0277418a05e90c376327e2df1b34bbe7396748 not found: ID does not exist" containerID="9f24ae24e134e07a6fadd9fa4b0277418a05e90c376327e2df1b34bbe7396748" Jul 10 00:23:31.824298 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.824252 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f24ae24e134e07a6fadd9fa4b0277418a05e90c376327e2df1b34bbe7396748"} err="failed to get container status \"9f24ae24e134e07a6fadd9fa4b0277418a05e90c376327e2df1b34bbe7396748\": rpc error: code = NotFound desc = could not find container \"9f24ae24e134e07a6fadd9fa4b0277418a05e90c376327e2df1b34bbe7396748\": container with ID starting with 9f24ae24e134e07a6fadd9fa4b0277418a05e90c376327e2df1b34bbe7396748 not found: ID does not exist" Jul 10 00:23:31.824298 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.824264 2559 scope.go:117] "RemoveContainer" containerID="1519faa7931802dee8cecdb128e41e0930e5c280115ae8439a97e4ef23c524f1" Jul 10 00:23:31.824483 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:23:31.824466 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1519faa7931802dee8cecdb128e41e0930e5c280115ae8439a97e4ef23c524f1\": container with ID starting with 1519faa7931802dee8cecdb128e41e0930e5c280115ae8439a97e4ef23c524f1 not found: ID does not exist" containerID="1519faa7931802dee8cecdb128e41e0930e5c280115ae8439a97e4ef23c524f1" Jul 10 00:23:31.824526 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.824488 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1519faa7931802dee8cecdb128e41e0930e5c280115ae8439a97e4ef23c524f1"} err="failed to get container status \"1519faa7931802dee8cecdb128e41e0930e5c280115ae8439a97e4ef23c524f1\": rpc error: code = NotFound desc = could not find container \"1519faa7931802dee8cecdb128e41e0930e5c280115ae8439a97e4ef23c524f1\": container with ID starting with 1519faa7931802dee8cecdb128e41e0930e5c280115ae8439a97e4ef23c524f1 not found: ID does not exist" Jul 10 00:23:31.824526 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.824502 2559 scope.go:117] "RemoveContainer" containerID="0c47199081163da248d911f32c0881a9e7dae2597dab7f1a286954800b61ad3b" Jul 10 00:23:31.824711 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:23:31.824695 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c47199081163da248d911f32c0881a9e7dae2597dab7f1a286954800b61ad3b\": container with ID starting with 0c47199081163da248d911f32c0881a9e7dae2597dab7f1a286954800b61ad3b not found: ID does not exist" containerID="0c47199081163da248d911f32c0881a9e7dae2597dab7f1a286954800b61ad3b" Jul 10 00:23:31.824766 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.824719 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c47199081163da248d911f32c0881a9e7dae2597dab7f1a286954800b61ad3b"} err="failed to get container status \"0c47199081163da248d911f32c0881a9e7dae2597dab7f1a286954800b61ad3b\": rpc error: code = NotFound desc = could not find container \"0c47199081163da248d911f32c0881a9e7dae2597dab7f1a286954800b61ad3b\": container with ID starting with 0c47199081163da248d911f32c0881a9e7dae2597dab7f1a286954800b61ad3b not found: ID does not exist" Jul 10 00:23:31.824766 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.824739 2559 scope.go:117] "RemoveContainer" containerID="7310bab68e3901cf14cc25ac4624db1ec4f37f71336c44b8e2b3d81477b186ea" Jul 10 00:23:31.824984 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:23:31.824968 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7310bab68e3901cf14cc25ac4624db1ec4f37f71336c44b8e2b3d81477b186ea\": container with ID starting with 7310bab68e3901cf14cc25ac4624db1ec4f37f71336c44b8e2b3d81477b186ea not found: ID does not exist" containerID="7310bab68e3901cf14cc25ac4624db1ec4f37f71336c44b8e2b3d81477b186ea" Jul 10 00:23:31.825023 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.824988 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7310bab68e3901cf14cc25ac4624db1ec4f37f71336c44b8e2b3d81477b186ea"} err="failed to get container status \"7310bab68e3901cf14cc25ac4624db1ec4f37f71336c44b8e2b3d81477b186ea\": rpc error: code = NotFound desc = could not find container \"7310bab68e3901cf14cc25ac4624db1ec4f37f71336c44b8e2b3d81477b186ea\": container with ID starting with 7310bab68e3901cf14cc25ac4624db1ec4f37f71336c44b8e2b3d81477b186ea not found: ID does not exist" Jul 10 00:23:31.825023 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.825002 2559 scope.go:117] "RemoveContainer" containerID="310b8ffdc94d0f1f0f98cca47884444ca72c09942d71686d70764cf9b72f3514" Jul 10 00:23:31.825244 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.825226 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"310b8ffdc94d0f1f0f98cca47884444ca72c09942d71686d70764cf9b72f3514"} err="failed to get container status \"310b8ffdc94d0f1f0f98cca47884444ca72c09942d71686d70764cf9b72f3514\": rpc error: code = NotFound desc = could not find container \"310b8ffdc94d0f1f0f98cca47884444ca72c09942d71686d70764cf9b72f3514\": container with ID starting with 310b8ffdc94d0f1f0f98cca47884444ca72c09942d71686d70764cf9b72f3514 not found: ID does not exist" Jul 10 00:23:31.825307 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.825246 2559 scope.go:117] "RemoveContainer" containerID="a58641b576bb66b2179ebb67a1508a4654a18a37eb3543ce5ae460d94eb37b4e" Jul 10 00:23:31.825484 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.825464 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a58641b576bb66b2179ebb67a1508a4654a18a37eb3543ce5ae460d94eb37b4e"} err="failed to get container status \"a58641b576bb66b2179ebb67a1508a4654a18a37eb3543ce5ae460d94eb37b4e\": rpc error: code = NotFound desc = could not find container \"a58641b576bb66b2179ebb67a1508a4654a18a37eb3543ce5ae460d94eb37b4e\": container with ID starting with a58641b576bb66b2179ebb67a1508a4654a18a37eb3543ce5ae460d94eb37b4e not found: ID does not exist" Jul 10 00:23:31.825530 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.825486 2559 scope.go:117] "RemoveContainer" containerID="7fda78e5bf2b315745a8af8d48695971fa58a8e91bff5a40c00dfa7ea95e1bcd" Jul 10 00:23:31.825668 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.825653 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fda78e5bf2b315745a8af8d48695971fa58a8e91bff5a40c00dfa7ea95e1bcd"} err="failed to get container status \"7fda78e5bf2b315745a8af8d48695971fa58a8e91bff5a40c00dfa7ea95e1bcd\": rpc error: code = NotFound desc = could not find container \"7fda78e5bf2b315745a8af8d48695971fa58a8e91bff5a40c00dfa7ea95e1bcd\": container with ID starting with 7fda78e5bf2b315745a8af8d48695971fa58a8e91bff5a40c00dfa7ea95e1bcd not found: ID does not exist" Jul 10 00:23:31.825710 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.825668 2559 scope.go:117] "RemoveContainer" containerID="9f24ae24e134e07a6fadd9fa4b0277418a05e90c376327e2df1b34bbe7396748" Jul 10 00:23:31.825866 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.825850 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f24ae24e134e07a6fadd9fa4b0277418a05e90c376327e2df1b34bbe7396748"} err="failed to get container status \"9f24ae24e134e07a6fadd9fa4b0277418a05e90c376327e2df1b34bbe7396748\": rpc error: code = NotFound desc = could not find container \"9f24ae24e134e07a6fadd9fa4b0277418a05e90c376327e2df1b34bbe7396748\": container with ID starting with 9f24ae24e134e07a6fadd9fa4b0277418a05e90c376327e2df1b34bbe7396748 not found: ID does not exist" Jul 10 00:23:31.825905 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.825868 2559 scope.go:117] "RemoveContainer" containerID="1519faa7931802dee8cecdb128e41e0930e5c280115ae8439a97e4ef23c524f1" Jul 10 00:23:31.826065 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.826048 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1519faa7931802dee8cecdb128e41e0930e5c280115ae8439a97e4ef23c524f1"} err="failed to get container status \"1519faa7931802dee8cecdb128e41e0930e5c280115ae8439a97e4ef23c524f1\": rpc error: code = NotFound desc = could not find container \"1519faa7931802dee8cecdb128e41e0930e5c280115ae8439a97e4ef23c524f1\": container with ID starting with 1519faa7931802dee8cecdb128e41e0930e5c280115ae8439a97e4ef23c524f1 not found: ID does not exist" Jul 10 00:23:31.826111 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.826066 2559 scope.go:117] "RemoveContainer" containerID="0c47199081163da248d911f32c0881a9e7dae2597dab7f1a286954800b61ad3b" Jul 10 00:23:31.826310 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.826292 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c47199081163da248d911f32c0881a9e7dae2597dab7f1a286954800b61ad3b"} err="failed to get container status \"0c47199081163da248d911f32c0881a9e7dae2597dab7f1a286954800b61ad3b\": rpc error: code = NotFound desc = could not find container \"0c47199081163da248d911f32c0881a9e7dae2597dab7f1a286954800b61ad3b\": container with ID starting with 0c47199081163da248d911f32c0881a9e7dae2597dab7f1a286954800b61ad3b not found: ID does not exist" Jul 10 00:23:31.826380 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.826310 2559 scope.go:117] "RemoveContainer" containerID="7310bab68e3901cf14cc25ac4624db1ec4f37f71336c44b8e2b3d81477b186ea" Jul 10 00:23:31.826811 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.826773 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7310bab68e3901cf14cc25ac4624db1ec4f37f71336c44b8e2b3d81477b186ea"} err="failed to get container status \"7310bab68e3901cf14cc25ac4624db1ec4f37f71336c44b8e2b3d81477b186ea\": rpc error: code = NotFound desc = could not find container \"7310bab68e3901cf14cc25ac4624db1ec4f37f71336c44b8e2b3d81477b186ea\": container with ID starting with 7310bab68e3901cf14cc25ac4624db1ec4f37f71336c44b8e2b3d81477b186ea not found: ID does not exist" Jul 10 00:23:31.828018 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.828002 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 00:23:31.828256 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.828242 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7380182e-1b4d-427c-8c17-537c70315c91" containerName="alertmanager" Jul 10 00:23:31.828308 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.828259 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7380182e-1b4d-427c-8c17-537c70315c91" containerName="alertmanager" Jul 10 00:23:31.828308 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.828270 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7380182e-1b4d-427c-8c17-537c70315c91" containerName="kube-rbac-proxy" Jul 10 00:23:31.828308 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.828276 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7380182e-1b4d-427c-8c17-537c70315c91" containerName="kube-rbac-proxy" Jul 10 00:23:31.828308 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.828282 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7380182e-1b4d-427c-8c17-537c70315c91" containerName="kube-rbac-proxy-metric" Jul 10 00:23:31.828308 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.828289 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7380182e-1b4d-427c-8c17-537c70315c91" containerName="kube-rbac-proxy-metric" Jul 10 00:23:31.828308 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.828295 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7380182e-1b4d-427c-8c17-537c70315c91" containerName="kube-rbac-proxy-web" Jul 10 00:23:31.828308 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.828301 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7380182e-1b4d-427c-8c17-537c70315c91" containerName="kube-rbac-proxy-web" Jul 10 00:23:31.828308 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.828310 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7380182e-1b4d-427c-8c17-537c70315c91" containerName="init-config-reloader" Jul 10 00:23:31.828549 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.828315 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7380182e-1b4d-427c-8c17-537c70315c91" containerName="init-config-reloader" Jul 10 00:23:31.828549 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.828322 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7380182e-1b4d-427c-8c17-537c70315c91" containerName="prom-label-proxy" Jul 10 00:23:31.828549 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.828327 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7380182e-1b4d-427c-8c17-537c70315c91" containerName="prom-label-proxy" Jul 10 00:23:31.828549 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.828335 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7380182e-1b4d-427c-8c17-537c70315c91" containerName="config-reloader" Jul 10 00:23:31.828549 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.828340 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7380182e-1b4d-427c-8c17-537c70315c91" containerName="config-reloader" Jul 10 00:23:31.828549 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.828379 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7380182e-1b4d-427c-8c17-537c70315c91" containerName="alertmanager" Jul 10 00:23:31.828549 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.828385 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7380182e-1b4d-427c-8c17-537c70315c91" containerName="config-reloader" Jul 10 00:23:31.828549 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.828391 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7380182e-1b4d-427c-8c17-537c70315c91" containerName="kube-rbac-proxy-web" Jul 10 00:23:31.828549 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.828396 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7380182e-1b4d-427c-8c17-537c70315c91" containerName="kube-rbac-proxy" Jul 10 00:23:31.828549 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.828401 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7380182e-1b4d-427c-8c17-537c70315c91" containerName="kube-rbac-proxy-metric" Jul 10 00:23:31.828549 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.828407 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7380182e-1b4d-427c-8c17-537c70315c91" containerName="prom-label-proxy" Jul 10 00:23:31.831312 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.831290 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.833527 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.833507 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 10 00:23:31.833643 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.833625 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 10 00:23:31.833709 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.833698 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 10 00:23:31.833764 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.833706 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 10 00:23:31.833764 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.833713 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 10 00:23:31.833867 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.833703 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 10 00:23:31.834015 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.834000 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 10 00:23:31.834061 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.834029 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 10 00:23:31.834061 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.834046 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zw4zw\"" Jul 10 00:23:31.839563 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.839540 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 10 00:23:31.845926 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.845908 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 00:23:31.876003 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.875979 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f96592fc-0ecc-40f4-be2a-d5c713e1041e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.876094 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.876009 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f96592fc-0ecc-40f4-be2a-d5c713e1041e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.876094 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.876036 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwxpz\" (UniqueName: \"kubernetes.io/projected/f96592fc-0ecc-40f4-be2a-d5c713e1041e-kube-api-access-wwxpz\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.876094 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.876054 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f96592fc-0ecc-40f4-be2a-d5c713e1041e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.876236 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.876092 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f96592fc-0ecc-40f4-be2a-d5c713e1041e-web-config\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.876236 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.876185 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f96592fc-0ecc-40f4-be2a-d5c713e1041e-config-out\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.876236 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.876215 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f96592fc-0ecc-40f4-be2a-d5c713e1041e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.876236 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.876234 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f96592fc-0ecc-40f4-be2a-d5c713e1041e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.876388 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.876249 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f96592fc-0ecc-40f4-be2a-d5c713e1041e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.876388 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.876272 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f96592fc-0ecc-40f4-be2a-d5c713e1041e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.876388 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.876323 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f96592fc-0ecc-40f4-be2a-d5c713e1041e-config-volume\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.876388 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.876347 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f96592fc-0ecc-40f4-be2a-d5c713e1041e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.876388 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.876364 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f96592fc-0ecc-40f4-be2a-d5c713e1041e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.977178 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.977156 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f96592fc-0ecc-40f4-be2a-d5c713e1041e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.977268 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.977187 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f96592fc-0ecc-40f4-be2a-d5c713e1041e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.977268 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.977206 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwxpz\" (UniqueName: \"kubernetes.io/projected/f96592fc-0ecc-40f4-be2a-d5c713e1041e-kube-api-access-wwxpz\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.977268 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.977224 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f96592fc-0ecc-40f4-be2a-d5c713e1041e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.977268 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.977241 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f96592fc-0ecc-40f4-be2a-d5c713e1041e-web-config\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.977461 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.977291 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f96592fc-0ecc-40f4-be2a-d5c713e1041e-config-out\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.977461 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.977319 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f96592fc-0ecc-40f4-be2a-d5c713e1041e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.977461 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.977345 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f96592fc-0ecc-40f4-be2a-d5c713e1041e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.977461 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.977366 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f96592fc-0ecc-40f4-be2a-d5c713e1041e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.977461 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.977393 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f96592fc-0ecc-40f4-be2a-d5c713e1041e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.977461 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.977426 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f96592fc-0ecc-40f4-be2a-d5c713e1041e-config-volume\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.977461 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.977452 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f96592fc-0ecc-40f4-be2a-d5c713e1041e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.977791 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.977477 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f96592fc-0ecc-40f4-be2a-d5c713e1041e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.977971 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.977945 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f96592fc-0ecc-40f4-be2a-d5c713e1041e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.978255 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.978231 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f96592fc-0ecc-40f4-be2a-d5c713e1041e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.980427 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.980055 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f96592fc-0ecc-40f4-be2a-d5c713e1041e-config-out\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.980427 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.980248 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f96592fc-0ecc-40f4-be2a-d5c713e1041e-web-config\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.980427 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.980377 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f96592fc-0ecc-40f4-be2a-d5c713e1041e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.980663 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.980435 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f96592fc-0ecc-40f4-be2a-d5c713e1041e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.980663 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.980520 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f96592fc-0ecc-40f4-be2a-d5c713e1041e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.980663 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.980618 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f96592fc-0ecc-40f4-be2a-d5c713e1041e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.980663 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.980643 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f96592fc-0ecc-40f4-be2a-d5c713e1041e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.980874 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.980688 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f96592fc-0ecc-40f4-be2a-d5c713e1041e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.980944 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.980922 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f96592fc-0ecc-40f4-be2a-d5c713e1041e-config-volume\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.982058 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.982038 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f96592fc-0ecc-40f4-be2a-d5c713e1041e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:31.985018 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:31.985001 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwxpz\" (UniqueName: \"kubernetes.io/projected/f96592fc-0ecc-40f4-be2a-d5c713e1041e-kube-api-access-wwxpz\") pod \"alertmanager-main-0\" (UID: \"f96592fc-0ecc-40f4-be2a-d5c713e1041e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:32.140906 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:32.140831 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:23:32.271204 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:32.271176 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 00:23:32.272923 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:23:32.272897 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96592fc_0ecc_40f4_be2a_d5c713e1041e.slice/crio-f024e74dd1b1bd7a63d9c2f271fcd8c2041e053710efd91cafb11336f3d2a572 WatchSource:0}: Error finding container f024e74dd1b1bd7a63d9c2f271fcd8c2041e053710efd91cafb11336f3d2a572: Status 404 returned error can't find the container with id f024e74dd1b1bd7a63d9c2f271fcd8c2041e053710efd91cafb11336f3d2a572 Jul 10 00:23:32.780059 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:32.780024 2559 generic.go:356] "Generic (PLEG): container finished" podID="f96592fc-0ecc-40f4-be2a-d5c713e1041e" containerID="e71b785c30472ed713fd9f41bbd00728920c4f9b2014d813737e994ee38ee472" exitCode=0 Jul 10 00:23:32.780482 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:32.780109 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f96592fc-0ecc-40f4-be2a-d5c713e1041e","Type":"ContainerDied","Data":"e71b785c30472ed713fd9f41bbd00728920c4f9b2014d813737e994ee38ee472"} Jul 10 00:23:32.780482 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:32.780196 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f96592fc-0ecc-40f4-be2a-d5c713e1041e","Type":"ContainerStarted","Data":"f024e74dd1b1bd7a63d9c2f271fcd8c2041e053710efd91cafb11336f3d2a572"} Jul 10 00:23:33.649482 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:33.649448 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7380182e-1b4d-427c-8c17-537c70315c91" path="/var/lib/kubelet/pods/7380182e-1b4d-427c-8c17-537c70315c91/volumes" Jul 10 00:23:33.787323 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:33.787292 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f96592fc-0ecc-40f4-be2a-d5c713e1041e","Type":"ContainerStarted","Data":"75fe2ef533c7fa8477018d2d917ab0992303711e4801bb6512aa94aec3c6b2d6"} Jul 10 00:23:33.787769 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:33.787361 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f96592fc-0ecc-40f4-be2a-d5c713e1041e","Type":"ContainerStarted","Data":"ef5c346df78a9390aaeda4e7af951b6d4240201848e5b572d9f16733fd3d91ef"} Jul 10 00:23:33.787769 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:33.787412 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f96592fc-0ecc-40f4-be2a-d5c713e1041e","Type":"ContainerStarted","Data":"a0c67594e84725c5b412cb08edfbc93f37ef07968a6dc01a436a03e556533335"} Jul 10 00:23:33.787769 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:33.787456 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f96592fc-0ecc-40f4-be2a-d5c713e1041e","Type":"ContainerStarted","Data":"fa536648ade86ff7633f562a7122dcc20f5e36c064f7b2fe5e32dd16bb5eee09"} Jul 10 00:23:33.787769 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:33.787500 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f96592fc-0ecc-40f4-be2a-d5c713e1041e","Type":"ContainerStarted","Data":"2a02a08706b9b23fd5b4c975c74e0c7b6fe4ec190146136a9f2fd95f341cb5de"} Jul 10 00:23:33.787769 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:33.787542 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f96592fc-0ecc-40f4-be2a-d5c713e1041e","Type":"ContainerStarted","Data":"3181480ebf7bd550a9a7f3a21d45875e441a0b422d4fe4578858c3a263606be3"} Jul 10 00:23:33.816353 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:23:33.816306 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.80076846 podStartE2EDuration="2.816277658s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.509198ms" isStatefulPod=false podCreationTimestamp="2026-07-10 00:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:23:33.815439081 +0000 UTC m=+346.748523392" watchObservedRunningTime="2026-07-10 00:23:33.816277658 +0000 UTC m=+346.749361958" Jul 10 00:24:51.324855 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:24:51.324821 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2zrgw"] Jul 10 00:24:51.326714 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:24:51.326698 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2zrgw" Jul 10 00:24:51.328762 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:24:51.328740 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 10 00:24:51.329433 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:24:51.329412 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 10 00:24:51.329546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:24:51.329487 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-6vw6c\"" Jul 10 00:24:51.337825 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:24:51.337802 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2zrgw"] Jul 10 00:24:51.404992 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:24:51.404968 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b1a9b28-460b-4086-ab26-c2f66ded094c-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2zrgw\" (UID: \"6b1a9b28-460b-4086-ab26-c2f66ded094c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2zrgw" Jul 10 00:24:51.405093 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:24:51.405015 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zxpg\" (UniqueName: \"kubernetes.io/projected/6b1a9b28-460b-4086-ab26-c2f66ded094c-kube-api-access-2zxpg\") pod \"cert-manager-cainjector-559cdd8c87-2zrgw\" (UID: \"6b1a9b28-460b-4086-ab26-c2f66ded094c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2zrgw" Jul 10 00:24:51.505923 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:24:51.505895 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b1a9b28-460b-4086-ab26-c2f66ded094c-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2zrgw\" (UID: \"6b1a9b28-460b-4086-ab26-c2f66ded094c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2zrgw" Jul 10 00:24:51.506013 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:24:51.505943 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zxpg\" (UniqueName: \"kubernetes.io/projected/6b1a9b28-460b-4086-ab26-c2f66ded094c-kube-api-access-2zxpg\") pod \"cert-manager-cainjector-559cdd8c87-2zrgw\" (UID: \"6b1a9b28-460b-4086-ab26-c2f66ded094c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2zrgw" Jul 10 00:24:51.513589 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:24:51.513564 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b1a9b28-460b-4086-ab26-c2f66ded094c-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2zrgw\" (UID: \"6b1a9b28-460b-4086-ab26-c2f66ded094c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2zrgw" Jul 10 00:24:51.513695 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:24:51.513681 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zxpg\" (UniqueName: \"kubernetes.io/projected/6b1a9b28-460b-4086-ab26-c2f66ded094c-kube-api-access-2zxpg\") pod \"cert-manager-cainjector-559cdd8c87-2zrgw\" (UID: \"6b1a9b28-460b-4086-ab26-c2f66ded094c\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2zrgw" Jul 10 00:24:51.635745 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:24:51.635673 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2zrgw" Jul 10 00:24:51.755372 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:24:51.755351 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2zrgw"] Jul 10 00:24:51.757866 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:24:51.757842 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b1a9b28_460b_4086_ab26_c2f66ded094c.slice/crio-b8d34fb6459e7d87cbb2db45924303d3e5aa8e8f287d3b162f977eecc8671d03 WatchSource:0}: Error finding container b8d34fb6459e7d87cbb2db45924303d3e5aa8e8f287d3b162f977eecc8671d03: Status 404 returned error can't find the container with id b8d34fb6459e7d87cbb2db45924303d3e5aa8e8f287d3b162f977eecc8671d03 Jul 10 00:24:51.759821 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:24:51.759802 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 00:24:52.064234 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:24:52.064201 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2zrgw" event={"ID":"6b1a9b28-460b-4086-ab26-c2f66ded094c","Type":"ContainerStarted","Data":"b8d34fb6459e7d87cbb2db45924303d3e5aa8e8f287d3b162f977eecc8671d03"} Jul 10 00:24:56.081561 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:24:56.081520 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2zrgw" event={"ID":"6b1a9b28-460b-4086-ab26-c2f66ded094c","Type":"ContainerStarted","Data":"42e37f3c654127bcc46d6db6ea0ecb484fd885684d177bb4cc3f0eded2781ac0"} Jul 10 00:24:56.096572 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:24:56.096530 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2zrgw" podStartSLOduration=1.240337592 podStartE2EDuration="5.096515022s" totalImagesPullingTime="3.85617743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:24:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:24:56.095279942 +0000 UTC m=+429.028364250" watchObservedRunningTime="2026-07-10 00:24:56.096515022 +0000 UTC m=+429.029599330" Jul 10 00:25:26.554502 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:26.554422 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-56d4fb96db-zq65z"] Jul 10 00:25:26.559263 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:26.559242 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-56d4fb96db-zq65z" Jul 10 00:25:26.561551 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:26.561531 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 10 00:25:26.562045 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:26.562020 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 10 00:25:26.562154 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:26.562053 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 10 00:25:26.562232 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:26.562159 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-htlhs\"" Jul 10 00:25:26.562232 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:26.562165 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 10 00:25:26.562315 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:26.562296 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 10 00:25:26.565050 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:26.565029 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-56d4fb96db-zq65z"] Jul 10 00:25:26.748679 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:26.748650 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3ad3bac2-deb5-4191-8edc-28e5cf1ac514-manager-config\") pod \"lws-controller-manager-56d4fb96db-zq65z\" (UID: \"3ad3bac2-deb5-4191-8edc-28e5cf1ac514\") " pod="openshift-lws-operator/lws-controller-manager-56d4fb96db-zq65z" Jul 10 00:25:26.748821 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:26.748680 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stc4v\" (UniqueName: \"kubernetes.io/projected/3ad3bac2-deb5-4191-8edc-28e5cf1ac514-kube-api-access-stc4v\") pod \"lws-controller-manager-56d4fb96db-zq65z\" (UID: \"3ad3bac2-deb5-4191-8edc-28e5cf1ac514\") " pod="openshift-lws-operator/lws-controller-manager-56d4fb96db-zq65z" Jul 10 00:25:26.748821 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:26.748762 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3ad3bac2-deb5-4191-8edc-28e5cf1ac514-metrics-cert\") pod \"lws-controller-manager-56d4fb96db-zq65z\" (UID: \"3ad3bac2-deb5-4191-8edc-28e5cf1ac514\") " pod="openshift-lws-operator/lws-controller-manager-56d4fb96db-zq65z" Jul 10 00:25:26.748821 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:26.748791 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ad3bac2-deb5-4191-8edc-28e5cf1ac514-cert\") pod \"lws-controller-manager-56d4fb96db-zq65z\" (UID: \"3ad3bac2-deb5-4191-8edc-28e5cf1ac514\") " pod="openshift-lws-operator/lws-controller-manager-56d4fb96db-zq65z" Jul 10 00:25:26.849848 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:26.849781 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3ad3bac2-deb5-4191-8edc-28e5cf1ac514-manager-config\") pod \"lws-controller-manager-56d4fb96db-zq65z\" (UID: \"3ad3bac2-deb5-4191-8edc-28e5cf1ac514\") " pod="openshift-lws-operator/lws-controller-manager-56d4fb96db-zq65z" Jul 10 00:25:26.849848 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:26.849811 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stc4v\" (UniqueName: \"kubernetes.io/projected/3ad3bac2-deb5-4191-8edc-28e5cf1ac514-kube-api-access-stc4v\") pod \"lws-controller-manager-56d4fb96db-zq65z\" (UID: \"3ad3bac2-deb5-4191-8edc-28e5cf1ac514\") " pod="openshift-lws-operator/lws-controller-manager-56d4fb96db-zq65z" Jul 10 00:25:26.850014 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:26.849860 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3ad3bac2-deb5-4191-8edc-28e5cf1ac514-metrics-cert\") pod \"lws-controller-manager-56d4fb96db-zq65z\" (UID: \"3ad3bac2-deb5-4191-8edc-28e5cf1ac514\") " pod="openshift-lws-operator/lws-controller-manager-56d4fb96db-zq65z" Jul 10 00:25:26.850014 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:26.849884 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ad3bac2-deb5-4191-8edc-28e5cf1ac514-cert\") pod \"lws-controller-manager-56d4fb96db-zq65z\" (UID: \"3ad3bac2-deb5-4191-8edc-28e5cf1ac514\") " pod="openshift-lws-operator/lws-controller-manager-56d4fb96db-zq65z" Jul 10 00:25:26.850529 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:26.850502 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3ad3bac2-deb5-4191-8edc-28e5cf1ac514-manager-config\") pod \"lws-controller-manager-56d4fb96db-zq65z\" (UID: \"3ad3bac2-deb5-4191-8edc-28e5cf1ac514\") " pod="openshift-lws-operator/lws-controller-manager-56d4fb96db-zq65z" Jul 10 00:25:26.852192 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:26.852172 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ad3bac2-deb5-4191-8edc-28e5cf1ac514-cert\") pod \"lws-controller-manager-56d4fb96db-zq65z\" (UID: \"3ad3bac2-deb5-4191-8edc-28e5cf1ac514\") " pod="openshift-lws-operator/lws-controller-manager-56d4fb96db-zq65z" Jul 10 00:25:26.852405 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:26.852386 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3ad3bac2-deb5-4191-8edc-28e5cf1ac514-metrics-cert\") pod \"lws-controller-manager-56d4fb96db-zq65z\" (UID: \"3ad3bac2-deb5-4191-8edc-28e5cf1ac514\") " pod="openshift-lws-operator/lws-controller-manager-56d4fb96db-zq65z" Jul 10 00:25:26.866659 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:26.866641 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stc4v\" (UniqueName: \"kubernetes.io/projected/3ad3bac2-deb5-4191-8edc-28e5cf1ac514-kube-api-access-stc4v\") pod \"lws-controller-manager-56d4fb96db-zq65z\" (UID: \"3ad3bac2-deb5-4191-8edc-28e5cf1ac514\") " pod="openshift-lws-operator/lws-controller-manager-56d4fb96db-zq65z" Jul 10 00:25:26.868495 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:26.868477 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-56d4fb96db-zq65z" Jul 10 00:25:26.993874 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:26.993845 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-56d4fb96db-zq65z"] Jul 10 00:25:26.997469 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:25:26.997446 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad3bac2_deb5_4191_8edc_28e5cf1ac514.slice/crio-2770c987888c7184f4dee6e9acde5af6c36a567e3af5812c3a750a06128446da WatchSource:0}: Error finding container 2770c987888c7184f4dee6e9acde5af6c36a567e3af5812c3a750a06128446da: Status 404 returned error can't find the container with id 2770c987888c7184f4dee6e9acde5af6c36a567e3af5812c3a750a06128446da Jul 10 00:25:27.188162 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:27.188068 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-56d4fb96db-zq65z" event={"ID":"3ad3bac2-deb5-4191-8edc-28e5cf1ac514","Type":"ContainerStarted","Data":"2770c987888c7184f4dee6e9acde5af6c36a567e3af5812c3a750a06128446da"} Jul 10 00:25:32.208173 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:32.208136 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-56d4fb96db-zq65z" event={"ID":"3ad3bac2-deb5-4191-8edc-28e5cf1ac514","Type":"ContainerStarted","Data":"1a9fe0b8f4318b7c14c0122fb293313be764faae6e2e4163b10148bffd7dd1e7"} Jul 10 00:25:32.208604 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:32.208265 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-56d4fb96db-zq65z" Jul 10 00:25:32.223252 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:32.223206 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-56d4fb96db-zq65z" podStartSLOduration=1.910851201 podStartE2EDuration="6.223194159s" totalImagesPullingTime="4.312342958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:25:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:25:32.222900296 +0000 UTC m=+465.155984621" watchObservedRunningTime="2026-07-10 00:25:32.223194159 +0000 UTC m=+465.156278466" Jul 10 00:25:43.213689 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:43.213653 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-56d4fb96db-zq65z" Jul 10 00:25:46.205380 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:46.205353 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85d875657f-z2vng"] Jul 10 00:25:46.208506 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:46.208490 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85d875657f-z2vng" Jul 10 00:25:46.225432 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:46.225408 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 00:25:46.225632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:46.225620 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 00:25:46.225706 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:46.225689 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 00:25:46.225736 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:46.225708 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-jktj7\"" Jul 10 00:25:46.225821 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:46.225804 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 00:25:46.256043 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:46.256020 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85d875657f-z2vng"] Jul 10 00:25:46.281850 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:46.281828 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dgwf\" (UniqueName: \"kubernetes.io/projected/5af46045-6974-437d-83ca-80cb4bcefa05-kube-api-access-5dgwf\") pod \"opendatahub-operator-controller-manager-85d875657f-z2vng\" (UID: \"5af46045-6974-437d-83ca-80cb4bcefa05\") " pod="opendatahub/opendatahub-operator-controller-manager-85d875657f-z2vng" Jul 10 00:25:46.281961 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:46.281862 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5af46045-6974-437d-83ca-80cb4bcefa05-tmp\") pod \"opendatahub-operator-controller-manager-85d875657f-z2vng\" (UID: \"5af46045-6974-437d-83ca-80cb4bcefa05\") " pod="opendatahub/opendatahub-operator-controller-manager-85d875657f-z2vng" Jul 10 00:25:46.281961 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:46.281882 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5af46045-6974-437d-83ca-80cb4bcefa05-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85d875657f-z2vng\" (UID: \"5af46045-6974-437d-83ca-80cb4bcefa05\") " pod="opendatahub/opendatahub-operator-controller-manager-85d875657f-z2vng" Jul 10 00:25:46.281961 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:46.281923 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5af46045-6974-437d-83ca-80cb4bcefa05-manifests\") pod \"opendatahub-operator-controller-manager-85d875657f-z2vng\" (UID: \"5af46045-6974-437d-83ca-80cb4bcefa05\") " pod="opendatahub/opendatahub-operator-controller-manager-85d875657f-z2vng" Jul 10 00:25:46.281961 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:46.281942 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5af46045-6974-437d-83ca-80cb4bcefa05-webhook-cert\") pod \"opendatahub-operator-controller-manager-85d875657f-z2vng\" (UID: \"5af46045-6974-437d-83ca-80cb4bcefa05\") " pod="opendatahub/opendatahub-operator-controller-manager-85d875657f-z2vng" Jul 10 00:25:46.382729 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:46.382695 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5af46045-6974-437d-83ca-80cb4bcefa05-tmp\") pod \"opendatahub-operator-controller-manager-85d875657f-z2vng\" (UID: \"5af46045-6974-437d-83ca-80cb4bcefa05\") " pod="opendatahub/opendatahub-operator-controller-manager-85d875657f-z2vng" Jul 10 00:25:46.382846 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:46.382732 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5af46045-6974-437d-83ca-80cb4bcefa05-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85d875657f-z2vng\" (UID: \"5af46045-6974-437d-83ca-80cb4bcefa05\") " pod="opendatahub/opendatahub-operator-controller-manager-85d875657f-z2vng" Jul 10 00:25:46.382846 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:46.382828 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5af46045-6974-437d-83ca-80cb4bcefa05-manifests\") pod \"opendatahub-operator-controller-manager-85d875657f-z2vng\" (UID: \"5af46045-6974-437d-83ca-80cb4bcefa05\") " pod="opendatahub/opendatahub-operator-controller-manager-85d875657f-z2vng" Jul 10 00:25:46.382923 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:46.382853 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5af46045-6974-437d-83ca-80cb4bcefa05-webhook-cert\") pod \"opendatahub-operator-controller-manager-85d875657f-z2vng\" (UID: \"5af46045-6974-437d-83ca-80cb4bcefa05\") " pod="opendatahub/opendatahub-operator-controller-manager-85d875657f-z2vng" Jul 10 00:25:46.382923 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:46.382916 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dgwf\" (UniqueName: \"kubernetes.io/projected/5af46045-6974-437d-83ca-80cb4bcefa05-kube-api-access-5dgwf\") pod \"opendatahub-operator-controller-manager-85d875657f-z2vng\" (UID: \"5af46045-6974-437d-83ca-80cb4bcefa05\") " pod="opendatahub/opendatahub-operator-controller-manager-85d875657f-z2vng" Jul 10 00:25:46.383195 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:46.383165 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5af46045-6974-437d-83ca-80cb4bcefa05-manifests\") pod \"opendatahub-operator-controller-manager-85d875657f-z2vng\" (UID: \"5af46045-6974-437d-83ca-80cb4bcefa05\") " pod="opendatahub/opendatahub-operator-controller-manager-85d875657f-z2vng" Jul 10 00:25:46.385038 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:46.385011 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5af46045-6974-437d-83ca-80cb4bcefa05-tmp\") pod \"opendatahub-operator-controller-manager-85d875657f-z2vng\" (UID: \"5af46045-6974-437d-83ca-80cb4bcefa05\") " pod="opendatahub/opendatahub-operator-controller-manager-85d875657f-z2vng" Jul 10 00:25:46.385156 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:46.385137 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5af46045-6974-437d-83ca-80cb4bcefa05-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85d875657f-z2vng\" (UID: \"5af46045-6974-437d-83ca-80cb4bcefa05\") " pod="opendatahub/opendatahub-operator-controller-manager-85d875657f-z2vng" Jul 10 00:25:46.385280 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:46.385263 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5af46045-6974-437d-83ca-80cb4bcefa05-webhook-cert\") pod \"opendatahub-operator-controller-manager-85d875657f-z2vng\" (UID: \"5af46045-6974-437d-83ca-80cb4bcefa05\") " pod="opendatahub/opendatahub-operator-controller-manager-85d875657f-z2vng" Jul 10 00:25:46.389710 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:46.389688 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dgwf\" (UniqueName: \"kubernetes.io/projected/5af46045-6974-437d-83ca-80cb4bcefa05-kube-api-access-5dgwf\") pod \"opendatahub-operator-controller-manager-85d875657f-z2vng\" (UID: \"5af46045-6974-437d-83ca-80cb4bcefa05\") " pod="opendatahub/opendatahub-operator-controller-manager-85d875657f-z2vng" Jul 10 00:25:46.517972 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:46.517940 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85d875657f-z2vng" Jul 10 00:25:46.649196 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:46.649171 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85d875657f-z2vng"] Jul 10 00:25:46.651578 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:25:46.651553 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af46045_6974_437d_83ca_80cb4bcefa05.slice/crio-808f7ce5952bbfaedb3e6f6dd915adab8712561dc548e3dfb50144a7455a3f99 WatchSource:0}: Error finding container 808f7ce5952bbfaedb3e6f6dd915adab8712561dc548e3dfb50144a7455a3f99: Status 404 returned error can't find the container with id 808f7ce5952bbfaedb3e6f6dd915adab8712561dc548e3dfb50144a7455a3f99 Jul 10 00:25:47.262427 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:47.262389 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85d875657f-z2vng" event={"ID":"5af46045-6974-437d-83ca-80cb4bcefa05","Type":"ContainerStarted","Data":"808f7ce5952bbfaedb3e6f6dd915adab8712561dc548e3dfb50144a7455a3f99"} Jul 10 00:25:50.275438 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:50.275408 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85d875657f-z2vng" event={"ID":"5af46045-6974-437d-83ca-80cb4bcefa05","Type":"ContainerStarted","Data":"6b0229cf54d7f3b80c134a5722a9e29dcababbfd394390623452f7c9ecfb139e"} Jul 10 00:25:51.280506 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:51.280470 2559 generic.go:356] "Generic (PLEG): container finished" podID="5af46045-6974-437d-83ca-80cb4bcefa05" containerID="6b0229cf54d7f3b80c134a5722a9e29dcababbfd394390623452f7c9ecfb139e" exitCode=0 Jul 10 00:25:51.280506 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:51.280507 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85d875657f-z2vng" event={"ID":"5af46045-6974-437d-83ca-80cb4bcefa05","Type":"ContainerDied","Data":"6b0229cf54d7f3b80c134a5722a9e29dcababbfd394390623452f7c9ecfb139e"} Jul 10 00:25:52.286175 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:52.286136 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85d875657f-z2vng" event={"ID":"5af46045-6974-437d-83ca-80cb4bcefa05","Type":"ContainerStarted","Data":"f154b4661df15f58585cbf4f792be63b1f1f50029a460bc3395a3b533db08c1c"} Jul 10 00:25:52.286581 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:52.286254 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-85d875657f-z2vng" Jul 10 00:25:52.308663 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:25:52.308605 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-85d875657f-z2vng" podStartSLOduration=1.8769194759999999 podStartE2EDuration="6.308589919s" totalImagesPullingTime="3.374933722s" totalInitContainerRuntime="1.056736721s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:25:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:25:52.306441731 +0000 UTC m=+485.239526039" watchObservedRunningTime="2026-07-10 00:25:52.308589919 +0000 UTC m=+485.241674297" Jul 10 00:26:03.292453 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:03.292422 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-85d875657f-z2vng" Jul 10 00:26:23.310339 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.310307 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87"] Jul 10 00:26:23.313324 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.313303 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.315449 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.315423 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 00:26:23.315548 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.315477 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 00:26:23.315548 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.315493 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 00:26:23.315548 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.315434 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-zqggv\"" Jul 10 00:26:23.324289 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.324269 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87"] Jul 10 00:26:23.366686 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.366665 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n7c4\" (UniqueName: \"kubernetes.io/projected/84663b2e-6507-4aac-8ad4-db9fa1621614-kube-api-access-8n7c4\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xnd87\" (UID: \"84663b2e-6507-4aac-8ad4-db9fa1621614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.366779 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.366709 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84663b2e-6507-4aac-8ad4-db9fa1621614-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xnd87\" (UID: \"84663b2e-6507-4aac-8ad4-db9fa1621614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.366779 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.366726 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84663b2e-6507-4aac-8ad4-db9fa1621614-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xnd87\" (UID: \"84663b2e-6507-4aac-8ad4-db9fa1621614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.366852 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.366787 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84663b2e-6507-4aac-8ad4-db9fa1621614-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xnd87\" (UID: \"84663b2e-6507-4aac-8ad4-db9fa1621614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.366852 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.366821 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84663b2e-6507-4aac-8ad4-db9fa1621614-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xnd87\" (UID: \"84663b2e-6507-4aac-8ad4-db9fa1621614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.366852 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.366845 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84663b2e-6507-4aac-8ad4-db9fa1621614-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xnd87\" (UID: \"84663b2e-6507-4aac-8ad4-db9fa1621614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.366941 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.366867 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84663b2e-6507-4aac-8ad4-db9fa1621614-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xnd87\" (UID: \"84663b2e-6507-4aac-8ad4-db9fa1621614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.366941 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.366924 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84663b2e-6507-4aac-8ad4-db9fa1621614-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xnd87\" (UID: \"84663b2e-6507-4aac-8ad4-db9fa1621614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.367002 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.366940 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84663b2e-6507-4aac-8ad4-db9fa1621614-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xnd87\" (UID: \"84663b2e-6507-4aac-8ad4-db9fa1621614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.467640 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.467608 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84663b2e-6507-4aac-8ad4-db9fa1621614-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xnd87\" (UID: \"84663b2e-6507-4aac-8ad4-db9fa1621614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.467784 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.467645 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84663b2e-6507-4aac-8ad4-db9fa1621614-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xnd87\" (UID: \"84663b2e-6507-4aac-8ad4-db9fa1621614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.467784 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.467666 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n7c4\" (UniqueName: \"kubernetes.io/projected/84663b2e-6507-4aac-8ad4-db9fa1621614-kube-api-access-8n7c4\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xnd87\" (UID: \"84663b2e-6507-4aac-8ad4-db9fa1621614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.467784 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.467714 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84663b2e-6507-4aac-8ad4-db9fa1621614-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xnd87\" (UID: \"84663b2e-6507-4aac-8ad4-db9fa1621614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.467784 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.467729 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84663b2e-6507-4aac-8ad4-db9fa1621614-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xnd87\" (UID: \"84663b2e-6507-4aac-8ad4-db9fa1621614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.467784 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.467749 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84663b2e-6507-4aac-8ad4-db9fa1621614-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xnd87\" (UID: \"84663b2e-6507-4aac-8ad4-db9fa1621614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.467784 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.467774 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84663b2e-6507-4aac-8ad4-db9fa1621614-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xnd87\" (UID: \"84663b2e-6507-4aac-8ad4-db9fa1621614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.468071 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.467800 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84663b2e-6507-4aac-8ad4-db9fa1621614-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xnd87\" (UID: \"84663b2e-6507-4aac-8ad4-db9fa1621614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.468071 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.467825 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84663b2e-6507-4aac-8ad4-db9fa1621614-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xnd87\" (UID: \"84663b2e-6507-4aac-8ad4-db9fa1621614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.468280 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.468261 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84663b2e-6507-4aac-8ad4-db9fa1621614-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xnd87\" (UID: \"84663b2e-6507-4aac-8ad4-db9fa1621614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.468372 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.468352 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84663b2e-6507-4aac-8ad4-db9fa1621614-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xnd87\" (UID: \"84663b2e-6507-4aac-8ad4-db9fa1621614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.468444 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.468421 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84663b2e-6507-4aac-8ad4-db9fa1621614-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xnd87\" (UID: \"84663b2e-6507-4aac-8ad4-db9fa1621614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.468497 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.468458 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84663b2e-6507-4aac-8ad4-db9fa1621614-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xnd87\" (UID: \"84663b2e-6507-4aac-8ad4-db9fa1621614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.468640 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.468621 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84663b2e-6507-4aac-8ad4-db9fa1621614-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xnd87\" (UID: \"84663b2e-6507-4aac-8ad4-db9fa1621614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.470246 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.470224 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84663b2e-6507-4aac-8ad4-db9fa1621614-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xnd87\" (UID: \"84663b2e-6507-4aac-8ad4-db9fa1621614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.470246 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.470242 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84663b2e-6507-4aac-8ad4-db9fa1621614-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xnd87\" (UID: \"84663b2e-6507-4aac-8ad4-db9fa1621614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.474647 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.474622 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84663b2e-6507-4aac-8ad4-db9fa1621614-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xnd87\" (UID: \"84663b2e-6507-4aac-8ad4-db9fa1621614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.475041 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.475018 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n7c4\" (UniqueName: \"kubernetes.io/projected/84663b2e-6507-4aac-8ad4-db9fa1621614-kube-api-access-8n7c4\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4xnd87\" (UID: \"84663b2e-6507-4aac-8ad4-db9fa1621614\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.625472 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.625399 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:23.763012 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:23.762987 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87"] Jul 10 00:26:23.765467 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:26:23.765445 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84663b2e_6507_4aac_8ad4_db9fa1621614.slice/crio-aefec8d17c174f97a07629b7d90e50e229b5c69e04b5dc231b3012ed88c742ac WatchSource:0}: Error finding container aefec8d17c174f97a07629b7d90e50e229b5c69e04b5dc231b3012ed88c742ac: Status 404 returned error can't find the container with id aefec8d17c174f97a07629b7d90e50e229b5c69e04b5dc231b3012ed88c742ac Jul 10 00:26:24.407461 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:24.407427 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" event={"ID":"84663b2e-6507-4aac-8ad4-db9fa1621614","Type":"ContainerStarted","Data":"aefec8d17c174f97a07629b7d90e50e229b5c69e04b5dc231b3012ed88c742ac"} Jul 10 00:26:26.784415 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:26.784383 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:26:26.784664 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:26.784451 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:26:26.784664 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:26.784479 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:26:27.420412 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:27.420378 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" event={"ID":"84663b2e-6507-4aac-8ad4-db9fa1621614","Type":"ContainerStarted","Data":"f5de0a463d70b941109b7c6223d49172cdc66da9c93e362e96d954f8cf477bce"} Jul 10 00:26:27.444992 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:27.440757 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" podStartSLOduration=1.423840326 podStartE2EDuration="4.44073865s" totalImagesPullingTime="3.016898324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:26:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:26:27.438526134 +0000 UTC m=+520.371610442" watchObservedRunningTime="2026-07-10 00:26:27.44073865 +0000 UTC m=+520.373822960" Jul 10 00:26:27.626464 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:27.626435 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:27.630672 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:27.630651 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:28.424482 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:28.424448 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:26:28.425347 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:26:28.425326 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4xnd87" Jul 10 00:27:07.062317 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:07.062276 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jvl6j"] Jul 10 00:27:07.072517 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:07.072488 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jvl6j" Jul 10 00:27:07.074617 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:07.074590 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jvl6j"] Jul 10 00:27:07.074939 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:07.074912 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-25645\"" Jul 10 00:27:07.075055 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:07.075034 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 00:27:07.075722 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:07.075704 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 00:27:07.083250 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:07.083227 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w2sc\" (UniqueName: \"kubernetes.io/projected/c653b777-f4fb-496d-bfd2-a705c804d897-kube-api-access-2w2sc\") pod \"kuadrant-operator-catalog-jvl6j\" (UID: \"c653b777-f4fb-496d-bfd2-a705c804d897\") " pod="kuadrant-system/kuadrant-operator-catalog-jvl6j" Jul 10 00:27:07.184440 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:07.184413 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w2sc\" (UniqueName: \"kubernetes.io/projected/c653b777-f4fb-496d-bfd2-a705c804d897-kube-api-access-2w2sc\") pod \"kuadrant-operator-catalog-jvl6j\" (UID: \"c653b777-f4fb-496d-bfd2-a705c804d897\") " pod="kuadrant-system/kuadrant-operator-catalog-jvl6j" Jul 10 00:27:07.191631 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:07.191607 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w2sc\" (UniqueName: \"kubernetes.io/projected/c653b777-f4fb-496d-bfd2-a705c804d897-kube-api-access-2w2sc\") pod \"kuadrant-operator-catalog-jvl6j\" (UID: \"c653b777-f4fb-496d-bfd2-a705c804d897\") " pod="kuadrant-system/kuadrant-operator-catalog-jvl6j" Jul 10 00:27:07.382460 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:07.382376 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jvl6j" Jul 10 00:27:07.429902 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:07.429867 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jvl6j"] Jul 10 00:27:07.508841 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:07.508810 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jvl6j"] Jul 10 00:27:07.511677 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:27:07.511648 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc653b777_f4fb_496d_bfd2_a705c804d897.slice/crio-75fdd1f6f7943b2413e5b69ca78058884099cf5f5d39d86bc6ea36ae4a9eaaca WatchSource:0}: Error finding container 75fdd1f6f7943b2413e5b69ca78058884099cf5f5d39d86bc6ea36ae4a9eaaca: Status 404 returned error can't find the container with id 75fdd1f6f7943b2413e5b69ca78058884099cf5f5d39d86bc6ea36ae4a9eaaca Jul 10 00:27:07.566454 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:07.566423 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jvl6j" event={"ID":"c653b777-f4fb-496d-bfd2-a705c804d897","Type":"ContainerStarted","Data":"75fdd1f6f7943b2413e5b69ca78058884099cf5f5d39d86bc6ea36ae4a9eaaca"} Jul 10 00:27:07.640000 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:07.639937 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cljqj"] Jul 10 00:27:07.645387 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:07.645370 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-cljqj" Jul 10 00:27:07.650755 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:07.650727 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cljqj"] Jul 10 00:27:07.687857 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:07.687833 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssfns\" (UniqueName: \"kubernetes.io/projected/2f80f79a-3264-4b3e-8fa7-a6166013b333-kube-api-access-ssfns\") pod \"kuadrant-operator-catalog-cljqj\" (UID: \"2f80f79a-3264-4b3e-8fa7-a6166013b333\") " pod="kuadrant-system/kuadrant-operator-catalog-cljqj" Jul 10 00:27:07.788369 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:07.788341 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssfns\" (UniqueName: \"kubernetes.io/projected/2f80f79a-3264-4b3e-8fa7-a6166013b333-kube-api-access-ssfns\") pod \"kuadrant-operator-catalog-cljqj\" (UID: \"2f80f79a-3264-4b3e-8fa7-a6166013b333\") " pod="kuadrant-system/kuadrant-operator-catalog-cljqj" Jul 10 00:27:07.795558 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:07.795533 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssfns\" (UniqueName: \"kubernetes.io/projected/2f80f79a-3264-4b3e-8fa7-a6166013b333-kube-api-access-ssfns\") pod \"kuadrant-operator-catalog-cljqj\" (UID: \"2f80f79a-3264-4b3e-8fa7-a6166013b333\") " pod="kuadrant-system/kuadrant-operator-catalog-cljqj" Jul 10 00:27:07.955586 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:07.955504 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-cljqj" Jul 10 00:27:08.077575 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:08.077549 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-cljqj"] Jul 10 00:27:08.125636 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:27:08.125606 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f80f79a_3264_4b3e_8fa7_a6166013b333.slice/crio-36586606f0c0bc88be75317edc7efd316e9dcd102e9d93784a7faf7edd436e36 WatchSource:0}: Error finding container 36586606f0c0bc88be75317edc7efd316e9dcd102e9d93784a7faf7edd436e36: Status 404 returned error can't find the container with id 36586606f0c0bc88be75317edc7efd316e9dcd102e9d93784a7faf7edd436e36 Jul 10 00:27:08.573139 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:08.573088 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-cljqj" event={"ID":"2f80f79a-3264-4b3e-8fa7-a6166013b333","Type":"ContainerStarted","Data":"36586606f0c0bc88be75317edc7efd316e9dcd102e9d93784a7faf7edd436e36"} Jul 10 00:27:10.582975 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:10.582922 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-cljqj" event={"ID":"2f80f79a-3264-4b3e-8fa7-a6166013b333","Type":"ContainerStarted","Data":"9ec71a3e091261573aafe189024dc84fba77b8035c929390d5e6463b0db00cee"} Jul 10 00:27:10.584512 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:10.584487 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jvl6j" event={"ID":"c653b777-f4fb-496d-bfd2-a705c804d897","Type":"ContainerStarted","Data":"5fcdb49ff4a0ecce6001c402327bc1fab41b93aa5a841b83476f9cb5b03c3925"} Jul 10 00:27:10.584622 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:10.584551 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-jvl6j" podUID="c653b777-f4fb-496d-bfd2-a705c804d897" containerName="registry-server" containerID="cri-o://5fcdb49ff4a0ecce6001c402327bc1fab41b93aa5a841b83476f9cb5b03c3925" gracePeriod=2 Jul 10 00:27:10.600082 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:10.600044 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-cljqj" podStartSLOduration=1.7732325599999998 podStartE2EDuration="3.60003071s" totalImagesPullingTime="1.82679815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:27:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:27:10.59836518 +0000 UTC m=+563.531449516" watchObservedRunningTime="2026-07-10 00:27:10.60003071 +0000 UTC m=+563.533115050" Jul 10 00:27:10.611033 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:10.610992 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-jvl6j" podStartSLOduration=1.166565901 podStartE2EDuration="3.610974383s" totalImagesPullingTime="2.444408482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:27:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:27:10.610770599 +0000 UTC m=+563.543854932" watchObservedRunningTime="2026-07-10 00:27:10.610974383 +0000 UTC m=+563.544058693" Jul 10 00:27:10.822202 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:10.822173 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jvl6j" Jul 10 00:27:10.911166 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:10.911068 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2w2sc\" (UniqueName: \"kubernetes.io/projected/c653b777-f4fb-496d-bfd2-a705c804d897-kube-api-access-2w2sc\") pod \"c653b777-f4fb-496d-bfd2-a705c804d897\" (UID: \"c653b777-f4fb-496d-bfd2-a705c804d897\") " Jul 10 00:27:10.913193 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:10.913169 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c653b777-f4fb-496d-bfd2-a705c804d897-kube-api-access-2w2sc" (OuterVolumeSpecName: "kube-api-access-2w2sc") pod "c653b777-f4fb-496d-bfd2-a705c804d897" (UID: "c653b777-f4fb-496d-bfd2-a705c804d897"). InnerVolumeSpecName "kube-api-access-2w2sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:27:11.011675 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:11.011641 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w2sc\" (UniqueName: \"kubernetes.io/projected/c653b777-f4fb-496d-bfd2-a705c804d897-kube-api-access-2w2sc\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:27:11.589712 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:11.589682 2559 generic.go:356] "Generic (PLEG): container finished" podID="c653b777-f4fb-496d-bfd2-a705c804d897" containerID="5fcdb49ff4a0ecce6001c402327bc1fab41b93aa5a841b83476f9cb5b03c3925" exitCode=0 Jul 10 00:27:11.590106 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:11.589731 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jvl6j" Jul 10 00:27:11.590106 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:11.589764 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jvl6j" event={"ID":"c653b777-f4fb-496d-bfd2-a705c804d897","Type":"ContainerDied","Data":"5fcdb49ff4a0ecce6001c402327bc1fab41b93aa5a841b83476f9cb5b03c3925"} Jul 10 00:27:11.590106 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:11.589811 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jvl6j" event={"ID":"c653b777-f4fb-496d-bfd2-a705c804d897","Type":"ContainerDied","Data":"75fdd1f6f7943b2413e5b69ca78058884099cf5f5d39d86bc6ea36ae4a9eaaca"} Jul 10 00:27:11.590106 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:11.589834 2559 scope.go:117] "RemoveContainer" containerID="5fcdb49ff4a0ecce6001c402327bc1fab41b93aa5a841b83476f9cb5b03c3925" Jul 10 00:27:11.598448 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:11.598424 2559 scope.go:117] "RemoveContainer" containerID="5fcdb49ff4a0ecce6001c402327bc1fab41b93aa5a841b83476f9cb5b03c3925" Jul 10 00:27:11.598705 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:27:11.598688 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fcdb49ff4a0ecce6001c402327bc1fab41b93aa5a841b83476f9cb5b03c3925\": container with ID starting with 5fcdb49ff4a0ecce6001c402327bc1fab41b93aa5a841b83476f9cb5b03c3925 not found: ID does not exist" containerID="5fcdb49ff4a0ecce6001c402327bc1fab41b93aa5a841b83476f9cb5b03c3925" Jul 10 00:27:11.598747 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:11.598714 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fcdb49ff4a0ecce6001c402327bc1fab41b93aa5a841b83476f9cb5b03c3925"} err="failed to get container status \"5fcdb49ff4a0ecce6001c402327bc1fab41b93aa5a841b83476f9cb5b03c3925\": rpc error: code = NotFound desc = could not find container \"5fcdb49ff4a0ecce6001c402327bc1fab41b93aa5a841b83476f9cb5b03c3925\": container with ID starting with 5fcdb49ff4a0ecce6001c402327bc1fab41b93aa5a841b83476f9cb5b03c3925 not found: ID does not exist" Jul 10 00:27:11.609268 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:11.609237 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jvl6j"] Jul 10 00:27:11.612843 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:11.612821 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jvl6j"] Jul 10 00:27:11.649529 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:11.649504 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c653b777-f4fb-496d-bfd2-a705c804d897" path="/var/lib/kubelet/pods/c653b777-f4fb-496d-bfd2-a705c804d897/volumes" Jul 10 00:27:17.955668 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:17.955632 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-cljqj" Jul 10 00:27:17.956074 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:17.955690 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-cljqj" Jul 10 00:27:17.977919 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:17.977895 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-cljqj" Jul 10 00:27:18.640509 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:18.640485 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-cljqj" Jul 10 00:27:40.269661 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:40.269608 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt"] Jul 10 00:27:40.270108 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:40.269907 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c653b777-f4fb-496d-bfd2-a705c804d897" containerName="registry-server" Jul 10 00:27:40.270108 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:40.269918 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c653b777-f4fb-496d-bfd2-a705c804d897" containerName="registry-server" Jul 10 00:27:40.270108 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:40.269963 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c653b777-f4fb-496d-bfd2-a705c804d897" containerName="registry-server" Jul 10 00:27:40.274639 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:40.274622 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt" Jul 10 00:27:40.277225 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:40.277202 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jzj9p\"" Jul 10 00:27:40.293612 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:40.293589 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt"] Jul 10 00:27:40.329906 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:40.329877 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c548b776-2053-4095-a7eb-5f6272eceacf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kv9kt\" (UID: \"c548b776-2053-4095-a7eb-5f6272eceacf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt" Jul 10 00:27:40.330046 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:40.329926 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7xdz\" (UniqueName: \"kubernetes.io/projected/c548b776-2053-4095-a7eb-5f6272eceacf-kube-api-access-b7xdz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kv9kt\" (UID: \"c548b776-2053-4095-a7eb-5f6272eceacf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt" Jul 10 00:27:40.430743 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:40.430713 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7xdz\" (UniqueName: \"kubernetes.io/projected/c548b776-2053-4095-a7eb-5f6272eceacf-kube-api-access-b7xdz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kv9kt\" (UID: \"c548b776-2053-4095-a7eb-5f6272eceacf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt" Jul 10 00:27:40.430854 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:40.430778 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c548b776-2053-4095-a7eb-5f6272eceacf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kv9kt\" (UID: \"c548b776-2053-4095-a7eb-5f6272eceacf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt" Jul 10 00:27:40.431180 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:40.431162 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c548b776-2053-4095-a7eb-5f6272eceacf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kv9kt\" (UID: \"c548b776-2053-4095-a7eb-5f6272eceacf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt" Jul 10 00:27:40.444684 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:40.444653 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7xdz\" (UniqueName: \"kubernetes.io/projected/c548b776-2053-4095-a7eb-5f6272eceacf-kube-api-access-b7xdz\") pod \"kuadrant-operator-controller-manager-8684c9ff49-kv9kt\" (UID: \"c548b776-2053-4095-a7eb-5f6272eceacf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt" Jul 10 00:27:40.584917 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:40.584842 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt" Jul 10 00:27:40.710511 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:40.710439 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt"] Jul 10 00:27:40.713381 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:27:40.713357 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc548b776_2053_4095_a7eb_5f6272eceacf.slice/crio-dca235f0a3573259e2272c3c9ccdb7da7c24cc2d7f103693cfe52cae0bbaac3b WatchSource:0}: Error finding container dca235f0a3573259e2272c3c9ccdb7da7c24cc2d7f103693cfe52cae0bbaac3b: Status 404 returned error can't find the container with id dca235f0a3573259e2272c3c9ccdb7da7c24cc2d7f103693cfe52cae0bbaac3b Jul 10 00:27:41.710454 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:41.710419 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt" event={"ID":"c548b776-2053-4095-a7eb-5f6272eceacf","Type":"ContainerStarted","Data":"dca235f0a3573259e2272c3c9ccdb7da7c24cc2d7f103693cfe52cae0bbaac3b"} Jul 10 00:27:45.133681 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:45.133640 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-vtg5w"] Jul 10 00:27:45.137038 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:45.137015 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vtg5w" Jul 10 00:27:45.139375 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:45.139354 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 10 00:27:45.139641 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:45.139615 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-pmrt9\"" Jul 10 00:27:45.146085 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:45.146027 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-vtg5w"] Jul 10 00:27:45.273183 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:45.273142 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27s2h\" (UniqueName: \"kubernetes.io/projected/62fd95d9-a931-4eb7-a4db-2cceb02cbd7b-kube-api-access-27s2h\") pod \"dns-operator-controller-manager-864dd78c4c-vtg5w\" (UID: \"62fd95d9-a931-4eb7-a4db-2cceb02cbd7b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vtg5w" Jul 10 00:27:45.374148 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:45.374091 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27s2h\" (UniqueName: \"kubernetes.io/projected/62fd95d9-a931-4eb7-a4db-2cceb02cbd7b-kube-api-access-27s2h\") pod \"dns-operator-controller-manager-864dd78c4c-vtg5w\" (UID: \"62fd95d9-a931-4eb7-a4db-2cceb02cbd7b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vtg5w" Jul 10 00:27:45.390967 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:45.390887 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27s2h\" (UniqueName: \"kubernetes.io/projected/62fd95d9-a931-4eb7-a4db-2cceb02cbd7b-kube-api-access-27s2h\") pod \"dns-operator-controller-manager-864dd78c4c-vtg5w\" (UID: \"62fd95d9-a931-4eb7-a4db-2cceb02cbd7b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vtg5w" Jul 10 00:27:45.450489 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:45.450457 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vtg5w" Jul 10 00:27:46.321671 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:46.321619 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-vtg5w"] Jul 10 00:27:46.324616 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:27:46.324584 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62fd95d9_a931_4eb7_a4db_2cceb02cbd7b.slice/crio-80ab031e1a0ff8f35944f4d908437161f964e0d6034bedc0af1ad91da7a19a9a WatchSource:0}: Error finding container 80ab031e1a0ff8f35944f4d908437161f964e0d6034bedc0af1ad91da7a19a9a: Status 404 returned error can't find the container with id 80ab031e1a0ff8f35944f4d908437161f964e0d6034bedc0af1ad91da7a19a9a Jul 10 00:27:46.733051 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:46.733011 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vtg5w" event={"ID":"62fd95d9-a931-4eb7-a4db-2cceb02cbd7b","Type":"ContainerStarted","Data":"80ab031e1a0ff8f35944f4d908437161f964e0d6034bedc0af1ad91da7a19a9a"} Jul 10 00:27:46.734413 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:46.734384 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt" event={"ID":"c548b776-2053-4095-a7eb-5f6272eceacf","Type":"ContainerStarted","Data":"80243e7c3193ac0c23f320f2798069aab17962f073f872615690baa16c2f32d0"} Jul 10 00:27:46.734533 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:46.734505 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt" Jul 10 00:27:46.752322 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:46.752252 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt" podStartSLOduration=1.209953805 podStartE2EDuration="6.752238231s" totalImagesPullingTime="5.542284426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:27:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:27:46.75087102 +0000 UTC m=+599.683955342" watchObservedRunningTime="2026-07-10 00:27:46.752238231 +0000 UTC m=+599.685322539" Jul 10 00:27:47.501867 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:47.501842 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7zrr_20198eb1-6b3e-4e30-9adf-d4f13df4ee3b/ovn-acl-logging/0.log" Jul 10 00:27:47.502212 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:47.501844 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7zrr_20198eb1-6b3e-4e30-9adf-d4f13df4ee3b/ovn-acl-logging/0.log" Jul 10 00:27:47.526103 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:47.526077 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzb7q"] Jul 10 00:27:47.530266 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:47.530252 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzb7q" Jul 10 00:27:47.532453 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:47.532429 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-jdxtx\"" Jul 10 00:27:47.546429 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:47.546406 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzb7q"] Jul 10 00:27:47.592067 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:47.592032 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqjvt\" (UniqueName: \"kubernetes.io/projected/fff18a8e-7dc4-4fce-8c3a-e928d67cd625-kube-api-access-bqjvt\") pod \"limitador-operator-controller-manager-859d7b78bd-pzb7q\" (UID: \"fff18a8e-7dc4-4fce-8c3a-e928d67cd625\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzb7q" Jul 10 00:27:47.692730 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:47.692692 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqjvt\" (UniqueName: \"kubernetes.io/projected/fff18a8e-7dc4-4fce-8c3a-e928d67cd625-kube-api-access-bqjvt\") pod \"limitador-operator-controller-manager-859d7b78bd-pzb7q\" (UID: \"fff18a8e-7dc4-4fce-8c3a-e928d67cd625\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzb7q" Jul 10 00:27:47.708377 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:47.703764 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqjvt\" (UniqueName: \"kubernetes.io/projected/fff18a8e-7dc4-4fce-8c3a-e928d67cd625-kube-api-access-bqjvt\") pod \"limitador-operator-controller-manager-859d7b78bd-pzb7q\" (UID: \"fff18a8e-7dc4-4fce-8c3a-e928d67cd625\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzb7q" Jul 10 00:27:47.841865 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:47.841829 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzb7q" Jul 10 00:27:47.985995 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:47.985964 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzb7q"] Jul 10 00:27:48.743382 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:48.743352 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzb7q" event={"ID":"fff18a8e-7dc4-4fce-8c3a-e928d67cd625","Type":"ContainerStarted","Data":"236cc80b156fe0ef4f5d43954f4fd416df50dff6e43b42387e16919bd0d9f746"} Jul 10 00:27:48.789878 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:48.789852 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 10 00:27:49.752209 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:49.752176 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vtg5w" event={"ID":"62fd95d9-a931-4eb7-a4db-2cceb02cbd7b","Type":"ContainerStarted","Data":"12ab0ee38c9d9465b1b06003e4919bf8f62eddd4248cdf62371c521acfad9fc8"} Jul 10 00:27:49.752609 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:49.752339 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vtg5w" Jul 10 00:27:49.770865 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:49.770820 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vtg5w" podStartSLOduration=2.310409825 podStartE2EDuration="4.77080289s" totalImagesPullingTime="2.460393065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:27:49.770398201 +0000 UTC m=+602.703482510" watchObservedRunningTime="2026-07-10 00:27:49.77080289 +0000 UTC m=+602.703887199" Jul 10 00:27:50.758586 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:50.758550 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzb7q" event={"ID":"fff18a8e-7dc4-4fce-8c3a-e928d67cd625","Type":"ContainerStarted","Data":"2221b82c32349b3ff0af9210b5d8d040005815a4f1b5e1afe4884f04e2a38462"} Jul 10 00:27:50.758968 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:50.758797 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzb7q" Jul 10 00:27:50.774697 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:50.774643 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzb7q" podStartSLOduration=1.216148447 podStartE2EDuration="3.774627182s" totalImagesPullingTime="2.558478735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:27:50.774088316 +0000 UTC m=+603.707172643" watchObservedRunningTime="2026-07-10 00:27:50.774627182 +0000 UTC m=+603.707711482" Jul 10 00:27:57.740196 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:57.740167 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt" Jul 10 00:27:58.985185 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:58.985152 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj"] Jul 10 00:27:58.988298 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:58.988279 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj" Jul 10 00:27:59.002898 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.002875 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj"] Jul 10 00:27:59.094261 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.094236 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2j9s\" (UniqueName: \"kubernetes.io/projected/3b9636ca-9bf1-4473-81e8-e27d0d854b65-kube-api-access-s2j9s\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-d2rkj\" (UID: \"3b9636ca-9bf1-4473-81e8-e27d0d854b65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj" Jul 10 00:27:59.094403 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.094300 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b9636ca-9bf1-4473-81e8-e27d0d854b65-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-d2rkj\" (UID: \"3b9636ca-9bf1-4473-81e8-e27d0d854b65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj" Jul 10 00:27:59.195323 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.195294 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2j9s\" (UniqueName: \"kubernetes.io/projected/3b9636ca-9bf1-4473-81e8-e27d0d854b65-kube-api-access-s2j9s\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-d2rkj\" (UID: \"3b9636ca-9bf1-4473-81e8-e27d0d854b65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj" Jul 10 00:27:59.195445 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.195353 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b9636ca-9bf1-4473-81e8-e27d0d854b65-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-d2rkj\" (UID: \"3b9636ca-9bf1-4473-81e8-e27d0d854b65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj" Jul 10 00:27:59.195686 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.195667 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b9636ca-9bf1-4473-81e8-e27d0d854b65-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-d2rkj\" (UID: \"3b9636ca-9bf1-4473-81e8-e27d0d854b65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj" Jul 10 00:27:59.203290 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.203263 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2j9s\" (UniqueName: \"kubernetes.io/projected/3b9636ca-9bf1-4473-81e8-e27d0d854b65-kube-api-access-s2j9s\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-d2rkj\" (UID: \"3b9636ca-9bf1-4473-81e8-e27d0d854b65\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj" Jul 10 00:27:59.297777 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.297752 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj" Jul 10 00:27:59.429488 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.429465 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj"] Jul 10 00:27:59.431859 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:27:59.431832 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9636ca_9bf1_4473_81e8_e27d0d854b65.slice/crio-48f6c28ab722464f9c1eef27b601525a3c0c87abb43af243161ec066e063e26e WatchSource:0}: Error finding container 48f6c28ab722464f9c1eef27b601525a3c0c87abb43af243161ec066e063e26e: Status 404 returned error can't find the container with id 48f6c28ab722464f9c1eef27b601525a3c0c87abb43af243161ec066e063e26e Jul 10 00:27:59.796655 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.796621 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj" event={"ID":"3b9636ca-9bf1-4473-81e8-e27d0d854b65","Type":"ContainerStarted","Data":"1f9e237eb6bc045479a3dc6d7095b5a300c6aa9a62a338f52b1fe432bb152fc0"} Jul 10 00:27:59.796885 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.796676 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj" event={"ID":"3b9636ca-9bf1-4473-81e8-e27d0d854b65","Type":"ContainerStarted","Data":"48f6c28ab722464f9c1eef27b601525a3c0c87abb43af243161ec066e063e26e"} Jul 10 00:27:59.796885 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.796703 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj" Jul 10 00:27:59.805817 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.805783 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj"] Jul 10 00:27:59.811084 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.811058 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj"] Jul 10 00:27:59.816029 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:27:59.815990 2559 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3b9636ca-9bf1-4473-81e8-e27d0d854b65\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-07-10T00:27:59Z\\\",\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"200m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://1f9e237eb6bc045479a3dc6d7095b5a300c6aa9a62a338f52b1fe432bb152fc0\\\",\\\"image\\\":\\\"quay.io/kuadrant/kuadrant-operator:v1.4.2\\\",\\\"imageID\\\":\\\"quay.io/kuadrant/kuadrant-operator@sha256:0ad6673d1a66a99ce9881d4f50b51e1f98b477a019a8efe7187a73396d773e58\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"manager\\\",\\\"ready\\\":false,\\\"resources\\\":{\\\"limits\\\":{\\\"cpu\\\":\\\"200m\\\",\\\"memory\\\":\\\"300Mi\\\"},\\\"requests\\\":{\\\"cpu\\\":\\\"200m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-07-10T00:27:59Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000680000],\\\"uid\\\":1000680000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/tmp/kuadrant\\\",\\\"name\\\":\\\"extensions-socket-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-s2j9s\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Running\\\",\\\"podIP\\\":\\\"10.133.0.25\\\",\\\"podIPs\\\":[{\\\"ip\\\":\\\"10.133.0.25\\\"}]}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-6fb7879bc9-d2rkj\": pods \"kuadrant-operator-controller-manager-6fb7879bc9-d2rkj\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj" Jul 10 00:27:59.825477 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.825455 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt"] Jul 10 00:27:59.825701 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.825661 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt" podUID="c548b776-2053-4095-a7eb-5f6272eceacf" containerName="manager" containerID="cri-o://80243e7c3193ac0c23f320f2798069aab17962f073f872615690baa16c2f32d0" gracePeriod=2 Jul 10 00:27:59.832166 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.832142 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xxs96"] Jul 10 00:27:59.832688 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.832667 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b9636ca-9bf1-4473-81e8-e27d0d854b65" containerName="manager" Jul 10 00:27:59.832742 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.832692 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b9636ca-9bf1-4473-81e8-e27d0d854b65" containerName="manager" Jul 10 00:27:59.832797 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.832784 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b9636ca-9bf1-4473-81e8-e27d0d854b65" containerName="manager" Jul 10 00:27:59.836074 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.836051 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xxs96" Jul 10 00:27:59.836686 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.836666 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt"] Jul 10 00:27:59.856847 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:27:59.856811 2559 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"c548b776-2053-4095-a7eb-5f6272eceacf\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-kv9kt\": pods \"kuadrant-operator-controller-manager-8684c9ff49-kv9kt\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt" Jul 10 00:27:59.859755 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.859730 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzb7q"] Jul 10 00:27:59.859838 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:27:59.859735 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-d2rkj\" is forbidden: User \"system:node:ip-10-0-132-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-30.ec2.internal' and this object" podUID="3b9636ca-9bf1-4473-81e8-e27d0d854b65" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj" Jul 10 00:27:59.860013 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.859953 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzb7q" podUID="fff18a8e-7dc4-4fce-8c3a-e928d67cd625" containerName="manager" containerID="cri-o://2221b82c32349b3ff0af9210b5d8d040005815a4f1b5e1afe4884f04e2a38462" gracePeriod=2 Jul 10 00:27:59.861566 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.861547 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzb7q" Jul 10 00:27:59.862318 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.862301 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xxs96"] Jul 10 00:27:59.863881 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:27:59.863854 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-kv9kt\" is forbidden: User \"system:node:ip-10-0-132-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-30.ec2.internal' and this object" podUID="c548b776-2053-4095-a7eb-5f6272eceacf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt" Jul 10 00:27:59.865647 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:27:59.865616 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-d2rkj\" is forbidden: User \"system:node:ip-10-0-132-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-30.ec2.internal' and this object" podUID="3b9636ca-9bf1-4473-81e8-e27d0d854b65" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj" Jul 10 00:27:59.869020 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.869000 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xzjjt"] Jul 10 00:27:59.869344 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.869329 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c548b776-2053-4095-a7eb-5f6272eceacf" containerName="manager" Jul 10 00:27:59.869413 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.869345 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c548b776-2053-4095-a7eb-5f6272eceacf" containerName="manager" Jul 10 00:27:59.869413 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.869406 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c548b776-2053-4095-a7eb-5f6272eceacf" containerName="manager" Jul 10 00:27:59.872647 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.872630 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzb7q"] Jul 10 00:27:59.872734 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.872713 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xzjjt" Jul 10 00:27:59.895536 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:27:59.895512 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xzjjt"] Jul 10 00:27:59.961105 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:27:59.961076 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pzb7q\" is forbidden: User \"system:node:ip-10-0-132-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-30.ec2.internal' and this object" podUID="fff18a8e-7dc4-4fce-8c3a-e928d67cd625" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzb7q" Jul 10 00:27:59.963464 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:27:59.963442 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-kv9kt\" is forbidden: User \"system:node:ip-10-0-132-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-30.ec2.internal' and this object" podUID="c548b776-2053-4095-a7eb-5f6272eceacf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt" Jul 10 00:27:59.965365 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:27:59.965342 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-d2rkj\" is forbidden: User \"system:node:ip-10-0-132-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-30.ec2.internal' and this object" podUID="3b9636ca-9bf1-4473-81e8-e27d0d854b65" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj" Jul 10 00:28:00.001041 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.001002 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0f06031-27c6-4a20-aa5a-87e5085165f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xzjjt\" (UID: \"d0f06031-27c6-4a20-aa5a-87e5085165f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xzjjt" Jul 10 00:28:00.001041 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.001043 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf84p\" (UniqueName: \"kubernetes.io/projected/d0f06031-27c6-4a20-aa5a-87e5085165f3-kube-api-access-gf84p\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xzjjt\" (UID: \"d0f06031-27c6-4a20-aa5a-87e5085165f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xzjjt" Jul 10 00:28:00.001566 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.001079 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1de26c4-4544-4137-a040-cf3977a3dd5e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xxs96\" (UID: \"b1de26c4-4544-4137-a040-cf3977a3dd5e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xxs96" Jul 10 00:28:00.001566 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.001170 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvnl9\" (UniqueName: \"kubernetes.io/projected/b1de26c4-4544-4137-a040-cf3977a3dd5e-kube-api-access-pvnl9\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xxs96\" (UID: \"b1de26c4-4544-4137-a040-cf3977a3dd5e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xxs96" Jul 10 00:28:00.087475 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.087454 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt" Jul 10 00:28:00.089717 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:28:00.089691 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pzb7q\" is forbidden: User \"system:node:ip-10-0-132-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-30.ec2.internal' and this object" podUID="fff18a8e-7dc4-4fce-8c3a-e928d67cd625" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzb7q" Jul 10 00:28:00.091569 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:28:00.091544 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-kv9kt\" is forbidden: User \"system:node:ip-10-0-132-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-30.ec2.internal' and this object" podUID="c548b776-2053-4095-a7eb-5f6272eceacf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt" Jul 10 00:28:00.092564 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.092549 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzb7q" Jul 10 00:28:00.093152 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:28:00.093133 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-d2rkj\" is forbidden: User \"system:node:ip-10-0-132-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-30.ec2.internal' and this object" podUID="3b9636ca-9bf1-4473-81e8-e27d0d854b65" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj" Jul 10 00:28:00.094738 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:28:00.094721 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-d2rkj\" is forbidden: User \"system:node:ip-10-0-132-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-30.ec2.internal' and this object" podUID="3b9636ca-9bf1-4473-81e8-e27d0d854b65" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj" Jul 10 00:28:00.096181 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:28:00.096166 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pzb7q\" is forbidden: User \"system:node:ip-10-0-132-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-30.ec2.internal' and this object" podUID="fff18a8e-7dc4-4fce-8c3a-e928d67cd625" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzb7q" Jul 10 00:28:00.097582 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:28:00.097565 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-kv9kt\" is forbidden: User \"system:node:ip-10-0-132-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-30.ec2.internal' and this object" podUID="c548b776-2053-4095-a7eb-5f6272eceacf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt" Jul 10 00:28:00.101956 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.101940 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvnl9\" (UniqueName: \"kubernetes.io/projected/b1de26c4-4544-4137-a040-cf3977a3dd5e-kube-api-access-pvnl9\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xxs96\" (UID: \"b1de26c4-4544-4137-a040-cf3977a3dd5e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xxs96" Jul 10 00:28:00.102006 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.101991 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0f06031-27c6-4a20-aa5a-87e5085165f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xzjjt\" (UID: \"d0f06031-27c6-4a20-aa5a-87e5085165f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xzjjt" Jul 10 00:28:00.102040 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.102010 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf84p\" (UniqueName: \"kubernetes.io/projected/d0f06031-27c6-4a20-aa5a-87e5085165f3-kube-api-access-gf84p\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xzjjt\" (UID: \"d0f06031-27c6-4a20-aa5a-87e5085165f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xzjjt" Jul 10 00:28:00.102040 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.102033 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1de26c4-4544-4137-a040-cf3977a3dd5e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xxs96\" (UID: \"b1de26c4-4544-4137-a040-cf3977a3dd5e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xxs96" Jul 10 00:28:00.102370 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.102353 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1de26c4-4544-4137-a040-cf3977a3dd5e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xxs96\" (UID: \"b1de26c4-4544-4137-a040-cf3977a3dd5e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xxs96" Jul 10 00:28:00.102424 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.102379 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0f06031-27c6-4a20-aa5a-87e5085165f3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xzjjt\" (UID: \"d0f06031-27c6-4a20-aa5a-87e5085165f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xzjjt" Jul 10 00:28:00.109424 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.109399 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf84p\" (UniqueName: \"kubernetes.io/projected/d0f06031-27c6-4a20-aa5a-87e5085165f3-kube-api-access-gf84p\") pod \"kuadrant-operator-controller-manager-8684c9ff49-xzjjt\" (UID: \"d0f06031-27c6-4a20-aa5a-87e5085165f3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xzjjt" Jul 10 00:28:00.109524 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.109508 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvnl9\" (UniqueName: \"kubernetes.io/projected/b1de26c4-4544-4137-a040-cf3977a3dd5e-kube-api-access-pvnl9\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-xxs96\" (UID: \"b1de26c4-4544-4137-a040-cf3977a3dd5e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xxs96" Jul 10 00:28:00.147327 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.147308 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xxs96" Jul 10 00:28:00.203374 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.202955 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqjvt\" (UniqueName: \"kubernetes.io/projected/fff18a8e-7dc4-4fce-8c3a-e928d67cd625-kube-api-access-bqjvt\") pod \"fff18a8e-7dc4-4fce-8c3a-e928d67cd625\" (UID: \"fff18a8e-7dc4-4fce-8c3a-e928d67cd625\") " Jul 10 00:28:00.203374 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.203075 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c548b776-2053-4095-a7eb-5f6272eceacf-extensions-socket-volume\") pod \"c548b776-2053-4095-a7eb-5f6272eceacf\" (UID: \"c548b776-2053-4095-a7eb-5f6272eceacf\") " Jul 10 00:28:00.203374 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.203109 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7xdz\" (UniqueName: \"kubernetes.io/projected/c548b776-2053-4095-a7eb-5f6272eceacf-kube-api-access-b7xdz\") pod \"c548b776-2053-4095-a7eb-5f6272eceacf\" (UID: \"c548b776-2053-4095-a7eb-5f6272eceacf\") " Jul 10 00:28:00.203594 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.203563 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c548b776-2053-4095-a7eb-5f6272eceacf-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c548b776-2053-4095-a7eb-5f6272eceacf" (UID: "c548b776-2053-4095-a7eb-5f6272eceacf"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:28:00.204921 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.204893 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff18a8e-7dc4-4fce-8c3a-e928d67cd625-kube-api-access-bqjvt" (OuterVolumeSpecName: "kube-api-access-bqjvt") pod "fff18a8e-7dc4-4fce-8c3a-e928d67cd625" (UID: "fff18a8e-7dc4-4fce-8c3a-e928d67cd625"). InnerVolumeSpecName "kube-api-access-bqjvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:28:00.205239 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.205209 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c548b776-2053-4095-a7eb-5f6272eceacf-kube-api-access-b7xdz" (OuterVolumeSpecName: "kube-api-access-b7xdz") pod "c548b776-2053-4095-a7eb-5f6272eceacf" (UID: "c548b776-2053-4095-a7eb-5f6272eceacf"). InnerVolumeSpecName "kube-api-access-b7xdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:28:00.272082 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.272053 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xzjjt" Jul 10 00:28:00.273704 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.273678 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xxs96"] Jul 10 00:28:00.276345 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:28:00.276278 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1de26c4_4544_4137_a040_cf3977a3dd5e.slice/crio-ef0e8df7f70fd6f8ec65c4e0c680aeb028beacc724e22de87394951255a2f831 WatchSource:0}: Error finding container ef0e8df7f70fd6f8ec65c4e0c680aeb028beacc724e22de87394951255a2f831: Status 404 returned error can't find the container with id ef0e8df7f70fd6f8ec65c4e0c680aeb028beacc724e22de87394951255a2f831 Jul 10 00:28:00.304539 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.304517 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqjvt\" (UniqueName: \"kubernetes.io/projected/fff18a8e-7dc4-4fce-8c3a-e928d67cd625-kube-api-access-bqjvt\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:28:00.304539 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.304540 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c548b776-2053-4095-a7eb-5f6272eceacf-extensions-socket-volume\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:28:00.304659 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.304549 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7xdz\" (UniqueName: \"kubernetes.io/projected/c548b776-2053-4095-a7eb-5f6272eceacf-kube-api-access-b7xdz\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:28:00.407179 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.407157 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xzjjt"] Jul 10 00:28:00.411802 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:28:00.411765 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f06031_27c6_4a20_aa5a_87e5085165f3.slice/crio-dfc9f12e6a4394f9d27ad892f6c2a82a667900020f8835a4e72889a73afa08ea WatchSource:0}: Error finding container dfc9f12e6a4394f9d27ad892f6c2a82a667900020f8835a4e72889a73afa08ea: Status 404 returned error can't find the container with id dfc9f12e6a4394f9d27ad892f6c2a82a667900020f8835a4e72889a73afa08ea Jul 10 00:28:00.761255 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.761226 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vtg5w" Jul 10 00:28:00.763217 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:28:00.763194 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-kv9kt\" is forbidden: User \"system:node:ip-10-0-132-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-30.ec2.internal' and this object" podUID="c548b776-2053-4095-a7eb-5f6272eceacf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt" Jul 10 00:28:00.765257 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:28:00.765237 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-d2rkj\" is forbidden: User \"system:node:ip-10-0-132-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-30.ec2.internal' and this object" podUID="3b9636ca-9bf1-4473-81e8-e27d0d854b65" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj" Jul 10 00:28:00.783155 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:28:00.783100 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pzb7q\" is forbidden: User \"system:node:ip-10-0-132-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-30.ec2.internal' and this object" podUID="fff18a8e-7dc4-4fce-8c3a-e928d67cd625" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzb7q" Jul 10 00:28:00.804029 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.804000 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xxs96" event={"ID":"b1de26c4-4544-4137-a040-cf3977a3dd5e","Type":"ContainerStarted","Data":"6dcea36a507d4c5840958d1f04c66c651f0002a711d29b13e4841e6640d8a070"} Jul 10 00:28:00.804199 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.804050 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xxs96" event={"ID":"b1de26c4-4544-4137-a040-cf3977a3dd5e","Type":"ContainerStarted","Data":"ef0e8df7f70fd6f8ec65c4e0c680aeb028beacc724e22de87394951255a2f831"} Jul 10 00:28:00.804199 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.804137 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xxs96" Jul 10 00:28:00.805756 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.805732 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xzjjt" event={"ID":"d0f06031-27c6-4a20-aa5a-87e5085165f3","Type":"ContainerStarted","Data":"2908c182236bebc67ef7c474a6b822158470a761234d294e0442c254179c3275"} Jul 10 00:28:00.805877 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.805780 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xzjjt" Jul 10 00:28:00.805877 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.805806 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xzjjt" event={"ID":"d0f06031-27c6-4a20-aa5a-87e5085165f3","Type":"ContainerStarted","Data":"dfc9f12e6a4394f9d27ad892f6c2a82a667900020f8835a4e72889a73afa08ea"} Jul 10 00:28:00.806206 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:28:00.806177 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pzb7q\" is forbidden: User \"system:node:ip-10-0-132-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-30.ec2.internal' and this object" podUID="fff18a8e-7dc4-4fce-8c3a-e928d67cd625" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzb7q" Jul 10 00:28:00.807013 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.806990 2559 generic.go:356] "Generic (PLEG): container finished" podID="c548b776-2053-4095-a7eb-5f6272eceacf" containerID="80243e7c3193ac0c23f320f2798069aab17962f073f872615690baa16c2f32d0" exitCode=0 Jul 10 00:28:00.807088 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.807021 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt" Jul 10 00:28:00.807088 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.807075 2559 scope.go:117] "RemoveContainer" containerID="80243e7c3193ac0c23f320f2798069aab17962f073f872615690baa16c2f32d0" Jul 10 00:28:00.808002 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:28:00.807978 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-kv9kt\" is forbidden: User \"system:node:ip-10-0-132-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-30.ec2.internal' and this object" podUID="c548b776-2053-4095-a7eb-5f6272eceacf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt" Jul 10 00:28:00.808464 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.808441 2559 generic.go:356] "Generic (PLEG): container finished" podID="fff18a8e-7dc4-4fce-8c3a-e928d67cd625" containerID="2221b82c32349b3ff0af9210b5d8d040005815a4f1b5e1afe4884f04e2a38462" exitCode=0 Jul 10 00:28:00.808572 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.808498 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzb7q" Jul 10 00:28:00.808890 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:28:00.808872 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj" container="manager" Jul 10 00:28:00.809755 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:28:00.809724 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-d2rkj\" is forbidden: User \"system:node:ip-10-0-132-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-30.ec2.internal' and this object" podUID="3b9636ca-9bf1-4473-81e8-e27d0d854b65" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj" Jul 10 00:28:00.817144 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.817108 2559 scope.go:117] "RemoveContainer" containerID="80243e7c3193ac0c23f320f2798069aab17962f073f872615690baa16c2f32d0" Jul 10 00:28:00.817614 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:28:00.817450 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80243e7c3193ac0c23f320f2798069aab17962f073f872615690baa16c2f32d0\": container with ID starting with 80243e7c3193ac0c23f320f2798069aab17962f073f872615690baa16c2f32d0 not found: ID does not exist" containerID="80243e7c3193ac0c23f320f2798069aab17962f073f872615690baa16c2f32d0" Jul 10 00:28:00.817614 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.817482 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80243e7c3193ac0c23f320f2798069aab17962f073f872615690baa16c2f32d0"} err="failed to get container status \"80243e7c3193ac0c23f320f2798069aab17962f073f872615690baa16c2f32d0\": rpc error: code = NotFound desc = could not find container \"80243e7c3193ac0c23f320f2798069aab17962f073f872615690baa16c2f32d0\": container with ID starting with 80243e7c3193ac0c23f320f2798069aab17962f073f872615690baa16c2f32d0 not found: ID does not exist" Jul 10 00:28:00.817614 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.817505 2559 scope.go:117] "RemoveContainer" containerID="2221b82c32349b3ff0af9210b5d8d040005815a4f1b5e1afe4884f04e2a38462" Jul 10 00:28:00.828251 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.828234 2559 scope.go:117] "RemoveContainer" containerID="2221b82c32349b3ff0af9210b5d8d040005815a4f1b5e1afe4884f04e2a38462" Jul 10 00:28:00.828567 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:28:00.828505 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2221b82c32349b3ff0af9210b5d8d040005815a4f1b5e1afe4884f04e2a38462\": container with ID starting with 2221b82c32349b3ff0af9210b5d8d040005815a4f1b5e1afe4884f04e2a38462 not found: ID does not exist" containerID="2221b82c32349b3ff0af9210b5d8d040005815a4f1b5e1afe4884f04e2a38462" Jul 10 00:28:00.828567 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.828535 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2221b82c32349b3ff0af9210b5d8d040005815a4f1b5e1afe4884f04e2a38462"} err="failed to get container status \"2221b82c32349b3ff0af9210b5d8d040005815a4f1b5e1afe4884f04e2a38462\": rpc error: code = NotFound desc = could not find container \"2221b82c32349b3ff0af9210b5d8d040005815a4f1b5e1afe4884f04e2a38462\": container with ID starting with 2221b82c32349b3ff0af9210b5d8d040005815a4f1b5e1afe4884f04e2a38462 not found: ID does not exist" Jul 10 00:28:00.830392 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.830338 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xxs96" podStartSLOduration=1.8303266040000001 podStartE2EDuration="1.830326604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:28:00.828554536 +0000 UTC m=+613.761638845" watchObservedRunningTime="2026-07-10 00:28:00.830326604 +0000 UTC m=+613.763410913" Jul 10 00:28:00.830515 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:28:00.830455 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-d2rkj\" is forbidden: User \"system:node:ip-10-0-132-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-30.ec2.internal' and this object" podUID="3b9636ca-9bf1-4473-81e8-e27d0d854b65" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj" Jul 10 00:28:00.849277 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:28:00.849243 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pzb7q\" is forbidden: User \"system:node:ip-10-0-132-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-30.ec2.internal' and this object" podUID="fff18a8e-7dc4-4fce-8c3a-e928d67cd625" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pzb7q" Jul 10 00:28:00.849441 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:00.849361 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xzjjt" podStartSLOduration=1.849345706 podStartE2EDuration="1.849345706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:28:00.847580911 +0000 UTC m=+613.780665239" watchObservedRunningTime="2026-07-10 00:28:00.849345706 +0000 UTC m=+613.782430017" Jul 10 00:28:00.850924 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:28:00.850902 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-kv9kt\" is forbidden: User \"system:node:ip-10-0-132-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-30.ec2.internal' and this object" podUID="c548b776-2053-4095-a7eb-5f6272eceacf" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-kv9kt" Jul 10 00:28:01.650569 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:01.650540 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c548b776-2053-4095-a7eb-5f6272eceacf" path="/var/lib/kubelet/pods/c548b776-2053-4095-a7eb-5f6272eceacf/volumes" Jul 10 00:28:01.650902 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:01.650847 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fff18a8e-7dc4-4fce-8c3a-e928d67cd625" path="/var/lib/kubelet/pods/fff18a8e-7dc4-4fce-8c3a-e928d67cd625/volumes" Jul 10 00:28:01.817302 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:01.817264 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj" podUID="3b9636ca-9bf1-4473-81e8-e27d0d854b65" containerName="manager" containerID="cri-o://1f9e237eb6bc045479a3dc6d7095b5a300c6aa9a62a338f52b1fe432bb152fc0" gracePeriod=2 Jul 10 00:28:02.061410 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:02.061384 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj" Jul 10 00:28:02.063541 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:28:02.063511 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-d2rkj\" is forbidden: User \"system:node:ip-10-0-132-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-30.ec2.internal' and this object" podUID="3b9636ca-9bf1-4473-81e8-e27d0d854b65" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj" Jul 10 00:28:02.221147 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:02.221051 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2j9s\" (UniqueName: \"kubernetes.io/projected/3b9636ca-9bf1-4473-81e8-e27d0d854b65-kube-api-access-s2j9s\") pod \"3b9636ca-9bf1-4473-81e8-e27d0d854b65\" (UID: \"3b9636ca-9bf1-4473-81e8-e27d0d854b65\") " Jul 10 00:28:02.221147 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:02.221100 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b9636ca-9bf1-4473-81e8-e27d0d854b65-extensions-socket-volume\") pod \"3b9636ca-9bf1-4473-81e8-e27d0d854b65\" (UID: \"3b9636ca-9bf1-4473-81e8-e27d0d854b65\") " Jul 10 00:28:02.221479 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:02.221458 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b9636ca-9bf1-4473-81e8-e27d0d854b65-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3b9636ca-9bf1-4473-81e8-e27d0d854b65" (UID: "3b9636ca-9bf1-4473-81e8-e27d0d854b65"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:28:02.223087 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:02.223061 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b9636ca-9bf1-4473-81e8-e27d0d854b65-kube-api-access-s2j9s" (OuterVolumeSpecName: "kube-api-access-s2j9s") pod "3b9636ca-9bf1-4473-81e8-e27d0d854b65" (UID: "3b9636ca-9bf1-4473-81e8-e27d0d854b65"). InnerVolumeSpecName "kube-api-access-s2j9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:28:02.322212 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:02.322169 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2j9s\" (UniqueName: \"kubernetes.io/projected/3b9636ca-9bf1-4473-81e8-e27d0d854b65-kube-api-access-s2j9s\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:28:02.322212 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:02.322211 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3b9636ca-9bf1-4473-81e8-e27d0d854b65-extensions-socket-volume\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:28:02.825192 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:02.825158 2559 generic.go:356] "Generic (PLEG): container finished" podID="3b9636ca-9bf1-4473-81e8-e27d0d854b65" containerID="1f9e237eb6bc045479a3dc6d7095b5a300c6aa9a62a338f52b1fe432bb152fc0" exitCode=0 Jul 10 00:28:02.825583 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:02.825205 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj" Jul 10 00:28:02.825583 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:02.825246 2559 scope.go:117] "RemoveContainer" containerID="1f9e237eb6bc045479a3dc6d7095b5a300c6aa9a62a338f52b1fe432bb152fc0" Jul 10 00:28:02.827866 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:28:02.827840 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-d2rkj\" is forbidden: User \"system:node:ip-10-0-132-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-30.ec2.internal' and this object" podUID="3b9636ca-9bf1-4473-81e8-e27d0d854b65" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj" Jul 10 00:28:02.833448 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:02.833428 2559 scope.go:117] "RemoveContainer" containerID="1f9e237eb6bc045479a3dc6d7095b5a300c6aa9a62a338f52b1fe432bb152fc0" Jul 10 00:28:02.833685 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:28:02.833665 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f9e237eb6bc045479a3dc6d7095b5a300c6aa9a62a338f52b1fe432bb152fc0\": container with ID starting with 1f9e237eb6bc045479a3dc6d7095b5a300c6aa9a62a338f52b1fe432bb152fc0 not found: ID does not exist" containerID="1f9e237eb6bc045479a3dc6d7095b5a300c6aa9a62a338f52b1fe432bb152fc0" Jul 10 00:28:02.833740 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:02.833692 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f9e237eb6bc045479a3dc6d7095b5a300c6aa9a62a338f52b1fe432bb152fc0"} err="failed to get container status \"1f9e237eb6bc045479a3dc6d7095b5a300c6aa9a62a338f52b1fe432bb152fc0\": rpc error: code = NotFound desc = could not find container \"1f9e237eb6bc045479a3dc6d7095b5a300c6aa9a62a338f52b1fe432bb152fc0\": container with ID starting with 1f9e237eb6bc045479a3dc6d7095b5a300c6aa9a62a338f52b1fe432bb152fc0 not found: ID does not exist" Jul 10 00:28:02.835371 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:28:02.835354 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-d2rkj\" is forbidden: User \"system:node:ip-10-0-132-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-30.ec2.internal' and this object" podUID="3b9636ca-9bf1-4473-81e8-e27d0d854b65" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-d2rkj" Jul 10 00:28:03.649598 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:03.649566 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b9636ca-9bf1-4473-81e8-e27d0d854b65" path="/var/lib/kubelet/pods/3b9636ca-9bf1-4473-81e8-e27d0d854b65/volumes" Jul 10 00:28:11.819839 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:11.819763 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xxs96" Jul 10 00:28:11.820321 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:11.819884 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xzjjt" Jul 10 00:28:11.890601 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:11.890573 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xzjjt"] Jul 10 00:28:11.890788 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:11.890763 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xzjjt" podUID="d0f06031-27c6-4a20-aa5a-87e5085165f3" containerName="manager" containerID="cri-o://2908c182236bebc67ef7c474a6b822158470a761234d294e0442c254179c3275" gracePeriod=10 Jul 10 00:28:12.134562 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.134529 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xzjjt" Jul 10 00:28:12.136379 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.136356 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ps96k"] Jul 10 00:28:12.136681 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.136667 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0f06031-27c6-4a20-aa5a-87e5085165f3" containerName="manager" Jul 10 00:28:12.136681 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.136682 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0f06031-27c6-4a20-aa5a-87e5085165f3" containerName="manager" Jul 10 00:28:12.136768 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.136702 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fff18a8e-7dc4-4fce-8c3a-e928d67cd625" containerName="manager" Jul 10 00:28:12.136768 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.136708 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fff18a8e-7dc4-4fce-8c3a-e928d67cd625" containerName="manager" Jul 10 00:28:12.136768 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.136753 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fff18a8e-7dc4-4fce-8c3a-e928d67cd625" containerName="manager" Jul 10 00:28:12.136768 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.136764 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0f06031-27c6-4a20-aa5a-87e5085165f3" containerName="manager" Jul 10 00:28:12.139780 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.139763 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ps96k" Jul 10 00:28:12.157974 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.157951 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ps96k"] Jul 10 00:28:12.192379 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.188533 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xskb\" (UniqueName: \"kubernetes.io/projected/065e87e2-bfa1-4f82-8108-e8ffd5121933-kube-api-access-8xskb\") pod \"kuadrant-operator-controller-manager-86878c769f-ps96k\" (UID: \"065e87e2-bfa1-4f82-8108-e8ffd5121933\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ps96k" Jul 10 00:28:12.192379 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.188611 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/065e87e2-bfa1-4f82-8108-e8ffd5121933-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ps96k\" (UID: \"065e87e2-bfa1-4f82-8108-e8ffd5121933\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ps96k" Jul 10 00:28:12.289764 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.289740 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0f06031-27c6-4a20-aa5a-87e5085165f3-extensions-socket-volume\") pod \"d0f06031-27c6-4a20-aa5a-87e5085165f3\" (UID: \"d0f06031-27c6-4a20-aa5a-87e5085165f3\") " Jul 10 00:28:12.289905 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.289779 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf84p\" (UniqueName: \"kubernetes.io/projected/d0f06031-27c6-4a20-aa5a-87e5085165f3-kube-api-access-gf84p\") pod \"d0f06031-27c6-4a20-aa5a-87e5085165f3\" (UID: \"d0f06031-27c6-4a20-aa5a-87e5085165f3\") " Jul 10 00:28:12.289905 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.289852 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/065e87e2-bfa1-4f82-8108-e8ffd5121933-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ps96k\" (UID: \"065e87e2-bfa1-4f82-8108-e8ffd5121933\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ps96k" Jul 10 00:28:12.290028 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.289925 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xskb\" (UniqueName: \"kubernetes.io/projected/065e87e2-bfa1-4f82-8108-e8ffd5121933-kube-api-access-8xskb\") pod \"kuadrant-operator-controller-manager-86878c769f-ps96k\" (UID: \"065e87e2-bfa1-4f82-8108-e8ffd5121933\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ps96k" Jul 10 00:28:12.290222 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.290196 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f06031-27c6-4a20-aa5a-87e5085165f3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d0f06031-27c6-4a20-aa5a-87e5085165f3" (UID: "d0f06031-27c6-4a20-aa5a-87e5085165f3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:28:12.290288 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.290268 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/065e87e2-bfa1-4f82-8108-e8ffd5121933-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ps96k\" (UID: \"065e87e2-bfa1-4f82-8108-e8ffd5121933\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ps96k" Jul 10 00:28:12.291690 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.291669 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f06031-27c6-4a20-aa5a-87e5085165f3-kube-api-access-gf84p" (OuterVolumeSpecName: "kube-api-access-gf84p") pod "d0f06031-27c6-4a20-aa5a-87e5085165f3" (UID: "d0f06031-27c6-4a20-aa5a-87e5085165f3"). InnerVolumeSpecName "kube-api-access-gf84p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:28:12.300825 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.300807 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xskb\" (UniqueName: \"kubernetes.io/projected/065e87e2-bfa1-4f82-8108-e8ffd5121933-kube-api-access-8xskb\") pod \"kuadrant-operator-controller-manager-86878c769f-ps96k\" (UID: \"065e87e2-bfa1-4f82-8108-e8ffd5121933\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ps96k" Jul 10 00:28:12.390285 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.390229 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0f06031-27c6-4a20-aa5a-87e5085165f3-extensions-socket-volume\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:28:12.390285 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.390248 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf84p\" (UniqueName: \"kubernetes.io/projected/d0f06031-27c6-4a20-aa5a-87e5085165f3-kube-api-access-gf84p\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:28:12.448881 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.448859 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ps96k" Jul 10 00:28:12.573674 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.573646 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ps96k"] Jul 10 00:28:12.576333 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:28:12.576304 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod065e87e2_bfa1_4f82_8108_e8ffd5121933.slice/crio-dd903b28503f7e4c44df5730ebb4eafc1b8621361140403ae65ff612a5362d84 WatchSource:0}: Error finding container dd903b28503f7e4c44df5730ebb4eafc1b8621361140403ae65ff612a5362d84: Status 404 returned error can't find the container with id dd903b28503f7e4c44df5730ebb4eafc1b8621361140403ae65ff612a5362d84 Jul 10 00:28:12.870503 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.870471 2559 generic.go:356] "Generic (PLEG): container finished" podID="d0f06031-27c6-4a20-aa5a-87e5085165f3" containerID="2908c182236bebc67ef7c474a6b822158470a761234d294e0442c254179c3275" exitCode=0 Jul 10 00:28:12.870913 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.870519 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xzjjt" event={"ID":"d0f06031-27c6-4a20-aa5a-87e5085165f3","Type":"ContainerDied","Data":"2908c182236bebc67ef7c474a6b822158470a761234d294e0442c254179c3275"} Jul 10 00:28:12.870913 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.870574 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xzjjt" event={"ID":"d0f06031-27c6-4a20-aa5a-87e5085165f3","Type":"ContainerDied","Data":"dfc9f12e6a4394f9d27ad892f6c2a82a667900020f8835a4e72889a73afa08ea"} Jul 10 00:28:12.870913 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.870531 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xzjjt" Jul 10 00:28:12.870913 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.870607 2559 scope.go:117] "RemoveContainer" containerID="2908c182236bebc67ef7c474a6b822158470a761234d294e0442c254179c3275" Jul 10 00:28:12.872418 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.872383 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ps96k" event={"ID":"065e87e2-bfa1-4f82-8108-e8ffd5121933","Type":"ContainerStarted","Data":"7b87d3b73e7ef74fa683fd64bbe2fa7efa1214730a9cf150281f7974d00fd788"} Jul 10 00:28:12.872527 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.872436 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ps96k" event={"ID":"065e87e2-bfa1-4f82-8108-e8ffd5121933","Type":"ContainerStarted","Data":"dd903b28503f7e4c44df5730ebb4eafc1b8621361140403ae65ff612a5362d84"} Jul 10 00:28:12.872527 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.872472 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ps96k" Jul 10 00:28:12.880083 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.880064 2559 scope.go:117] "RemoveContainer" containerID="2908c182236bebc67ef7c474a6b822158470a761234d294e0442c254179c3275" Jul 10 00:28:12.880393 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:28:12.880367 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2908c182236bebc67ef7c474a6b822158470a761234d294e0442c254179c3275\": container with ID starting with 2908c182236bebc67ef7c474a6b822158470a761234d294e0442c254179c3275 not found: ID does not exist" containerID="2908c182236bebc67ef7c474a6b822158470a761234d294e0442c254179c3275" Jul 10 00:28:12.880452 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.880401 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2908c182236bebc67ef7c474a6b822158470a761234d294e0442c254179c3275"} err="failed to get container status \"2908c182236bebc67ef7c474a6b822158470a761234d294e0442c254179c3275\": rpc error: code = NotFound desc = could not find container \"2908c182236bebc67ef7c474a6b822158470a761234d294e0442c254179c3275\": container with ID starting with 2908c182236bebc67ef7c474a6b822158470a761234d294e0442c254179c3275 not found: ID does not exist" Jul 10 00:28:12.913128 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.913081 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ps96k" podStartSLOduration=0.913067386 podStartE2EDuration="913.067386ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:28:12.910773627 +0000 UTC m=+625.843857962" watchObservedRunningTime="2026-07-10 00:28:12.913067386 +0000 UTC m=+625.846151694" Jul 10 00:28:12.928938 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.928909 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xzjjt"] Jul 10 00:28:12.931071 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:12.931054 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-xzjjt"] Jul 10 00:28:13.649590 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:13.649557 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0f06031-27c6-4a20-aa5a-87e5085165f3" path="/var/lib/kubelet/pods/d0f06031-27c6-4a20-aa5a-87e5085165f3/volumes" Jul 10 00:28:23.880440 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:23.880409 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ps96k" Jul 10 00:28:23.929454 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:23.929422 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xxs96"] Jul 10 00:28:23.929665 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:23.929628 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xxs96" podUID="b1de26c4-4544-4137-a040-cf3977a3dd5e" containerName="manager" containerID="cri-o://6dcea36a507d4c5840958d1f04c66c651f0002a711d29b13e4841e6640d8a070" gracePeriod=10 Jul 10 00:28:24.164517 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:24.164493 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xxs96" Jul 10 00:28:24.179458 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:24.179437 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvnl9\" (UniqueName: \"kubernetes.io/projected/b1de26c4-4544-4137-a040-cf3977a3dd5e-kube-api-access-pvnl9\") pod \"b1de26c4-4544-4137-a040-cf3977a3dd5e\" (UID: \"b1de26c4-4544-4137-a040-cf3977a3dd5e\") " Jul 10 00:28:24.179572 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:24.179515 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1de26c4-4544-4137-a040-cf3977a3dd5e-extensions-socket-volume\") pod \"b1de26c4-4544-4137-a040-cf3977a3dd5e\" (UID: \"b1de26c4-4544-4137-a040-cf3977a3dd5e\") " Jul 10 00:28:24.179939 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:24.179883 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1de26c4-4544-4137-a040-cf3977a3dd5e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b1de26c4-4544-4137-a040-cf3977a3dd5e" (UID: "b1de26c4-4544-4137-a040-cf3977a3dd5e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:28:24.181973 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:24.181951 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1de26c4-4544-4137-a040-cf3977a3dd5e-kube-api-access-pvnl9" (OuterVolumeSpecName: "kube-api-access-pvnl9") pod "b1de26c4-4544-4137-a040-cf3977a3dd5e" (UID: "b1de26c4-4544-4137-a040-cf3977a3dd5e"). InnerVolumeSpecName "kube-api-access-pvnl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:28:24.281069 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:24.281042 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1de26c4-4544-4137-a040-cf3977a3dd5e-extensions-socket-volume\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:28:24.281069 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:24.281069 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvnl9\" (UniqueName: \"kubernetes.io/projected/b1de26c4-4544-4137-a040-cf3977a3dd5e-kube-api-access-pvnl9\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:28:24.928154 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:24.928103 2559 generic.go:356] "Generic (PLEG): container finished" podID="b1de26c4-4544-4137-a040-cf3977a3dd5e" containerID="6dcea36a507d4c5840958d1f04c66c651f0002a711d29b13e4841e6640d8a070" exitCode=0 Jul 10 00:28:24.928608 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:24.928159 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xxs96" Jul 10 00:28:24.928608 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:24.928193 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xxs96" event={"ID":"b1de26c4-4544-4137-a040-cf3977a3dd5e","Type":"ContainerDied","Data":"6dcea36a507d4c5840958d1f04c66c651f0002a711d29b13e4841e6640d8a070"} Jul 10 00:28:24.928608 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:24.928256 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xxs96" event={"ID":"b1de26c4-4544-4137-a040-cf3977a3dd5e","Type":"ContainerDied","Data":"ef0e8df7f70fd6f8ec65c4e0c680aeb028beacc724e22de87394951255a2f831"} Jul 10 00:28:24.928608 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:24.928290 2559 scope.go:117] "RemoveContainer" containerID="6dcea36a507d4c5840958d1f04c66c651f0002a711d29b13e4841e6640d8a070" Jul 10 00:28:24.937408 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:24.937248 2559 scope.go:117] "RemoveContainer" containerID="6dcea36a507d4c5840958d1f04c66c651f0002a711d29b13e4841e6640d8a070" Jul 10 00:28:24.937526 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:28:24.937508 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dcea36a507d4c5840958d1f04c66c651f0002a711d29b13e4841e6640d8a070\": container with ID starting with 6dcea36a507d4c5840958d1f04c66c651f0002a711d29b13e4841e6640d8a070 not found: ID does not exist" containerID="6dcea36a507d4c5840958d1f04c66c651f0002a711d29b13e4841e6640d8a070" Jul 10 00:28:24.937568 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:24.937535 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dcea36a507d4c5840958d1f04c66c651f0002a711d29b13e4841e6640d8a070"} err="failed to get container status \"6dcea36a507d4c5840958d1f04c66c651f0002a711d29b13e4841e6640d8a070\": rpc error: code = NotFound desc = could not find container \"6dcea36a507d4c5840958d1f04c66c651f0002a711d29b13e4841e6640d8a070\": container with ID starting with 6dcea36a507d4c5840958d1f04c66c651f0002a711d29b13e4841e6640d8a070 not found: ID does not exist" Jul 10 00:28:24.956282 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:24.956260 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xxs96"] Jul 10 00:28:24.959564 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:24.959546 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-xxs96"] Jul 10 00:28:25.649474 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:25.649444 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1de26c4-4544-4137-a040-cf3977a3dd5e" path="/var/lib/kubelet/pods/b1de26c4-4544-4137-a040-cf3977a3dd5e/volumes" Jul 10 00:28:28.417033 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.416998 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk"] Jul 10 00:28:28.417394 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.417315 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1de26c4-4544-4137-a040-cf3977a3dd5e" containerName="manager" Jul 10 00:28:28.417394 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.417327 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1de26c4-4544-4137-a040-cf3977a3dd5e" containerName="manager" Jul 10 00:28:28.417462 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.417400 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1de26c4-4544-4137-a040-cf3977a3dd5e" containerName="manager" Jul 10 00:28:28.420134 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.420102 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.422260 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.422238 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-27mt6\"" Jul 10 00:28:28.424527 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.424503 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk"] Jul 10 00:28:28.516093 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.516059 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf310fe5-e414-4b36-97d9-c99cb1da8844-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-pfqlk\" (UID: \"cf310fe5-e414-4b36-97d9-c99cb1da8844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.516271 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.516109 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf310fe5-e414-4b36-97d9-c99cb1da8844-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-pfqlk\" (UID: \"cf310fe5-e414-4b36-97d9-c99cb1da8844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.516271 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.516161 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf310fe5-e414-4b36-97d9-c99cb1da8844-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-pfqlk\" (UID: \"cf310fe5-e414-4b36-97d9-c99cb1da8844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.516271 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.516198 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf310fe5-e414-4b36-97d9-c99cb1da8844-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-pfqlk\" (UID: \"cf310fe5-e414-4b36-97d9-c99cb1da8844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.516271 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.516227 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf310fe5-e414-4b36-97d9-c99cb1da8844-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-pfqlk\" (UID: \"cf310fe5-e414-4b36-97d9-c99cb1da8844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.516271 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.516250 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf310fe5-e414-4b36-97d9-c99cb1da8844-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-pfqlk\" (UID: \"cf310fe5-e414-4b36-97d9-c99cb1da8844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.516271 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.516277 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpc2k\" (UniqueName: \"kubernetes.io/projected/cf310fe5-e414-4b36-97d9-c99cb1da8844-kube-api-access-mpc2k\") pod \"maas-default-gateway-openshift-default-56654dcf7-pfqlk\" (UID: \"cf310fe5-e414-4b36-97d9-c99cb1da8844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.516477 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.516346 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf310fe5-e414-4b36-97d9-c99cb1da8844-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-pfqlk\" (UID: \"cf310fe5-e414-4b36-97d9-c99cb1da8844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.516477 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.516383 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf310fe5-e414-4b36-97d9-c99cb1da8844-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-pfqlk\" (UID: \"cf310fe5-e414-4b36-97d9-c99cb1da8844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.617228 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.617199 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpc2k\" (UniqueName: \"kubernetes.io/projected/cf310fe5-e414-4b36-97d9-c99cb1da8844-kube-api-access-mpc2k\") pod \"maas-default-gateway-openshift-default-56654dcf7-pfqlk\" (UID: \"cf310fe5-e414-4b36-97d9-c99cb1da8844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.617374 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.617249 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf310fe5-e414-4b36-97d9-c99cb1da8844-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-pfqlk\" (UID: \"cf310fe5-e414-4b36-97d9-c99cb1da8844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.617374 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.617270 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf310fe5-e414-4b36-97d9-c99cb1da8844-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-pfqlk\" (UID: \"cf310fe5-e414-4b36-97d9-c99cb1da8844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.617374 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.617309 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf310fe5-e414-4b36-97d9-c99cb1da8844-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-pfqlk\" (UID: \"cf310fe5-e414-4b36-97d9-c99cb1da8844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.617374 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.617330 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf310fe5-e414-4b36-97d9-c99cb1da8844-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-pfqlk\" (UID: \"cf310fe5-e414-4b36-97d9-c99cb1da8844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.617374 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.617358 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf310fe5-e414-4b36-97d9-c99cb1da8844-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-pfqlk\" (UID: \"cf310fe5-e414-4b36-97d9-c99cb1da8844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.617618 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.617381 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf310fe5-e414-4b36-97d9-c99cb1da8844-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-pfqlk\" (UID: \"cf310fe5-e414-4b36-97d9-c99cb1da8844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.617618 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.617428 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf310fe5-e414-4b36-97d9-c99cb1da8844-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-pfqlk\" (UID: \"cf310fe5-e414-4b36-97d9-c99cb1da8844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.617618 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.617452 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf310fe5-e414-4b36-97d9-c99cb1da8844-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-pfqlk\" (UID: \"cf310fe5-e414-4b36-97d9-c99cb1da8844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.617799 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.617772 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf310fe5-e414-4b36-97d9-c99cb1da8844-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-pfqlk\" (UID: \"cf310fe5-e414-4b36-97d9-c99cb1da8844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.617912 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.617809 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf310fe5-e414-4b36-97d9-c99cb1da8844-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-pfqlk\" (UID: \"cf310fe5-e414-4b36-97d9-c99cb1da8844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.617960 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.617913 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf310fe5-e414-4b36-97d9-c99cb1da8844-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-pfqlk\" (UID: \"cf310fe5-e414-4b36-97d9-c99cb1da8844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.618009 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.617974 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf310fe5-e414-4b36-97d9-c99cb1da8844-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-pfqlk\" (UID: \"cf310fe5-e414-4b36-97d9-c99cb1da8844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.618228 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.618207 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf310fe5-e414-4b36-97d9-c99cb1da8844-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-pfqlk\" (UID: \"cf310fe5-e414-4b36-97d9-c99cb1da8844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.619945 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.619924 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf310fe5-e414-4b36-97d9-c99cb1da8844-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-pfqlk\" (UID: \"cf310fe5-e414-4b36-97d9-c99cb1da8844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.620054 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.619946 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf310fe5-e414-4b36-97d9-c99cb1da8844-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-pfqlk\" (UID: \"cf310fe5-e414-4b36-97d9-c99cb1da8844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.628294 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.628272 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf310fe5-e414-4b36-97d9-c99cb1da8844-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-pfqlk\" (UID: \"cf310fe5-e414-4b36-97d9-c99cb1da8844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.628487 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.628470 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpc2k\" (UniqueName: \"kubernetes.io/projected/cf310fe5-e414-4b36-97d9-c99cb1da8844-kube-api-access-mpc2k\") pod \"maas-default-gateway-openshift-default-56654dcf7-pfqlk\" (UID: \"cf310fe5-e414-4b36-97d9-c99cb1da8844\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.731244 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.731186 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:28.863606 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.863559 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk"] Jul 10 00:28:28.865690 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:28:28.865665 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf310fe5_e414_4b36_97d9_c99cb1da8844.slice/crio-b7068b211c88f04c09700a1a1e3bb692235b86c860e817054cc8fea9dec9a672 WatchSource:0}: Error finding container b7068b211c88f04c09700a1a1e3bb692235b86c860e817054cc8fea9dec9a672: Status 404 returned error can't find the container with id b7068b211c88f04c09700a1a1e3bb692235b86c860e817054cc8fea9dec9a672 Jul 10 00:28:28.867641 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.867602 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:28:28.867712 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.867670 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:28:28.867712 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.867701 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:28:28.949133 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.949085 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" event={"ID":"cf310fe5-e414-4b36-97d9-c99cb1da8844","Type":"ContainerStarted","Data":"d820428d2c33b032c4c59d7b46bd52b49e04fc982f2d7310801909bd916d84c5"} Jul 10 00:28:28.949262 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.949165 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" event={"ID":"cf310fe5-e414-4b36-97d9-c99cb1da8844","Type":"ContainerStarted","Data":"b7068b211c88f04c09700a1a1e3bb692235b86c860e817054cc8fea9dec9a672"} Jul 10 00:28:28.973694 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:28.973466 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" podStartSLOduration=0.973443915 podStartE2EDuration="973.443915ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:28:28.972841035 +0000 UTC m=+641.905925370" watchObservedRunningTime="2026-07-10 00:28:28.973443915 +0000 UTC m=+641.906528224" Jul 10 00:28:29.731472 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:29.731439 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:29.735832 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:29.735809 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:29.953514 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:29.953487 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:29.954376 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:29.954358 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:28:44.625415 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:44.625375 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gv496"] Jul 10 00:28:44.628779 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:44.628754 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-gv496" Jul 10 00:28:44.630758 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:44.630741 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-lc6p2\"" Jul 10 00:28:44.630863 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:44.630744 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 10 00:28:44.635645 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:44.635615 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gv496"] Jul 10 00:28:44.723356 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:44.723324 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gv496"] Jul 10 00:28:44.744616 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:44.744590 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/46fa569b-4ed4-4fcb-95ee-d3312a87bcc1-config-file\") pod \"limitador-limitador-64c94fd64f-gv496\" (UID: \"46fa569b-4ed4-4fcb-95ee-d3312a87bcc1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gv496" Jul 10 00:28:44.744749 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:44.744634 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8cl8\" (UniqueName: \"kubernetes.io/projected/46fa569b-4ed4-4fcb-95ee-d3312a87bcc1-kube-api-access-d8cl8\") pod \"limitador-limitador-64c94fd64f-gv496\" (UID: \"46fa569b-4ed4-4fcb-95ee-d3312a87bcc1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gv496" Jul 10 00:28:44.845608 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:44.845578 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8cl8\" (UniqueName: \"kubernetes.io/projected/46fa569b-4ed4-4fcb-95ee-d3312a87bcc1-kube-api-access-d8cl8\") pod \"limitador-limitador-64c94fd64f-gv496\" (UID: \"46fa569b-4ed4-4fcb-95ee-d3312a87bcc1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gv496" Jul 10 00:28:44.845748 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:44.845660 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/46fa569b-4ed4-4fcb-95ee-d3312a87bcc1-config-file\") pod \"limitador-limitador-64c94fd64f-gv496\" (UID: \"46fa569b-4ed4-4fcb-95ee-d3312a87bcc1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gv496" Jul 10 00:28:44.846242 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:44.846224 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/46fa569b-4ed4-4fcb-95ee-d3312a87bcc1-config-file\") pod \"limitador-limitador-64c94fd64f-gv496\" (UID: \"46fa569b-4ed4-4fcb-95ee-d3312a87bcc1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gv496" Jul 10 00:28:44.852724 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:44.852699 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8cl8\" (UniqueName: \"kubernetes.io/projected/46fa569b-4ed4-4fcb-95ee-d3312a87bcc1-kube-api-access-d8cl8\") pod \"limitador-limitador-64c94fd64f-gv496\" (UID: \"46fa569b-4ed4-4fcb-95ee-d3312a87bcc1\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-gv496" Jul 10 00:28:44.939831 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:44.939778 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-gv496" Jul 10 00:28:45.069716 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:45.069687 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gv496"] Jul 10 00:28:45.072083 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:28:45.072056 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46fa569b_4ed4_4fcb_95ee_d3312a87bcc1.slice/crio-407e5e6b711701cbecd0d7103f035197baa21602f9ea9ede19df830b5debf591 WatchSource:0}: Error finding container 407e5e6b711701cbecd0d7103f035197baa21602f9ea9ede19df830b5debf591: Status 404 returned error can't find the container with id 407e5e6b711701cbecd0d7103f035197baa21602f9ea9ede19df830b5debf591 Jul 10 00:28:45.450518 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:45.450488 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-kgwf2"] Jul 10 00:28:45.457375 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:45.457333 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kgwf2" Jul 10 00:28:45.459248 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:45.459208 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-kgwf2"] Jul 10 00:28:45.459423 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:45.459392 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-dllv7\"" Jul 10 00:28:45.551941 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:45.551910 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6z2w\" (UniqueName: \"kubernetes.io/projected/d92c716a-48a1-4e22-a3f4-2e58ea92a8c1-kube-api-access-w6z2w\") pod \"authorino-58cd955675-kgwf2\" (UID: \"d92c716a-48a1-4e22-a3f4-2e58ea92a8c1\") " pod="kuadrant-system/authorino-58cd955675-kgwf2" Jul 10 00:28:45.653056 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:45.653028 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6z2w\" (UniqueName: \"kubernetes.io/projected/d92c716a-48a1-4e22-a3f4-2e58ea92a8c1-kube-api-access-w6z2w\") pod \"authorino-58cd955675-kgwf2\" (UID: \"d92c716a-48a1-4e22-a3f4-2e58ea92a8c1\") " pod="kuadrant-system/authorino-58cd955675-kgwf2" Jul 10 00:28:45.659983 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:45.659955 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6z2w\" (UniqueName: \"kubernetes.io/projected/d92c716a-48a1-4e22-a3f4-2e58ea92a8c1-kube-api-access-w6z2w\") pod \"authorino-58cd955675-kgwf2\" (UID: \"d92c716a-48a1-4e22-a3f4-2e58ea92a8c1\") " pod="kuadrant-system/authorino-58cd955675-kgwf2" Jul 10 00:28:45.766972 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:45.766926 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kgwf2" Jul 10 00:28:45.934940 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:45.934905 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-kgwf2"] Jul 10 00:28:45.939937 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:28:45.939894 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd92c716a_48a1_4e22_a3f4_2e58ea92a8c1.slice/crio-3536a1a5287843f248012174adf14548cd03b3f7112ee71c636badb31741df06 WatchSource:0}: Error finding container 3536a1a5287843f248012174adf14548cd03b3f7112ee71c636badb31741df06: Status 404 returned error can't find the container with id 3536a1a5287843f248012174adf14548cd03b3f7112ee71c636badb31741df06 Jul 10 00:28:46.029772 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:46.029630 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-gv496" event={"ID":"46fa569b-4ed4-4fcb-95ee-d3312a87bcc1","Type":"ContainerStarted","Data":"407e5e6b711701cbecd0d7103f035197baa21602f9ea9ede19df830b5debf591"} Jul 10 00:28:46.031269 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:46.031239 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kgwf2" event={"ID":"d92c716a-48a1-4e22-a3f4-2e58ea92a8c1","Type":"ContainerStarted","Data":"3536a1a5287843f248012174adf14548cd03b3f7112ee71c636badb31741df06"} Jul 10 00:28:50.054954 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:50.054914 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kgwf2" event={"ID":"d92c716a-48a1-4e22-a3f4-2e58ea92a8c1","Type":"ContainerStarted","Data":"db0e584c30c3b820e7595dd5d3e6162de5b672db55bb63505eec90b5c9c551a3"} Jul 10 00:28:50.056483 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:50.056455 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-gv496" event={"ID":"46fa569b-4ed4-4fcb-95ee-d3312a87bcc1","Type":"ContainerStarted","Data":"104c4a4749fb7b9f651caf7f300fcfef2c87da6f4b85faea2a95f3098c617c81"} Jul 10 00:28:50.056620 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:50.056522 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-gv496" Jul 10 00:28:50.075750 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:50.075709 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-kgwf2" podStartSLOduration=1.3363422329999999 podStartE2EDuration="5.075695024s" totalImagesPullingTime="3.739352791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:28:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:28:50.074283862 +0000 UTC m=+663.007368183" watchObservedRunningTime="2026-07-10 00:28:50.075695024 +0000 UTC m=+663.008779332" Jul 10 00:28:50.806916 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:50.806864 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-gv496" podStartSLOduration=2.257508031 podStartE2EDuration="6.806845262s" totalImagesPullingTime="4.549337231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:28:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:28:50.099991085 +0000 UTC m=+663.033075418" watchObservedRunningTime="2026-07-10 00:28:50.806845262 +0000 UTC m=+663.739929628" Jul 10 00:28:50.807174 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:50.807158 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-kgwf2"] Jul 10 00:28:52.065312 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:52.065269 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-kgwf2" podUID="d92c716a-48a1-4e22-a3f4-2e58ea92a8c1" containerName="authorino" containerID="cri-o://db0e584c30c3b820e7595dd5d3e6162de5b672db55bb63505eec90b5c9c551a3" gracePeriod=30 Jul 10 00:28:52.310496 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:52.310474 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kgwf2" Jul 10 00:28:52.410080 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:52.410004 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6z2w\" (UniqueName: \"kubernetes.io/projected/d92c716a-48a1-4e22-a3f4-2e58ea92a8c1-kube-api-access-w6z2w\") pod \"d92c716a-48a1-4e22-a3f4-2e58ea92a8c1\" (UID: \"d92c716a-48a1-4e22-a3f4-2e58ea92a8c1\") " Jul 10 00:28:52.411953 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:52.411928 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d92c716a-48a1-4e22-a3f4-2e58ea92a8c1-kube-api-access-w6z2w" (OuterVolumeSpecName: "kube-api-access-w6z2w") pod "d92c716a-48a1-4e22-a3f4-2e58ea92a8c1" (UID: "d92c716a-48a1-4e22-a3f4-2e58ea92a8c1"). InnerVolumeSpecName "kube-api-access-w6z2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:28:52.510899 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:52.510875 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6z2w\" (UniqueName: \"kubernetes.io/projected/d92c716a-48a1-4e22-a3f4-2e58ea92a8c1-kube-api-access-w6z2w\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:28:53.070668 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:53.070632 2559 generic.go:356] "Generic (PLEG): container finished" podID="d92c716a-48a1-4e22-a3f4-2e58ea92a8c1" containerID="db0e584c30c3b820e7595dd5d3e6162de5b672db55bb63505eec90b5c9c551a3" exitCode=0 Jul 10 00:28:53.071192 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:53.070676 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kgwf2" Jul 10 00:28:53.071192 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:53.070681 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kgwf2" event={"ID":"d92c716a-48a1-4e22-a3f4-2e58ea92a8c1","Type":"ContainerDied","Data":"db0e584c30c3b820e7595dd5d3e6162de5b672db55bb63505eec90b5c9c551a3"} Jul 10 00:28:53.071192 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:53.070722 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kgwf2" event={"ID":"d92c716a-48a1-4e22-a3f4-2e58ea92a8c1","Type":"ContainerDied","Data":"3536a1a5287843f248012174adf14548cd03b3f7112ee71c636badb31741df06"} Jul 10 00:28:53.071192 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:53.070742 2559 scope.go:117] "RemoveContainer" containerID="db0e584c30c3b820e7595dd5d3e6162de5b672db55bb63505eec90b5c9c551a3" Jul 10 00:28:53.079104 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:53.079086 2559 scope.go:117] "RemoveContainer" containerID="db0e584c30c3b820e7595dd5d3e6162de5b672db55bb63505eec90b5c9c551a3" Jul 10 00:28:53.079389 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:28:53.079371 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0e584c30c3b820e7595dd5d3e6162de5b672db55bb63505eec90b5c9c551a3\": container with ID starting with db0e584c30c3b820e7595dd5d3e6162de5b672db55bb63505eec90b5c9c551a3 not found: ID does not exist" containerID="db0e584c30c3b820e7595dd5d3e6162de5b672db55bb63505eec90b5c9c551a3" Jul 10 00:28:53.079450 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:53.079395 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0e584c30c3b820e7595dd5d3e6162de5b672db55bb63505eec90b5c9c551a3"} err="failed to get container status \"db0e584c30c3b820e7595dd5d3e6162de5b672db55bb63505eec90b5c9c551a3\": rpc error: code = NotFound desc = could not find container \"db0e584c30c3b820e7595dd5d3e6162de5b672db55bb63505eec90b5c9c551a3\": container with ID starting with db0e584c30c3b820e7595dd5d3e6162de5b672db55bb63505eec90b5c9c551a3 not found: ID does not exist" Jul 10 00:28:53.089902 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:53.089883 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-kgwf2"] Jul 10 00:28:53.091980 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:53.091956 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-kgwf2"] Jul 10 00:28:53.648922 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:28:53.648895 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d92c716a-48a1-4e22-a3f4-2e58ea92a8c1" path="/var/lib/kubelet/pods/d92c716a-48a1-4e22-a3f4-2e58ea92a8c1/volumes" Jul 10 00:29:01.061432 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:01.061406 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-gv496" Jul 10 00:29:01.812949 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:01.812913 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gv496"] Jul 10 00:29:01.813392 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:01.813346 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-gv496" podUID="46fa569b-4ed4-4fcb-95ee-d3312a87bcc1" containerName="limitador" containerID="cri-o://104c4a4749fb7b9f651caf7f300fcfef2c87da6f4b85faea2a95f3098c617c81" gracePeriod=30 Jul 10 00:29:02.759700 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:02.759677 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-gv496" Jul 10 00:29:02.888206 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:02.888143 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8cl8\" (UniqueName: \"kubernetes.io/projected/46fa569b-4ed4-4fcb-95ee-d3312a87bcc1-kube-api-access-d8cl8\") pod \"46fa569b-4ed4-4fcb-95ee-d3312a87bcc1\" (UID: \"46fa569b-4ed4-4fcb-95ee-d3312a87bcc1\") " Jul 10 00:29:02.888206 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:02.888192 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/46fa569b-4ed4-4fcb-95ee-d3312a87bcc1-config-file\") pod \"46fa569b-4ed4-4fcb-95ee-d3312a87bcc1\" (UID: \"46fa569b-4ed4-4fcb-95ee-d3312a87bcc1\") " Jul 10 00:29:02.888529 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:02.888503 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46fa569b-4ed4-4fcb-95ee-d3312a87bcc1-config-file" (OuterVolumeSpecName: "config-file") pod "46fa569b-4ed4-4fcb-95ee-d3312a87bcc1" (UID: "46fa569b-4ed4-4fcb-95ee-d3312a87bcc1"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:29:02.890213 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:02.890181 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46fa569b-4ed4-4fcb-95ee-d3312a87bcc1-kube-api-access-d8cl8" (OuterVolumeSpecName: "kube-api-access-d8cl8") pod "46fa569b-4ed4-4fcb-95ee-d3312a87bcc1" (UID: "46fa569b-4ed4-4fcb-95ee-d3312a87bcc1"). InnerVolumeSpecName "kube-api-access-d8cl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:29:02.989355 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:02.989327 2559 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/46fa569b-4ed4-4fcb-95ee-d3312a87bcc1-config-file\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:29:02.989355 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:02.989352 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8cl8\" (UniqueName: \"kubernetes.io/projected/46fa569b-4ed4-4fcb-95ee-d3312a87bcc1-kube-api-access-d8cl8\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:29:03.112492 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:03.112467 2559 generic.go:356] "Generic (PLEG): container finished" podID="46fa569b-4ed4-4fcb-95ee-d3312a87bcc1" containerID="104c4a4749fb7b9f651caf7f300fcfef2c87da6f4b85faea2a95f3098c617c81" exitCode=0 Jul 10 00:29:03.112638 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:03.112517 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-gv496" Jul 10 00:29:03.112638 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:03.112554 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-gv496" event={"ID":"46fa569b-4ed4-4fcb-95ee-d3312a87bcc1","Type":"ContainerDied","Data":"104c4a4749fb7b9f651caf7f300fcfef2c87da6f4b85faea2a95f3098c617c81"} Jul 10 00:29:03.112638 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:03.112607 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-gv496" event={"ID":"46fa569b-4ed4-4fcb-95ee-d3312a87bcc1","Type":"ContainerDied","Data":"407e5e6b711701cbecd0d7103f035197baa21602f9ea9ede19df830b5debf591"} Jul 10 00:29:03.112638 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:03.112636 2559 scope.go:117] "RemoveContainer" containerID="104c4a4749fb7b9f651caf7f300fcfef2c87da6f4b85faea2a95f3098c617c81" Jul 10 00:29:03.120794 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:03.120776 2559 scope.go:117] "RemoveContainer" containerID="104c4a4749fb7b9f651caf7f300fcfef2c87da6f4b85faea2a95f3098c617c81" Jul 10 00:29:03.121053 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:29:03.121028 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"104c4a4749fb7b9f651caf7f300fcfef2c87da6f4b85faea2a95f3098c617c81\": container with ID starting with 104c4a4749fb7b9f651caf7f300fcfef2c87da6f4b85faea2a95f3098c617c81 not found: ID does not exist" containerID="104c4a4749fb7b9f651caf7f300fcfef2c87da6f4b85faea2a95f3098c617c81" Jul 10 00:29:03.121107 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:03.121065 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"104c4a4749fb7b9f651caf7f300fcfef2c87da6f4b85faea2a95f3098c617c81"} err="failed to get container status \"104c4a4749fb7b9f651caf7f300fcfef2c87da6f4b85faea2a95f3098c617c81\": rpc error: code = NotFound desc = could not find container \"104c4a4749fb7b9f651caf7f300fcfef2c87da6f4b85faea2a95f3098c617c81\": container with ID starting with 104c4a4749fb7b9f651caf7f300fcfef2c87da6f4b85faea2a95f3098c617c81 not found: ID does not exist" Jul 10 00:29:03.133050 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:03.133026 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gv496"] Jul 10 00:29:03.136695 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:03.136676 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-gv496"] Jul 10 00:29:03.649842 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:03.649814 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46fa569b-4ed4-4fcb-95ee-d3312a87bcc1" path="/var/lib/kubelet/pods/46fa569b-4ed4-4fcb-95ee-d3312a87bcc1/volumes" Jul 10 00:29:07.571280 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:07.571237 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-42kf5"] Jul 10 00:29:07.571724 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:07.571675 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46fa569b-4ed4-4fcb-95ee-d3312a87bcc1" containerName="limitador" Jul 10 00:29:07.571724 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:07.571693 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="46fa569b-4ed4-4fcb-95ee-d3312a87bcc1" containerName="limitador" Jul 10 00:29:07.571830 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:07.571734 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d92c716a-48a1-4e22-a3f4-2e58ea92a8c1" containerName="authorino" Jul 10 00:29:07.571830 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:07.571743 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d92c716a-48a1-4e22-a3f4-2e58ea92a8c1" containerName="authorino" Jul 10 00:29:07.571830 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:07.571819 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="46fa569b-4ed4-4fcb-95ee-d3312a87bcc1" containerName="limitador" Jul 10 00:29:07.571971 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:07.571834 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d92c716a-48a1-4e22-a3f4-2e58ea92a8c1" containerName="authorino" Jul 10 00:29:07.576167 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:07.576145 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-42kf5" Jul 10 00:29:07.578252 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:07.578228 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 10 00:29:07.578362 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:07.578325 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-bbd6d\"" Jul 10 00:29:07.579050 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:07.579023 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 00:29:07.579050 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:07.579032 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 00:29:07.583722 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:07.583698 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-42kf5"] Jul 10 00:29:07.728844 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:07.728813 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5c86\" (UniqueName: \"kubernetes.io/projected/eaffba7c-7ac3-4b24-90f1-4ab01fd79f9c-kube-api-access-q5c86\") pod \"postgres-69fb4594cf-42kf5\" (UID: \"eaffba7c-7ac3-4b24-90f1-4ab01fd79f9c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-42kf5" Jul 10 00:29:07.729004 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:07.728855 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/eaffba7c-7ac3-4b24-90f1-4ab01fd79f9c-data\") pod \"postgres-69fb4594cf-42kf5\" (UID: \"eaffba7c-7ac3-4b24-90f1-4ab01fd79f9c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-42kf5" Jul 10 00:29:07.830015 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:07.829923 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5c86\" (UniqueName: \"kubernetes.io/projected/eaffba7c-7ac3-4b24-90f1-4ab01fd79f9c-kube-api-access-q5c86\") pod \"postgres-69fb4594cf-42kf5\" (UID: \"eaffba7c-7ac3-4b24-90f1-4ab01fd79f9c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-42kf5" Jul 10 00:29:07.830015 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:07.829985 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/eaffba7c-7ac3-4b24-90f1-4ab01fd79f9c-data\") pod \"postgres-69fb4594cf-42kf5\" (UID: \"eaffba7c-7ac3-4b24-90f1-4ab01fd79f9c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-42kf5" Jul 10 00:29:07.830426 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:07.830403 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/eaffba7c-7ac3-4b24-90f1-4ab01fd79f9c-data\") pod \"postgres-69fb4594cf-42kf5\" (UID: \"eaffba7c-7ac3-4b24-90f1-4ab01fd79f9c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-42kf5" Jul 10 00:29:07.837302 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:07.837276 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5c86\" (UniqueName: \"kubernetes.io/projected/eaffba7c-7ac3-4b24-90f1-4ab01fd79f9c-kube-api-access-q5c86\") pod \"postgres-69fb4594cf-42kf5\" (UID: \"eaffba7c-7ac3-4b24-90f1-4ab01fd79f9c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-42kf5" Jul 10 00:29:07.894226 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:07.894208 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-42kf5" Jul 10 00:29:08.016237 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:08.016213 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-42kf5"] Jul 10 00:29:08.018087 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:29:08.018061 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaffba7c_7ac3_4b24_90f1_4ab01fd79f9c.slice/crio-56cab73541ca101e89493aee8eab19d0c8e0ca0542f7a7ce1c89fffd44230c93 WatchSource:0}: Error finding container 56cab73541ca101e89493aee8eab19d0c8e0ca0542f7a7ce1c89fffd44230c93: Status 404 returned error can't find the container with id 56cab73541ca101e89493aee8eab19d0c8e0ca0542f7a7ce1c89fffd44230c93 Jul 10 00:29:08.135957 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:08.135888 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-42kf5" event={"ID":"eaffba7c-7ac3-4b24-90f1-4ab01fd79f9c","Type":"ContainerStarted","Data":"56cab73541ca101e89493aee8eab19d0c8e0ca0542f7a7ce1c89fffd44230c93"} Jul 10 00:29:13.165220 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:13.165180 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-42kf5" event={"ID":"eaffba7c-7ac3-4b24-90f1-4ab01fd79f9c","Type":"ContainerStarted","Data":"cebccf39c19d8ece4fe21dcec97d75f3e8beb11f44d8f9b4a423841ced211af8"} Jul 10 00:29:13.165562 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:13.165297 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-42kf5" Jul 10 00:29:13.180696 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:13.180650 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-42kf5" podStartSLOduration=1.229695123 podStartE2EDuration="6.180638039s" totalImagesPullingTime="4.950942916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:29:13.178564974 +0000 UTC m=+686.111649276" watchObservedRunningTime="2026-07-10 00:29:13.180638039 +0000 UTC m=+686.113722348" Jul 10 00:29:19.198283 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:19.198255 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-42kf5" Jul 10 00:29:26.766610 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:26.766577 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55b69f98cf-mhdtk"] Jul 10 00:29:26.787503 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:26.787471 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55b69f98cf-mhdtk"] Jul 10 00:29:26.787673 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:26.787560 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55b69f98cf-mhdtk" Jul 10 00:29:26.789840 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:26.789813 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 10 00:29:26.789840 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:26.789823 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 00:29:26.790026 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:26.789844 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hv6tr\"" Jul 10 00:29:26.892675 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:26.892645 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/966a606c-02f5-47ea-8a3f-0340f2bf9d2f-webhook-cert\") pod \"maas-controller-55b69f98cf-mhdtk\" (UID: \"966a606c-02f5-47ea-8a3f-0340f2bf9d2f\") " pod="opendatahub/maas-controller-55b69f98cf-mhdtk" Jul 10 00:29:26.892675 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:26.892679 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv6bp\" (UniqueName: \"kubernetes.io/projected/966a606c-02f5-47ea-8a3f-0340f2bf9d2f-kube-api-access-sv6bp\") pod \"maas-controller-55b69f98cf-mhdtk\" (UID: \"966a606c-02f5-47ea-8a3f-0340f2bf9d2f\") " pod="opendatahub/maas-controller-55b69f98cf-mhdtk" Jul 10 00:29:26.950523 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:26.950493 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5fc87c469-s2zgx"] Jul 10 00:29:26.953458 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:26.953438 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fc87c469-s2zgx" Jul 10 00:29:26.962100 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:26.962077 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fc87c469-s2zgx"] Jul 10 00:29:26.993727 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:26.993705 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/966a606c-02f5-47ea-8a3f-0340f2bf9d2f-webhook-cert\") pod \"maas-controller-55b69f98cf-mhdtk\" (UID: \"966a606c-02f5-47ea-8a3f-0340f2bf9d2f\") " pod="opendatahub/maas-controller-55b69f98cf-mhdtk" Jul 10 00:29:26.993816 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:26.993739 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv6bp\" (UniqueName: \"kubernetes.io/projected/966a606c-02f5-47ea-8a3f-0340f2bf9d2f-kube-api-access-sv6bp\") pod \"maas-controller-55b69f98cf-mhdtk\" (UID: \"966a606c-02f5-47ea-8a3f-0340f2bf9d2f\") " pod="opendatahub/maas-controller-55b69f98cf-mhdtk" Jul 10 00:29:26.993816 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:26.993762 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr7sj\" (UniqueName: \"kubernetes.io/projected/53398c89-f037-4eef-92a7-1121be7f9a2c-kube-api-access-kr7sj\") pod \"maas-controller-5fc87c469-s2zgx\" (UID: \"53398c89-f037-4eef-92a7-1121be7f9a2c\") " pod="opendatahub/maas-controller-5fc87c469-s2zgx" Jul 10 00:29:26.993816 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:26.993810 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53398c89-f037-4eef-92a7-1121be7f9a2c-webhook-cert\") pod \"maas-controller-5fc87c469-s2zgx\" (UID: \"53398c89-f037-4eef-92a7-1121be7f9a2c\") " pod="opendatahub/maas-controller-5fc87c469-s2zgx" Jul 10 00:29:26.996005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:26.995980 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/966a606c-02f5-47ea-8a3f-0340f2bf9d2f-webhook-cert\") pod \"maas-controller-55b69f98cf-mhdtk\" (UID: \"966a606c-02f5-47ea-8a3f-0340f2bf9d2f\") " pod="opendatahub/maas-controller-55b69f98cf-mhdtk" Jul 10 00:29:27.000382 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:27.000360 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv6bp\" (UniqueName: \"kubernetes.io/projected/966a606c-02f5-47ea-8a3f-0340f2bf9d2f-kube-api-access-sv6bp\") pod \"maas-controller-55b69f98cf-mhdtk\" (UID: \"966a606c-02f5-47ea-8a3f-0340f2bf9d2f\") " pod="opendatahub/maas-controller-55b69f98cf-mhdtk" Jul 10 00:29:27.094741 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:27.094670 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr7sj\" (UniqueName: \"kubernetes.io/projected/53398c89-f037-4eef-92a7-1121be7f9a2c-kube-api-access-kr7sj\") pod \"maas-controller-5fc87c469-s2zgx\" (UID: \"53398c89-f037-4eef-92a7-1121be7f9a2c\") " pod="opendatahub/maas-controller-5fc87c469-s2zgx" Jul 10 00:29:27.094741 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:27.094721 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53398c89-f037-4eef-92a7-1121be7f9a2c-webhook-cert\") pod \"maas-controller-5fc87c469-s2zgx\" (UID: \"53398c89-f037-4eef-92a7-1121be7f9a2c\") " pod="opendatahub/maas-controller-5fc87c469-s2zgx" Jul 10 00:29:27.096890 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:27.096865 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53398c89-f037-4eef-92a7-1121be7f9a2c-webhook-cert\") pod \"maas-controller-5fc87c469-s2zgx\" (UID: \"53398c89-f037-4eef-92a7-1121be7f9a2c\") " pod="opendatahub/maas-controller-5fc87c469-s2zgx" Jul 10 00:29:27.097679 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:27.097663 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55b69f98cf-mhdtk" Jul 10 00:29:27.102015 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:27.101991 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr7sj\" (UniqueName: \"kubernetes.io/projected/53398c89-f037-4eef-92a7-1121be7f9a2c-kube-api-access-kr7sj\") pod \"maas-controller-5fc87c469-s2zgx\" (UID: \"53398c89-f037-4eef-92a7-1121be7f9a2c\") " pod="opendatahub/maas-controller-5fc87c469-s2zgx" Jul 10 00:29:27.222402 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:27.222377 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55b69f98cf-mhdtk"] Jul 10 00:29:27.224295 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:29:27.224266 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod966a606c_02f5_47ea_8a3f_0340f2bf9d2f.slice/crio-ff7a39883be577355a65117ca45357d5d6c3507f8d700623a03bebc7b75e9cfc WatchSource:0}: Error finding container ff7a39883be577355a65117ca45357d5d6c3507f8d700623a03bebc7b75e9cfc: Status 404 returned error can't find the container with id ff7a39883be577355a65117ca45357d5d6c3507f8d700623a03bebc7b75e9cfc Jul 10 00:29:27.264006 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:27.263985 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fc87c469-s2zgx" Jul 10 00:29:27.386287 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:27.386264 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fc87c469-s2zgx"] Jul 10 00:29:27.387849 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:29:27.387822 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53398c89_f037_4eef_92a7_1121be7f9a2c.slice/crio-a0260493b81b556b0fae50b3d979af3882a6ff0d563af88c4d6eb3b46c347bdf WatchSource:0}: Error finding container a0260493b81b556b0fae50b3d979af3882a6ff0d563af88c4d6eb3b46c347bdf: Status 404 returned error can't find the container with id a0260493b81b556b0fae50b3d979af3882a6ff0d563af88c4d6eb3b46c347bdf Jul 10 00:29:28.233421 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:28.233358 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fc87c469-s2zgx" event={"ID":"53398c89-f037-4eef-92a7-1121be7f9a2c","Type":"ContainerStarted","Data":"a0260493b81b556b0fae50b3d979af3882a6ff0d563af88c4d6eb3b46c347bdf"} Jul 10 00:29:28.235667 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:28.235628 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55b69f98cf-mhdtk" event={"ID":"966a606c-02f5-47ea-8a3f-0340f2bf9d2f","Type":"ContainerStarted","Data":"ff7a39883be577355a65117ca45357d5d6c3507f8d700623a03bebc7b75e9cfc"} Jul 10 00:29:30.247703 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:30.247666 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fc87c469-s2zgx" event={"ID":"53398c89-f037-4eef-92a7-1121be7f9a2c","Type":"ContainerStarted","Data":"a2039159048504b8098e6db52d036e67af94c28646b5d9b3be8bd737d37e7608"} Jul 10 00:29:30.248164 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:30.247778 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fc87c469-s2zgx" Jul 10 00:29:30.249159 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:30.249105 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55b69f98cf-mhdtk" event={"ID":"966a606c-02f5-47ea-8a3f-0340f2bf9d2f","Type":"ContainerStarted","Data":"802ecd4e6dc2cca964954ba8da385cdc8633ba4a50b661880be9aca4ac2d71a7"} Jul 10 00:29:30.249264 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:30.249179 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55b69f98cf-mhdtk" Jul 10 00:29:30.268285 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:30.268241 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5fc87c469-s2zgx" podStartSLOduration=2.178153335 podStartE2EDuration="4.268226958s" totalImagesPullingTime="2.090073623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:29:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:29:30.267586955 +0000 UTC m=+703.200671265" watchObservedRunningTime="2026-07-10 00:29:30.268226958 +0000 UTC m=+703.201311267" Jul 10 00:29:30.290427 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:30.290385 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-55b69f98cf-mhdtk" podStartSLOduration=2.027242388 podStartE2EDuration="4.290373377s" totalImagesPullingTime="2.263130989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:29:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:29:30.289641571 +0000 UTC m=+703.222725878" watchObservedRunningTime="2026-07-10 00:29:30.290373377 +0000 UTC m=+703.223457685" Jul 10 00:29:41.258169 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:41.258140 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55b69f98cf-mhdtk" Jul 10 00:29:41.258570 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:41.258224 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fc87c469-s2zgx" Jul 10 00:29:41.313321 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:41.313285 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55b69f98cf-mhdtk"] Jul 10 00:29:41.313522 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:41.313487 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-55b69f98cf-mhdtk" podUID="966a606c-02f5-47ea-8a3f-0340f2bf9d2f" containerName="manager" containerID="cri-o://802ecd4e6dc2cca964954ba8da385cdc8633ba4a50b661880be9aca4ac2d71a7" gracePeriod=30 Jul 10 00:29:41.560337 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:41.560313 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55b69f98cf-mhdtk" Jul 10 00:29:41.618304 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:41.618273 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/966a606c-02f5-47ea-8a3f-0340f2bf9d2f-webhook-cert\") pod \"966a606c-02f5-47ea-8a3f-0340f2bf9d2f\" (UID: \"966a606c-02f5-47ea-8a3f-0340f2bf9d2f\") " Jul 10 00:29:41.618429 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:41.618333 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sv6bp\" (UniqueName: \"kubernetes.io/projected/966a606c-02f5-47ea-8a3f-0340f2bf9d2f-kube-api-access-sv6bp\") pod \"966a606c-02f5-47ea-8a3f-0340f2bf9d2f\" (UID: \"966a606c-02f5-47ea-8a3f-0340f2bf9d2f\") " Jul 10 00:29:41.620342 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:41.620308 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/966a606c-02f5-47ea-8a3f-0340f2bf9d2f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "966a606c-02f5-47ea-8a3f-0340f2bf9d2f" (UID: "966a606c-02f5-47ea-8a3f-0340f2bf9d2f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:29:41.620439 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:41.620356 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966a606c-02f5-47ea-8a3f-0340f2bf9d2f-kube-api-access-sv6bp" (OuterVolumeSpecName: "kube-api-access-sv6bp") pod "966a606c-02f5-47ea-8a3f-0340f2bf9d2f" (UID: "966a606c-02f5-47ea-8a3f-0340f2bf9d2f"). InnerVolumeSpecName "kube-api-access-sv6bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:29:41.720568 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:41.720543 2559 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/966a606c-02f5-47ea-8a3f-0340f2bf9d2f-webhook-cert\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:29:41.720568 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:41.720566 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv6bp\" (UniqueName: \"kubernetes.io/projected/966a606c-02f5-47ea-8a3f-0340f2bf9d2f-kube-api-access-sv6bp\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:29:42.306032 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:42.305996 2559 generic.go:356] "Generic (PLEG): container finished" podID="966a606c-02f5-47ea-8a3f-0340f2bf9d2f" containerID="802ecd4e6dc2cca964954ba8da385cdc8633ba4a50b661880be9aca4ac2d71a7" exitCode=0 Jul 10 00:29:42.306476 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:42.306047 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55b69f98cf-mhdtk" event={"ID":"966a606c-02f5-47ea-8a3f-0340f2bf9d2f","Type":"ContainerDied","Data":"802ecd4e6dc2cca964954ba8da385cdc8633ba4a50b661880be9aca4ac2d71a7"} Jul 10 00:29:42.306476 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:42.306058 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55b69f98cf-mhdtk" Jul 10 00:29:42.306476 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:42.306145 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55b69f98cf-mhdtk" event={"ID":"966a606c-02f5-47ea-8a3f-0340f2bf9d2f","Type":"ContainerDied","Data":"ff7a39883be577355a65117ca45357d5d6c3507f8d700623a03bebc7b75e9cfc"} Jul 10 00:29:42.306476 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:42.306170 2559 scope.go:117] "RemoveContainer" containerID="802ecd4e6dc2cca964954ba8da385cdc8633ba4a50b661880be9aca4ac2d71a7" Jul 10 00:29:42.314431 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:42.314281 2559 scope.go:117] "RemoveContainer" containerID="802ecd4e6dc2cca964954ba8da385cdc8633ba4a50b661880be9aca4ac2d71a7" Jul 10 00:29:42.314555 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:29:42.314537 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"802ecd4e6dc2cca964954ba8da385cdc8633ba4a50b661880be9aca4ac2d71a7\": container with ID starting with 802ecd4e6dc2cca964954ba8da385cdc8633ba4a50b661880be9aca4ac2d71a7 not found: ID does not exist" containerID="802ecd4e6dc2cca964954ba8da385cdc8633ba4a50b661880be9aca4ac2d71a7" Jul 10 00:29:42.314603 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:42.314563 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"802ecd4e6dc2cca964954ba8da385cdc8633ba4a50b661880be9aca4ac2d71a7"} err="failed to get container status \"802ecd4e6dc2cca964954ba8da385cdc8633ba4a50b661880be9aca4ac2d71a7\": rpc error: code = NotFound desc = could not find container \"802ecd4e6dc2cca964954ba8da385cdc8633ba4a50b661880be9aca4ac2d71a7\": container with ID starting with 802ecd4e6dc2cca964954ba8da385cdc8633ba4a50b661880be9aca4ac2d71a7 not found: ID does not exist" Jul 10 00:29:42.321258 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:42.321236 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55b69f98cf-mhdtk"] Jul 10 00:29:42.326329 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:42.326309 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-55b69f98cf-mhdtk"] Jul 10 00:29:43.648796 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:29:43.648766 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="966a606c-02f5-47ea-8a3f-0340f2bf9d2f" path="/var/lib/kubelet/pods/966a606c-02f5-47ea-8a3f-0340f2bf9d2f/volumes" Jul 10 00:30:00.724208 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:00.724172 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6886d9bcc8-g665s"] Jul 10 00:30:00.724783 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:00.724474 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="966a606c-02f5-47ea-8a3f-0340f2bf9d2f" containerName="manager" Jul 10 00:30:00.724783 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:00.724484 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="966a606c-02f5-47ea-8a3f-0340f2bf9d2f" containerName="manager" Jul 10 00:30:00.724783 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:00.724548 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="966a606c-02f5-47ea-8a3f-0340f2bf9d2f" containerName="manager" Jul 10 00:30:00.728815 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:00.728795 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6886d9bcc8-g665s" Jul 10 00:30:00.731143 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:00.731102 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 10 00:30:00.731247 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:00.731231 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-wbjng\"" Jul 10 00:30:00.740694 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:00.740670 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6886d9bcc8-g665s"] Jul 10 00:30:00.870604 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:00.870570 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gspwx\" (UniqueName: \"kubernetes.io/projected/349020b7-43cd-4927-8058-1103e4e6f20a-kube-api-access-gspwx\") pod \"maas-api-6886d9bcc8-g665s\" (UID: \"349020b7-43cd-4927-8058-1103e4e6f20a\") " pod="odh-ai-gateway-infra/maas-api-6886d9bcc8-g665s" Jul 10 00:30:00.870775 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:00.870661 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/349020b7-43cd-4927-8058-1103e4e6f20a-openshift-service-ca\") pod \"maas-api-6886d9bcc8-g665s\" (UID: \"349020b7-43cd-4927-8058-1103e4e6f20a\") " pod="odh-ai-gateway-infra/maas-api-6886d9bcc8-g665s" Jul 10 00:30:00.870775 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:00.870720 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/349020b7-43cd-4927-8058-1103e4e6f20a-maas-api-tls\") pod \"maas-api-6886d9bcc8-g665s\" (UID: \"349020b7-43cd-4927-8058-1103e4e6f20a\") " pod="odh-ai-gateway-infra/maas-api-6886d9bcc8-g665s" Jul 10 00:30:00.971529 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:00.971495 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/349020b7-43cd-4927-8058-1103e4e6f20a-openshift-service-ca\") pod \"maas-api-6886d9bcc8-g665s\" (UID: \"349020b7-43cd-4927-8058-1103e4e6f20a\") " pod="odh-ai-gateway-infra/maas-api-6886d9bcc8-g665s" Jul 10 00:30:00.971709 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:00.971551 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/349020b7-43cd-4927-8058-1103e4e6f20a-maas-api-tls\") pod \"maas-api-6886d9bcc8-g665s\" (UID: \"349020b7-43cd-4927-8058-1103e4e6f20a\") " pod="odh-ai-gateway-infra/maas-api-6886d9bcc8-g665s" Jul 10 00:30:00.971709 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:00.971622 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gspwx\" (UniqueName: \"kubernetes.io/projected/349020b7-43cd-4927-8058-1103e4e6f20a-kube-api-access-gspwx\") pod \"maas-api-6886d9bcc8-g665s\" (UID: \"349020b7-43cd-4927-8058-1103e4e6f20a\") " pod="odh-ai-gateway-infra/maas-api-6886d9bcc8-g665s" Jul 10 00:30:00.972250 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:00.972225 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/349020b7-43cd-4927-8058-1103e4e6f20a-openshift-service-ca\") pod \"maas-api-6886d9bcc8-g665s\" (UID: \"349020b7-43cd-4927-8058-1103e4e6f20a\") " pod="odh-ai-gateway-infra/maas-api-6886d9bcc8-g665s" Jul 10 00:30:00.973964 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:00.973937 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/349020b7-43cd-4927-8058-1103e4e6f20a-maas-api-tls\") pod \"maas-api-6886d9bcc8-g665s\" (UID: \"349020b7-43cd-4927-8058-1103e4e6f20a\") " pod="odh-ai-gateway-infra/maas-api-6886d9bcc8-g665s" Jul 10 00:30:00.978988 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:00.978941 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gspwx\" (UniqueName: \"kubernetes.io/projected/349020b7-43cd-4927-8058-1103e4e6f20a-kube-api-access-gspwx\") pod \"maas-api-6886d9bcc8-g665s\" (UID: \"349020b7-43cd-4927-8058-1103e4e6f20a\") " pod="odh-ai-gateway-infra/maas-api-6886d9bcc8-g665s" Jul 10 00:30:01.039482 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.039458 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6886d9bcc8-g665s" Jul 10 00:30:01.182652 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.182626 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6886d9bcc8-g665s"] Jul 10 00:30:01.184847 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:30:01.184817 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod349020b7_43cd_4927_8058_1103e4e6f20a.slice/crio-1e56bedf6314ee0824f17eddcc99328f8db370366f27d084fcc9974736601758 WatchSource:0}: Error finding container 1e56bedf6314ee0824f17eddcc99328f8db370366f27d084fcc9974736601758: Status 404 returned error can't find the container with id 1e56bedf6314ee0824f17eddcc99328f8db370366f27d084fcc9974736601758 Jul 10 00:30:01.186385 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.186364 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 00:30:01.220012 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.219982 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-56gc5"] Jul 10 00:30:01.224369 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.224349 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-56gc5" Jul 10 00:30:01.226597 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.226577 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 10 00:30:01.226678 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.226608 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-7gc7r\"" Jul 10 00:30:01.232832 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.232807 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-56gc5"] Jul 10 00:30:01.258147 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.258101 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-6bgxv"] Jul 10 00:30:01.261363 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.261341 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-6bgxv" Jul 10 00:30:01.272391 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.272367 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-6bgxv"] Jul 10 00:30:01.375296 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.375261 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9nrc\" (UniqueName: \"kubernetes.io/projected/ebc9f72c-30bf-4071-ae1c-928caeb9d033-kube-api-access-t9nrc\") pod \"payload-processing-f9b8d9b9b-6bgxv\" (UID: \"ebc9f72c-30bf-4071-ae1c-928caeb9d033\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-6bgxv" Jul 10 00:30:01.375450 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.375303 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/541871dd-0ab5-41ed-87c7-b688171167b0-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-56gc5\" (UID: \"541871dd-0ab5-41ed-87c7-b688171167b0\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-56gc5" Jul 10 00:30:01.375496 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.375444 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnxhl\" (UniqueName: \"kubernetes.io/projected/541871dd-0ab5-41ed-87c7-b688171167b0-kube-api-access-fnxhl\") pod \"payload-pre-processing-74584bb9f8-56gc5\" (UID: \"541871dd-0ab5-41ed-87c7-b688171167b0\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-56gc5" Jul 10 00:30:01.375534 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.375496 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc9f72c-30bf-4071-ae1c-928caeb9d033-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-6bgxv\" (UID: \"ebc9f72c-30bf-4071-ae1c-928caeb9d033\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-6bgxv" Jul 10 00:30:01.389061 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.389023 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6886d9bcc8-g665s" event={"ID":"349020b7-43cd-4927-8058-1103e4e6f20a","Type":"ContainerStarted","Data":"1e56bedf6314ee0824f17eddcc99328f8db370366f27d084fcc9974736601758"} Jul 10 00:30:01.476717 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.476681 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9nrc\" (UniqueName: \"kubernetes.io/projected/ebc9f72c-30bf-4071-ae1c-928caeb9d033-kube-api-access-t9nrc\") pod \"payload-processing-f9b8d9b9b-6bgxv\" (UID: \"ebc9f72c-30bf-4071-ae1c-928caeb9d033\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-6bgxv" Jul 10 00:30:01.476890 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.476725 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/541871dd-0ab5-41ed-87c7-b688171167b0-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-56gc5\" (UID: \"541871dd-0ab5-41ed-87c7-b688171167b0\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-56gc5" Jul 10 00:30:01.476890 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.476781 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnxhl\" (UniqueName: \"kubernetes.io/projected/541871dd-0ab5-41ed-87c7-b688171167b0-kube-api-access-fnxhl\") pod \"payload-pre-processing-74584bb9f8-56gc5\" (UID: \"541871dd-0ab5-41ed-87c7-b688171167b0\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-56gc5" Jul 10 00:30:01.476890 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.476820 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc9f72c-30bf-4071-ae1c-928caeb9d033-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-6bgxv\" (UID: \"ebc9f72c-30bf-4071-ae1c-928caeb9d033\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-6bgxv" Jul 10 00:30:01.477545 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.477518 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc9f72c-30bf-4071-ae1c-928caeb9d033-plugins-config-volume\") pod \"payload-processing-f9b8d9b9b-6bgxv\" (UID: \"ebc9f72c-30bf-4071-ae1c-928caeb9d033\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-6bgxv" Jul 10 00:30:01.477643 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.477544 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/541871dd-0ab5-41ed-87c7-b688171167b0-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-56gc5\" (UID: \"541871dd-0ab5-41ed-87c7-b688171167b0\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-56gc5" Jul 10 00:30:01.484286 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.484226 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnxhl\" (UniqueName: \"kubernetes.io/projected/541871dd-0ab5-41ed-87c7-b688171167b0-kube-api-access-fnxhl\") pod \"payload-pre-processing-74584bb9f8-56gc5\" (UID: \"541871dd-0ab5-41ed-87c7-b688171167b0\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-56gc5" Jul 10 00:30:01.484286 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.484246 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9nrc\" (UniqueName: \"kubernetes.io/projected/ebc9f72c-30bf-4071-ae1c-928caeb9d033-kube-api-access-t9nrc\") pod \"payload-processing-f9b8d9b9b-6bgxv\" (UID: \"ebc9f72c-30bf-4071-ae1c-928caeb9d033\") " pod="openshift-ingress/payload-processing-f9b8d9b9b-6bgxv" Jul 10 00:30:01.535964 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.535938 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-56gc5" Jul 10 00:30:01.570878 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.570847 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-f9b8d9b9b-6bgxv" Jul 10 00:30:01.676025 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.675992 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-56gc5"] Jul 10 00:30:01.679775 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:30:01.678649 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod541871dd_0ab5_41ed_87c7_b688171167b0.slice/crio-60ef49efa161436f7405cca785529d534a55646f87aedf5cd6f4c0a502ba04d9 WatchSource:0}: Error finding container 60ef49efa161436f7405cca785529d534a55646f87aedf5cd6f4c0a502ba04d9: Status 404 returned error can't find the container with id 60ef49efa161436f7405cca785529d534a55646f87aedf5cd6f4c0a502ba04d9 Jul 10 00:30:01.716140 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:01.716097 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-f9b8d9b9b-6bgxv"] Jul 10 00:30:01.717986 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:30:01.717962 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc9f72c_30bf_4071_ae1c_928caeb9d033.slice/crio-b1ff9f0955db6dcfb5ac6b64d94526cfba459a7ba37d0530319c9d72b6f9817f WatchSource:0}: Error finding container b1ff9f0955db6dcfb5ac6b64d94526cfba459a7ba37d0530319c9d72b6f9817f: Status 404 returned error can't find the container with id b1ff9f0955db6dcfb5ac6b64d94526cfba459a7ba37d0530319c9d72b6f9817f Jul 10 00:30:02.395415 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:02.395375 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-56gc5" event={"ID":"541871dd-0ab5-41ed-87c7-b688171167b0","Type":"ContainerStarted","Data":"60ef49efa161436f7405cca785529d534a55646f87aedf5cd6f4c0a502ba04d9"} Jul 10 00:30:02.396747 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:02.396715 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-6bgxv" event={"ID":"ebc9f72c-30bf-4071-ae1c-928caeb9d033","Type":"ContainerStarted","Data":"b1ff9f0955db6dcfb5ac6b64d94526cfba459a7ba37d0530319c9d72b6f9817f"} Jul 10 00:30:04.411863 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:04.411825 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-f9b8d9b9b-6bgxv" event={"ID":"ebc9f72c-30bf-4071-ae1c-928caeb9d033","Type":"ContainerStarted","Data":"3242db91f468994f5e9c117fcef9fb65531b5e2e5352566d74d4871689818711"} Jul 10 00:30:04.412355 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:04.411948 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-6bgxv" Jul 10 00:30:04.413505 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:04.413475 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-56gc5" event={"ID":"541871dd-0ab5-41ed-87c7-b688171167b0","Type":"ContainerStarted","Data":"213b514e5998dc6050339d1f0d8c720e6b440c927688a29e47bfb9bf1e270fa9"} Jul 10 00:30:04.413617 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:04.413564 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-56gc5" Jul 10 00:30:04.414914 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:04.414894 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6886d9bcc8-g665s" event={"ID":"349020b7-43cd-4927-8058-1103e4e6f20a","Type":"ContainerStarted","Data":"9158bdd9674fc5579eecad223bb9b04df50f3a6834b7f784f4d696e1683d282b"} Jul 10 00:30:04.415030 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:04.415016 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6886d9bcc8-g665s" Jul 10 00:30:04.427072 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:04.427030 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-f9b8d9b9b-6bgxv" podStartSLOduration=1.508253652 podStartE2EDuration="3.427017494s" totalImagesPullingTime="1.918763842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:30:04.425407118 +0000 UTC m=+737.358491438" watchObservedRunningTime="2026-07-10 00:30:04.427017494 +0000 UTC m=+737.360101802" Jul 10 00:30:04.439579 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:04.439543 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-56gc5" podStartSLOduration=1.48219841 podStartE2EDuration="3.43953113s" totalImagesPullingTime="1.95733272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:30:04.437877164 +0000 UTC m=+737.370961490" watchObservedRunningTime="2026-07-10 00:30:04.43953113 +0000 UTC m=+737.372615439" Jul 10 00:30:04.453354 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:04.453316 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6886d9bcc8-g665s" podStartSLOduration=2.081423517 podStartE2EDuration="4.45330338s" totalImagesPullingTime="2.371879863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:30:04.451714452 +0000 UTC m=+737.384798780" watchObservedRunningTime="2026-07-10 00:30:04.45330338 +0000 UTC m=+737.386387688" Jul 10 00:30:09.441685 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:09.441655 2559 generic.go:356] "Generic (PLEG): container finished" podID="cf310fe5-e414-4b36-97d9-c99cb1da8844" containerID="d820428d2c33b032c4c59d7b46bd52b49e04fc982f2d7310801909bd916d84c5" exitCode=0 Jul 10 00:30:09.442019 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:09.441729 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" event={"ID":"cf310fe5-e414-4b36-97d9-c99cb1da8844","Type":"ContainerDied","Data":"d820428d2c33b032c4c59d7b46bd52b49e04fc982f2d7310801909bd916d84c5"} Jul 10 00:30:09.442089 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:09.442076 2559 scope.go:117] "RemoveContainer" containerID="d820428d2c33b032c4c59d7b46bd52b49e04fc982f2d7310801909bd916d84c5" Jul 10 00:30:09.443094 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:09.443071 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:30:09.443184 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:09.443152 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:30:09.443225 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:09.443185 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:30:10.425698 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:10.425666 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6886d9bcc8-g665s" Jul 10 00:30:10.448988 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:10.448960 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" event={"ID":"cf310fe5-e414-4b36-97d9-c99cb1da8844","Type":"ContainerStarted","Data":"7a6b4adad560133b9324e0b00136d888812b73bb78c01f5e01661a54d3c6269a"} Jul 10 00:30:10.731956 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:10.731851 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:30:10.733074 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:10.733044 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-pfqlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 10 00:30:10.733228 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:10.733099 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" podUID="cf310fe5-e414-4b36-97d9-c99cb1da8844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 10 00:30:11.732359 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:11.732324 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-pfqlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 10 00:30:11.732735 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:11.732381 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" podUID="cf310fe5-e414-4b36-97d9-c99cb1da8844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 10 00:30:12.731672 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:12.731626 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-pfqlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 10 00:30:12.731836 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:12.731698 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" podUID="cf310fe5-e414-4b36-97d9-c99cb1da8844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 10 00:30:13.732305 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:13.732270 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-pfqlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 10 00:30:13.732685 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:13.732336 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" podUID="cf310fe5-e414-4b36-97d9-c99cb1da8844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 10 00:30:14.736146 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:14.736099 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:30:14.736606 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:14.736591 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:30:14.737340 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:14.737316 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-pfqlk" Jul 10 00:30:15.422171 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:15.422140 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-f9b8d9b9b-6bgxv" Jul 10 00:30:15.422337 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:15.422217 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-56gc5" Jul 10 00:30:31.572591 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:31.572561 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv"] Jul 10 00:30:31.576246 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:31.576224 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv" Jul 10 00:30:31.578229 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:31.578192 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 00:30:31.578948 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:31.578916 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 10 00:30:31.578948 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:31.578919 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 00:30:31.579091 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:31.578969 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ztfmx\"" Jul 10 00:30:31.584229 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:31.584207 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv"] Jul 10 00:30:31.645073 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:31.645043 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1739f22c-f99b-4680-bdc2-f71c855d6c84-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r9qfv\" (UID: \"1739f22c-f99b-4680-bdc2-f71c855d6c84\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv" Jul 10 00:30:31.645258 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:31.645107 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1739f22c-f99b-4680-bdc2-f71c855d6c84-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r9qfv\" (UID: \"1739f22c-f99b-4680-bdc2-f71c855d6c84\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv" Jul 10 00:30:31.645258 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:31.645229 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1739f22c-f99b-4680-bdc2-f71c855d6c84-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r9qfv\" (UID: \"1739f22c-f99b-4680-bdc2-f71c855d6c84\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv" Jul 10 00:30:31.645375 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:31.645276 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1739f22c-f99b-4680-bdc2-f71c855d6c84-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r9qfv\" (UID: \"1739f22c-f99b-4680-bdc2-f71c855d6c84\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv" Jul 10 00:30:31.645375 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:31.645360 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1739f22c-f99b-4680-bdc2-f71c855d6c84-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r9qfv\" (UID: \"1739f22c-f99b-4680-bdc2-f71c855d6c84\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv" Jul 10 00:30:31.645481 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:31.645390 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgr9z\" (UniqueName: \"kubernetes.io/projected/1739f22c-f99b-4680-bdc2-f71c855d6c84-kube-api-access-zgr9z\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r9qfv\" (UID: \"1739f22c-f99b-4680-bdc2-f71c855d6c84\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv" Jul 10 00:30:31.746470 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:31.746436 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1739f22c-f99b-4680-bdc2-f71c855d6c84-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r9qfv\" (UID: \"1739f22c-f99b-4680-bdc2-f71c855d6c84\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv" Jul 10 00:30:31.746642 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:31.746495 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1739f22c-f99b-4680-bdc2-f71c855d6c84-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r9qfv\" (UID: \"1739f22c-f99b-4680-bdc2-f71c855d6c84\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv" Jul 10 00:30:31.746642 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:31.746523 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1739f22c-f99b-4680-bdc2-f71c855d6c84-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r9qfv\" (UID: \"1739f22c-f99b-4680-bdc2-f71c855d6c84\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv" Jul 10 00:30:31.746642 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:31.746568 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1739f22c-f99b-4680-bdc2-f71c855d6c84-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r9qfv\" (UID: \"1739f22c-f99b-4680-bdc2-f71c855d6c84\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv" Jul 10 00:30:31.746642 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:31.746596 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgr9z\" (UniqueName: \"kubernetes.io/projected/1739f22c-f99b-4680-bdc2-f71c855d6c84-kube-api-access-zgr9z\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r9qfv\" (UID: \"1739f22c-f99b-4680-bdc2-f71c855d6c84\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv" Jul 10 00:30:31.746642 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:31.746632 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1739f22c-f99b-4680-bdc2-f71c855d6c84-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r9qfv\" (UID: \"1739f22c-f99b-4680-bdc2-f71c855d6c84\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv" Jul 10 00:30:31.746897 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:31.746878 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1739f22c-f99b-4680-bdc2-f71c855d6c84-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r9qfv\" (UID: \"1739f22c-f99b-4680-bdc2-f71c855d6c84\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv" Jul 10 00:30:31.746954 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:31.746893 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1739f22c-f99b-4680-bdc2-f71c855d6c84-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r9qfv\" (UID: \"1739f22c-f99b-4680-bdc2-f71c855d6c84\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv" Jul 10 00:30:31.747009 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:31.746979 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1739f22c-f99b-4680-bdc2-f71c855d6c84-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r9qfv\" (UID: \"1739f22c-f99b-4680-bdc2-f71c855d6c84\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv" Jul 10 00:30:31.748781 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:31.748762 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1739f22c-f99b-4680-bdc2-f71c855d6c84-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r9qfv\" (UID: \"1739f22c-f99b-4680-bdc2-f71c855d6c84\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv" Jul 10 00:30:31.748890 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:31.748872 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1739f22c-f99b-4680-bdc2-f71c855d6c84-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r9qfv\" (UID: \"1739f22c-f99b-4680-bdc2-f71c855d6c84\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv" Jul 10 00:30:31.753285 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:31.753260 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgr9z\" (UniqueName: \"kubernetes.io/projected/1739f22c-f99b-4680-bdc2-f71c855d6c84-kube-api-access-zgr9z\") pod \"e2e-distinct-simulated-kserve-59975d69f9-r9qfv\" (UID: \"1739f22c-f99b-4680-bdc2-f71c855d6c84\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv" Jul 10 00:30:31.887354 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:31.887270 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv" Jul 10 00:30:32.016760 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:32.016736 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv"] Jul 10 00:30:32.018585 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:30:32.018548 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1739f22c_f99b_4680_bdc2_f71c855d6c84.slice/crio-dda1d994c080e5c2d866c39c782806e36ff65a272336ffa3b5db1ce2cf71b7f5 WatchSource:0}: Error finding container dda1d994c080e5c2d866c39c782806e36ff65a272336ffa3b5db1ce2cf71b7f5: Status 404 returned error can't find the container with id dda1d994c080e5c2d866c39c782806e36ff65a272336ffa3b5db1ce2cf71b7f5 Jul 10 00:30:32.552806 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:32.552764 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv" event={"ID":"1739f22c-f99b-4680-bdc2-f71c855d6c84","Type":"ContainerStarted","Data":"dda1d994c080e5c2d866c39c782806e36ff65a272336ffa3b5db1ce2cf71b7f5"} Jul 10 00:30:33.559232 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:33.559202 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv" event={"ID":"1739f22c-f99b-4680-bdc2-f71c855d6c84","Type":"ContainerStarted","Data":"bb180bbfce53662547f193619386e57dce10463e1bf9e9543aea48c647d8dc3d"} Jul 10 00:30:33.575235 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:33.575192 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv" podStartSLOduration=1.5618621799999999 podStartE2EDuration="2.575178664s" totalImagesPullingTime="1.013316484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:30:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:30:33.573844795 +0000 UTC m=+766.506929118" watchObservedRunningTime="2026-07-10 00:30:33.575178664 +0000 UTC m=+766.508262972" Jul 10 00:30:41.887758 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:41.887725 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv" Jul 10 00:30:41.888163 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:41.887786 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv" Jul 10 00:30:41.900070 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:41.900046 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv" Jul 10 00:30:42.612196 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:42.612167 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-r9qfv" Jul 10 00:30:54.070900 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:54.070865 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p"] Jul 10 00:30:54.074462 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:54.074440 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p" Jul 10 00:30:54.076476 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:54.076447 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 10 00:30:54.081935 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:54.081910 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p"] Jul 10 00:30:54.239376 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:54.239338 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57vtl\" (UniqueName: \"kubernetes.io/projected/22823e39-0e4a-45ec-91b5-5cec70045e28-kube-api-access-57vtl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p\" (UID: \"22823e39-0e4a-45ec-91b5-5cec70045e28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p" Jul 10 00:30:54.239533 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:54.239382 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22823e39-0e4a-45ec-91b5-5cec70045e28-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p\" (UID: \"22823e39-0e4a-45ec-91b5-5cec70045e28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p" Jul 10 00:30:54.239533 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:54.239493 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22823e39-0e4a-45ec-91b5-5cec70045e28-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p\" (UID: \"22823e39-0e4a-45ec-91b5-5cec70045e28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p" Jul 10 00:30:54.239625 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:54.239546 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22823e39-0e4a-45ec-91b5-5cec70045e28-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p\" (UID: \"22823e39-0e4a-45ec-91b5-5cec70045e28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p" Jul 10 00:30:54.239625 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:54.239601 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22823e39-0e4a-45ec-91b5-5cec70045e28-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p\" (UID: \"22823e39-0e4a-45ec-91b5-5cec70045e28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p" Jul 10 00:30:54.239714 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:54.239638 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22823e39-0e4a-45ec-91b5-5cec70045e28-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p\" (UID: \"22823e39-0e4a-45ec-91b5-5cec70045e28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p" Jul 10 00:30:54.340738 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:54.340658 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22823e39-0e4a-45ec-91b5-5cec70045e28-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p\" (UID: \"22823e39-0e4a-45ec-91b5-5cec70045e28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p" Jul 10 00:30:54.340738 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:54.340700 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22823e39-0e4a-45ec-91b5-5cec70045e28-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p\" (UID: \"22823e39-0e4a-45ec-91b5-5cec70045e28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p" Jul 10 00:30:54.340738 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:54.340723 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22823e39-0e4a-45ec-91b5-5cec70045e28-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p\" (UID: \"22823e39-0e4a-45ec-91b5-5cec70045e28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p" Jul 10 00:30:54.341005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:54.340759 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57vtl\" (UniqueName: \"kubernetes.io/projected/22823e39-0e4a-45ec-91b5-5cec70045e28-kube-api-access-57vtl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p\" (UID: \"22823e39-0e4a-45ec-91b5-5cec70045e28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p" Jul 10 00:30:54.341005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:54.340784 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22823e39-0e4a-45ec-91b5-5cec70045e28-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p\" (UID: \"22823e39-0e4a-45ec-91b5-5cec70045e28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p" Jul 10 00:30:54.341005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:54.340824 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22823e39-0e4a-45ec-91b5-5cec70045e28-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p\" (UID: \"22823e39-0e4a-45ec-91b5-5cec70045e28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p" Jul 10 00:30:54.341187 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:54.341106 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22823e39-0e4a-45ec-91b5-5cec70045e28-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p\" (UID: \"22823e39-0e4a-45ec-91b5-5cec70045e28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p" Jul 10 00:30:54.341251 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:54.341231 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22823e39-0e4a-45ec-91b5-5cec70045e28-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p\" (UID: \"22823e39-0e4a-45ec-91b5-5cec70045e28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p" Jul 10 00:30:54.341328 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:54.341302 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22823e39-0e4a-45ec-91b5-5cec70045e28-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p\" (UID: \"22823e39-0e4a-45ec-91b5-5cec70045e28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p" Jul 10 00:30:54.343067 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:54.343036 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22823e39-0e4a-45ec-91b5-5cec70045e28-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p\" (UID: \"22823e39-0e4a-45ec-91b5-5cec70045e28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p" Jul 10 00:30:54.343278 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:54.343262 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22823e39-0e4a-45ec-91b5-5cec70045e28-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p\" (UID: \"22823e39-0e4a-45ec-91b5-5cec70045e28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p" Jul 10 00:30:54.348355 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:54.348330 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57vtl\" (UniqueName: \"kubernetes.io/projected/22823e39-0e4a-45ec-91b5-5cec70045e28-kube-api-access-57vtl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p\" (UID: \"22823e39-0e4a-45ec-91b5-5cec70045e28\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p" Jul 10 00:30:54.385331 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:54.385305 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p" Jul 10 00:30:54.513673 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:54.513650 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p"] Jul 10 00:30:54.515219 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:30:54.515187 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22823e39_0e4a_45ec_91b5_5cec70045e28.slice/crio-5eb1ae9c8a032ec33ed064c821d5b6c0293637cd0b5ee27874266f5273685ef4 WatchSource:0}: Error finding container 5eb1ae9c8a032ec33ed064c821d5b6c0293637cd0b5ee27874266f5273685ef4: Status 404 returned error can't find the container with id 5eb1ae9c8a032ec33ed064c821d5b6c0293637cd0b5ee27874266f5273685ef4 Jul 10 00:30:54.653488 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:54.653424 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p" event={"ID":"22823e39-0e4a-45ec-91b5-5cec70045e28","Type":"ContainerStarted","Data":"5eb1ae9c8a032ec33ed064c821d5b6c0293637cd0b5ee27874266f5273685ef4"} Jul 10 00:30:55.658929 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:55.658901 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p" event={"ID":"22823e39-0e4a-45ec-91b5-5cec70045e28","Type":"ContainerStarted","Data":"19ec23248efd504f1f85455241e1bdc4172ce865901d61295b8879643ab78b7d"} Jul 10 00:30:55.674799 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:30:55.674757 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p" podStartSLOduration=1.48950186 podStartE2EDuration="1.674743287s" totalImagesPullingTime="185.241427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:30:55.67256815 +0000 UTC m=+788.605652481" watchObservedRunningTime="2026-07-10 00:30:55.674743287 +0000 UTC m=+788.607827595" Jul 10 00:31:04.386001 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:04.385969 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p" Jul 10 00:31:04.386496 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:04.386046 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p" Jul 10 00:31:04.398622 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:04.398593 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p" Jul 10 00:31:04.709132 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:04.709040 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p" Jul 10 00:31:13.977864 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:13.977793 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn"] Jul 10 00:31:13.980126 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:13.980094 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn" Jul 10 00:31:13.982271 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:13.982250 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 10 00:31:13.992420 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:13.992395 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn"] Jul 10 00:31:14.012650 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:14.012627 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69258ad-d5af-4af9-bd35-a2a631670d5d-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnbgn\" (UID: \"f69258ad-d5af-4af9-bd35-a2a631670d5d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn" Jul 10 00:31:14.012761 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:14.012651 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69258ad-d5af-4af9-bd35-a2a631670d5d-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnbgn\" (UID: \"f69258ad-d5af-4af9-bd35-a2a631670d5d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn" Jul 10 00:31:14.012761 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:14.012686 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69258ad-d5af-4af9-bd35-a2a631670d5d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnbgn\" (UID: \"f69258ad-d5af-4af9-bd35-a2a631670d5d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn" Jul 10 00:31:14.012846 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:14.012813 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69258ad-d5af-4af9-bd35-a2a631670d5d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnbgn\" (UID: \"f69258ad-d5af-4af9-bd35-a2a631670d5d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn" Jul 10 00:31:14.012846 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:14.012841 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsh9z\" (UniqueName: \"kubernetes.io/projected/f69258ad-d5af-4af9-bd35-a2a631670d5d-kube-api-access-qsh9z\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnbgn\" (UID: \"f69258ad-d5af-4af9-bd35-a2a631670d5d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn" Jul 10 00:31:14.012916 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:14.012870 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69258ad-d5af-4af9-bd35-a2a631670d5d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnbgn\" (UID: \"f69258ad-d5af-4af9-bd35-a2a631670d5d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn" Jul 10 00:31:14.113972 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:14.113938 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69258ad-d5af-4af9-bd35-a2a631670d5d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnbgn\" (UID: \"f69258ad-d5af-4af9-bd35-a2a631670d5d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn" Jul 10 00:31:14.114289 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:14.114014 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69258ad-d5af-4af9-bd35-a2a631670d5d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnbgn\" (UID: \"f69258ad-d5af-4af9-bd35-a2a631670d5d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn" Jul 10 00:31:14.114289 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:14.114039 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsh9z\" (UniqueName: \"kubernetes.io/projected/f69258ad-d5af-4af9-bd35-a2a631670d5d-kube-api-access-qsh9z\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnbgn\" (UID: \"f69258ad-d5af-4af9-bd35-a2a631670d5d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn" Jul 10 00:31:14.114289 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:14.114063 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69258ad-d5af-4af9-bd35-a2a631670d5d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnbgn\" (UID: \"f69258ad-d5af-4af9-bd35-a2a631670d5d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn" Jul 10 00:31:14.114289 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:14.114088 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69258ad-d5af-4af9-bd35-a2a631670d5d-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnbgn\" (UID: \"f69258ad-d5af-4af9-bd35-a2a631670d5d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn" Jul 10 00:31:14.114289 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:14.114125 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69258ad-d5af-4af9-bd35-a2a631670d5d-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnbgn\" (UID: \"f69258ad-d5af-4af9-bd35-a2a631670d5d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn" Jul 10 00:31:14.114831 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:14.114516 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69258ad-d5af-4af9-bd35-a2a631670d5d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnbgn\" (UID: \"f69258ad-d5af-4af9-bd35-a2a631670d5d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn" Jul 10 00:31:14.114831 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:14.114636 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69258ad-d5af-4af9-bd35-a2a631670d5d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnbgn\" (UID: \"f69258ad-d5af-4af9-bd35-a2a631670d5d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn" Jul 10 00:31:14.114831 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:14.114700 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69258ad-d5af-4af9-bd35-a2a631670d5d-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnbgn\" (UID: \"f69258ad-d5af-4af9-bd35-a2a631670d5d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn" Jul 10 00:31:14.116715 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:14.116690 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69258ad-d5af-4af9-bd35-a2a631670d5d-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnbgn\" (UID: \"f69258ad-d5af-4af9-bd35-a2a631670d5d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn" Jul 10 00:31:14.116824 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:14.116806 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69258ad-d5af-4af9-bd35-a2a631670d5d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnbgn\" (UID: \"f69258ad-d5af-4af9-bd35-a2a631670d5d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn" Jul 10 00:31:14.121353 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:14.121333 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsh9z\" (UniqueName: \"kubernetes.io/projected/f69258ad-d5af-4af9-bd35-a2a631670d5d-kube-api-access-qsh9z\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-jnbgn\" (UID: \"f69258ad-d5af-4af9-bd35-a2a631670d5d\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn" Jul 10 00:31:14.292470 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:14.292422 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn" Jul 10 00:31:14.418859 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:14.418830 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn"] Jul 10 00:31:14.420962 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:31:14.420936 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69258ad_d5af_4af9_bd35_a2a631670d5d.slice/crio-a3de31641321b1398e1da88ca7da6e77146dc1890410e18ef59b69acda04539b WatchSource:0}: Error finding container a3de31641321b1398e1da88ca7da6e77146dc1890410e18ef59b69acda04539b: Status 404 returned error can't find the container with id a3de31641321b1398e1da88ca7da6e77146dc1890410e18ef59b69acda04539b Jul 10 00:31:14.743803 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:14.743767 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn" event={"ID":"f69258ad-d5af-4af9-bd35-a2a631670d5d","Type":"ContainerStarted","Data":"f523a58f7540c3d00dc17e89d4cc53680d151fbb00690170f7f70e1acc712c82"} Jul 10 00:31:14.743961 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:14.743821 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn" event={"ID":"f69258ad-d5af-4af9-bd35-a2a631670d5d","Type":"ContainerStarted","Data":"a3de31641321b1398e1da88ca7da6e77146dc1890410e18ef59b69acda04539b"} Jul 10 00:31:14.758383 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:14.758339 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn" podStartSLOduration=1.600813113 podStartE2EDuration="1.758325562s" totalImagesPullingTime="157.512449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:31:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:31:14.758070552 +0000 UTC m=+807.691154862" watchObservedRunningTime="2026-07-10 00:31:14.758325562 +0000 UTC m=+807.691409871" Jul 10 00:31:24.292885 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:24.292849 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn" Jul 10 00:31:24.293255 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:24.292907 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn" Jul 10 00:31:24.305315 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:24.305291 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn" Jul 10 00:31:24.799038 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:31:24.799010 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-jnbgn" Jul 10 00:32:47.531263 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:32:47.531227 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7zrr_20198eb1-6b3e-4e30-9adf-d4f13df4ee3b/ovn-acl-logging/0.log" Jul 10 00:32:47.531927 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:32:47.531914 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7zrr_20198eb1-6b3e-4e30-9adf-d4f13df4ee3b/ovn-acl-logging/0.log" Jul 10 00:34:05.148973 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:05.148939 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fc87c469-s2zgx"] Jul 10 00:34:05.149505 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:05.149192 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5fc87c469-s2zgx" podUID="53398c89-f037-4eef-92a7-1121be7f9a2c" containerName="manager" containerID="cri-o://a2039159048504b8098e6db52d036e67af94c28646b5d9b3be8bd737d37e7608" gracePeriod=30 Jul 10 00:34:05.398781 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:05.398758 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fc87c469-s2zgx" Jul 10 00:34:05.449841 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:05.449769 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53398c89-f037-4eef-92a7-1121be7f9a2c-webhook-cert\") pod \"53398c89-f037-4eef-92a7-1121be7f9a2c\" (UID: \"53398c89-f037-4eef-92a7-1121be7f9a2c\") " Jul 10 00:34:05.449974 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:05.449856 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kr7sj\" (UniqueName: \"kubernetes.io/projected/53398c89-f037-4eef-92a7-1121be7f9a2c-kube-api-access-kr7sj\") pod \"53398c89-f037-4eef-92a7-1121be7f9a2c\" (UID: \"53398c89-f037-4eef-92a7-1121be7f9a2c\") " Jul 10 00:34:05.451820 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:05.451780 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53398c89-f037-4eef-92a7-1121be7f9a2c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "53398c89-f037-4eef-92a7-1121be7f9a2c" (UID: "53398c89-f037-4eef-92a7-1121be7f9a2c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:34:05.451962 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:05.451880 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53398c89-f037-4eef-92a7-1121be7f9a2c-kube-api-access-kr7sj" (OuterVolumeSpecName: "kube-api-access-kr7sj") pod "53398c89-f037-4eef-92a7-1121be7f9a2c" (UID: "53398c89-f037-4eef-92a7-1121be7f9a2c"). InnerVolumeSpecName "kube-api-access-kr7sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:34:05.530185 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:05.530147 2559 generic.go:356] "Generic (PLEG): container finished" podID="53398c89-f037-4eef-92a7-1121be7f9a2c" containerID="a2039159048504b8098e6db52d036e67af94c28646b5d9b3be8bd737d37e7608" exitCode=0 Jul 10 00:34:05.530185 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:05.530187 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fc87c469-s2zgx" Jul 10 00:34:05.530405 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:05.530231 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fc87c469-s2zgx" event={"ID":"53398c89-f037-4eef-92a7-1121be7f9a2c","Type":"ContainerDied","Data":"a2039159048504b8098e6db52d036e67af94c28646b5d9b3be8bd737d37e7608"} Jul 10 00:34:05.530405 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:05.530289 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fc87c469-s2zgx" event={"ID":"53398c89-f037-4eef-92a7-1121be7f9a2c","Type":"ContainerDied","Data":"a0260493b81b556b0fae50b3d979af3882a6ff0d563af88c4d6eb3b46c347bdf"} Jul 10 00:34:05.530405 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:05.530312 2559 scope.go:117] "RemoveContainer" containerID="a2039159048504b8098e6db52d036e67af94c28646b5d9b3be8bd737d37e7608" Jul 10 00:34:05.547250 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:05.547230 2559 scope.go:117] "RemoveContainer" containerID="a2039159048504b8098e6db52d036e67af94c28646b5d9b3be8bd737d37e7608" Jul 10 00:34:05.547539 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:34:05.547516 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2039159048504b8098e6db52d036e67af94c28646b5d9b3be8bd737d37e7608\": container with ID starting with a2039159048504b8098e6db52d036e67af94c28646b5d9b3be8bd737d37e7608 not found: ID does not exist" containerID="a2039159048504b8098e6db52d036e67af94c28646b5d9b3be8bd737d37e7608" Jul 10 00:34:05.547588 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:05.547549 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2039159048504b8098e6db52d036e67af94c28646b5d9b3be8bd737d37e7608"} err="failed to get container status \"a2039159048504b8098e6db52d036e67af94c28646b5d9b3be8bd737d37e7608\": rpc error: code = NotFound desc = could not find container \"a2039159048504b8098e6db52d036e67af94c28646b5d9b3be8bd737d37e7608\": container with ID starting with a2039159048504b8098e6db52d036e67af94c28646b5d9b3be8bd737d37e7608 not found: ID does not exist" Jul 10 00:34:05.551241 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:05.551224 2559 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/53398c89-f037-4eef-92a7-1121be7f9a2c-webhook-cert\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:34:05.551296 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:05.551244 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr7sj\" (UniqueName: \"kubernetes.io/projected/53398c89-f037-4eef-92a7-1121be7f9a2c-kube-api-access-kr7sj\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:34:05.557448 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:05.557426 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fc87c469-s2zgx"] Jul 10 00:34:05.560718 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:05.560698 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5fc87c469-s2zgx"] Jul 10 00:34:05.650578 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:05.650549 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53398c89-f037-4eef-92a7-1121be7f9a2c" path="/var/lib/kubelet/pods/53398c89-f037-4eef-92a7-1121be7f9a2c/volumes" Jul 10 00:34:06.463351 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:06.463317 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5fc87c469-xw8qr"] Jul 10 00:34:06.463825 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:06.463664 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53398c89-f037-4eef-92a7-1121be7f9a2c" containerName="manager" Jul 10 00:34:06.463825 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:06.463675 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="53398c89-f037-4eef-92a7-1121be7f9a2c" containerName="manager" Jul 10 00:34:06.463825 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:06.463740 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="53398c89-f037-4eef-92a7-1121be7f9a2c" containerName="manager" Jul 10 00:34:06.467968 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:06.467946 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fc87c469-xw8qr" Jul 10 00:34:06.470883 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:06.470860 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 00:34:06.470996 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:06.470899 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 10 00:34:06.470996 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:06.470925 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hv6tr\"" Jul 10 00:34:06.474976 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:06.474953 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fc87c469-xw8qr"] Jul 10 00:34:06.559417 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:06.559385 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d94fc25-7cec-4ffa-9d26-ae8d3de15848-webhook-cert\") pod \"maas-controller-5fc87c469-xw8qr\" (UID: \"4d94fc25-7cec-4ffa-9d26-ae8d3de15848\") " pod="opendatahub/maas-controller-5fc87c469-xw8qr" Jul 10 00:34:06.559577 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:06.559441 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f6b4\" (UniqueName: \"kubernetes.io/projected/4d94fc25-7cec-4ffa-9d26-ae8d3de15848-kube-api-access-5f6b4\") pod \"maas-controller-5fc87c469-xw8qr\" (UID: \"4d94fc25-7cec-4ffa-9d26-ae8d3de15848\") " pod="opendatahub/maas-controller-5fc87c469-xw8qr" Jul 10 00:34:06.660388 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:06.660351 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d94fc25-7cec-4ffa-9d26-ae8d3de15848-webhook-cert\") pod \"maas-controller-5fc87c469-xw8qr\" (UID: \"4d94fc25-7cec-4ffa-9d26-ae8d3de15848\") " pod="opendatahub/maas-controller-5fc87c469-xw8qr" Jul 10 00:34:06.660557 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:06.660421 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f6b4\" (UniqueName: \"kubernetes.io/projected/4d94fc25-7cec-4ffa-9d26-ae8d3de15848-kube-api-access-5f6b4\") pod \"maas-controller-5fc87c469-xw8qr\" (UID: \"4d94fc25-7cec-4ffa-9d26-ae8d3de15848\") " pod="opendatahub/maas-controller-5fc87c469-xw8qr" Jul 10 00:34:06.662854 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:06.662826 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d94fc25-7cec-4ffa-9d26-ae8d3de15848-webhook-cert\") pod \"maas-controller-5fc87c469-xw8qr\" (UID: \"4d94fc25-7cec-4ffa-9d26-ae8d3de15848\") " pod="opendatahub/maas-controller-5fc87c469-xw8qr" Jul 10 00:34:06.668688 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:06.668663 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f6b4\" (UniqueName: \"kubernetes.io/projected/4d94fc25-7cec-4ffa-9d26-ae8d3de15848-kube-api-access-5f6b4\") pod \"maas-controller-5fc87c469-xw8qr\" (UID: \"4d94fc25-7cec-4ffa-9d26-ae8d3de15848\") " pod="opendatahub/maas-controller-5fc87c469-xw8qr" Jul 10 00:34:06.779980 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:06.779953 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fc87c469-xw8qr" Jul 10 00:34:07.118924 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:07.118902 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fc87c469-xw8qr"] Jul 10 00:34:07.121101 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:34:07.121062 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d94fc25_7cec_4ffa_9d26_ae8d3de15848.slice/crio-d1a8481c8af5aefab05bc86333b59be04ba85f6b2009fc7b20d857e01e80c722 WatchSource:0}: Error finding container d1a8481c8af5aefab05bc86333b59be04ba85f6b2009fc7b20d857e01e80c722: Status 404 returned error can't find the container with id d1a8481c8af5aefab05bc86333b59be04ba85f6b2009fc7b20d857e01e80c722 Jul 10 00:34:07.544204 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:07.544169 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fc87c469-xw8qr" event={"ID":"4d94fc25-7cec-4ffa-9d26-ae8d3de15848","Type":"ContainerStarted","Data":"d1a8481c8af5aefab05bc86333b59be04ba85f6b2009fc7b20d857e01e80c722"} Jul 10 00:34:08.550257 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:08.550217 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fc87c469-xw8qr" event={"ID":"4d94fc25-7cec-4ffa-9d26-ae8d3de15848","Type":"ContainerStarted","Data":"f83863ade9b11a427cd3fbebd0996e93328d85eaa8e63c2bb22c4d057b5339b1"} Jul 10 00:34:08.550714 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:08.550283 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fc87c469-xw8qr" Jul 10 00:34:08.565543 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:08.565498 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5fc87c469-xw8qr" podStartSLOduration=1.647144771 podStartE2EDuration="2.565482213s" totalImagesPullingTime="918.337442ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:34:08.564707196 +0000 UTC m=+981.497791505" watchObservedRunningTime="2026-07-10 00:34:08.565482213 +0000 UTC m=+981.498566522" Jul 10 00:34:19.567298 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:34:19.567214 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fc87c469-xw8qr" Jul 10 00:37:47.562092 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:37:47.562063 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7zrr_20198eb1-6b3e-4e30-9adf-d4f13df4ee3b/ovn-acl-logging/0.log" Jul 10 00:37:47.564710 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:37:47.563813 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7zrr_20198eb1-6b3e-4e30-9adf-d4f13df4ee3b/ovn-acl-logging/0.log" Jul 10 00:39:00.542734 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:39:00.542697 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ps96k"] Jul 10 00:39:00.543328 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:39:00.542890 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ps96k" podUID="065e87e2-bfa1-4f82-8108-e8ffd5121933" containerName="manager" containerID="cri-o://7b87d3b73e7ef74fa683fd64bbe2fa7efa1214730a9cf150281f7974d00fd788" gracePeriod=10 Jul 10 00:39:03.685354 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:39:03.685324 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ps96k" Jul 10 00:39:03.806025 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:39:03.805993 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xskb\" (UniqueName: \"kubernetes.io/projected/065e87e2-bfa1-4f82-8108-e8ffd5121933-kube-api-access-8xskb\") pod \"065e87e2-bfa1-4f82-8108-e8ffd5121933\" (UID: \"065e87e2-bfa1-4f82-8108-e8ffd5121933\") " Jul 10 00:39:03.806259 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:39:03.806144 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/065e87e2-bfa1-4f82-8108-e8ffd5121933-extensions-socket-volume\") pod \"065e87e2-bfa1-4f82-8108-e8ffd5121933\" (UID: \"065e87e2-bfa1-4f82-8108-e8ffd5121933\") " Jul 10 00:39:03.806487 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:39:03.806464 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/065e87e2-bfa1-4f82-8108-e8ffd5121933-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "065e87e2-bfa1-4f82-8108-e8ffd5121933" (UID: "065e87e2-bfa1-4f82-8108-e8ffd5121933"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:39:03.808061 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:39:03.808046 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/065e87e2-bfa1-4f82-8108-e8ffd5121933-kube-api-access-8xskb" (OuterVolumeSpecName: "kube-api-access-8xskb") pod "065e87e2-bfa1-4f82-8108-e8ffd5121933" (UID: "065e87e2-bfa1-4f82-8108-e8ffd5121933"). InnerVolumeSpecName "kube-api-access-8xskb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:39:03.907542 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:39:03.907500 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/065e87e2-bfa1-4f82-8108-e8ffd5121933-extensions-socket-volume\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:39:03.907542 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:39:03.907535 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xskb\" (UniqueName: \"kubernetes.io/projected/065e87e2-bfa1-4f82-8108-e8ffd5121933-kube-api-access-8xskb\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:39:03.955418 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:39:03.955379 2559 generic.go:356] "Generic (PLEG): container finished" podID="065e87e2-bfa1-4f82-8108-e8ffd5121933" containerID="7b87d3b73e7ef74fa683fd64bbe2fa7efa1214730a9cf150281f7974d00fd788" exitCode=0 Jul 10 00:39:03.955418 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:39:03.955420 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ps96k" Jul 10 00:39:03.955685 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:39:03.955447 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ps96k" event={"ID":"065e87e2-bfa1-4f82-8108-e8ffd5121933","Type":"ContainerDied","Data":"7b87d3b73e7ef74fa683fd64bbe2fa7efa1214730a9cf150281f7974d00fd788"} Jul 10 00:39:03.955685 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:39:03.955500 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ps96k" event={"ID":"065e87e2-bfa1-4f82-8108-e8ffd5121933","Type":"ContainerDied","Data":"dd903b28503f7e4c44df5730ebb4eafc1b8621361140403ae65ff612a5362d84"} Jul 10 00:39:03.955685 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:39:03.955522 2559 scope.go:117] "RemoveContainer" containerID="7b87d3b73e7ef74fa683fd64bbe2fa7efa1214730a9cf150281f7974d00fd788" Jul 10 00:39:03.966847 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:39:03.966829 2559 scope.go:117] "RemoveContainer" containerID="7b87d3b73e7ef74fa683fd64bbe2fa7efa1214730a9cf150281f7974d00fd788" Jul 10 00:39:03.969755 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:39:03.969706 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b87d3b73e7ef74fa683fd64bbe2fa7efa1214730a9cf150281f7974d00fd788\": container with ID starting with 7b87d3b73e7ef74fa683fd64bbe2fa7efa1214730a9cf150281f7974d00fd788 not found: ID does not exist" containerID="7b87d3b73e7ef74fa683fd64bbe2fa7efa1214730a9cf150281f7974d00fd788" Jul 10 00:39:03.969865 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:39:03.969751 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b87d3b73e7ef74fa683fd64bbe2fa7efa1214730a9cf150281f7974d00fd788"} err="failed to get container status \"7b87d3b73e7ef74fa683fd64bbe2fa7efa1214730a9cf150281f7974d00fd788\": rpc error: code = NotFound desc = could not find container \"7b87d3b73e7ef74fa683fd64bbe2fa7efa1214730a9cf150281f7974d00fd788\": container with ID starting with 7b87d3b73e7ef74fa683fd64bbe2fa7efa1214730a9cf150281f7974d00fd788 not found: ID does not exist" Jul 10 00:39:03.980663 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:39:03.980599 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ps96k"] Jul 10 00:39:03.989584 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:39:03.989557 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ps96k"] Jul 10 00:39:05.649979 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:39:05.649935 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="065e87e2-bfa1-4f82-8108-e8ffd5121933" path="/var/lib/kubelet/pods/065e87e2-bfa1-4f82-8108-e8ffd5121933/volumes" Jul 10 00:40:01.916217 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:40:01.916180 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-57jts"] Jul 10 00:40:01.916673 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:40:01.916526 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="065e87e2-bfa1-4f82-8108-e8ffd5121933" containerName="manager" Jul 10 00:40:01.916673 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:40:01.916537 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="065e87e2-bfa1-4f82-8108-e8ffd5121933" containerName="manager" Jul 10 00:40:01.919644 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:40:01.917012 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="065e87e2-bfa1-4f82-8108-e8ffd5121933" containerName="manager" Jul 10 00:40:01.924764 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:40:01.924740 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-57jts" Jul 10 00:40:01.926723 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:40:01.926692 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jzj9p\"" Jul 10 00:40:01.929940 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:40:01.929919 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-57jts"] Jul 10 00:40:02.028218 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:40:02.028183 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctqwc\" (UniqueName: \"kubernetes.io/projected/86b52ed9-417f-456f-a759-b59c75763236-kube-api-access-ctqwc\") pod \"kuadrant-operator-controller-manager-86878c769f-57jts\" (UID: \"86b52ed9-417f-456f-a759-b59c75763236\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-57jts" Jul 10 00:40:02.028381 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:40:02.028235 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86b52ed9-417f-456f-a759-b59c75763236-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-57jts\" (UID: \"86b52ed9-417f-456f-a759-b59c75763236\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-57jts" Jul 10 00:40:02.129211 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:40:02.129178 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctqwc\" (UniqueName: \"kubernetes.io/projected/86b52ed9-417f-456f-a759-b59c75763236-kube-api-access-ctqwc\") pod \"kuadrant-operator-controller-manager-86878c769f-57jts\" (UID: \"86b52ed9-417f-456f-a759-b59c75763236\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-57jts" Jul 10 00:40:02.129343 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:40:02.129235 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86b52ed9-417f-456f-a759-b59c75763236-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-57jts\" (UID: \"86b52ed9-417f-456f-a759-b59c75763236\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-57jts" Jul 10 00:40:02.129650 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:40:02.129631 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86b52ed9-417f-456f-a759-b59c75763236-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-57jts\" (UID: \"86b52ed9-417f-456f-a759-b59c75763236\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-57jts" Jul 10 00:40:02.139740 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:40:02.139721 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctqwc\" (UniqueName: \"kubernetes.io/projected/86b52ed9-417f-456f-a759-b59c75763236-kube-api-access-ctqwc\") pod \"kuadrant-operator-controller-manager-86878c769f-57jts\" (UID: \"86b52ed9-417f-456f-a759-b59c75763236\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-57jts" Jul 10 00:40:02.236750 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:40:02.236658 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-57jts" Jul 10 00:40:02.375977 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:40:02.375949 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-57jts"] Jul 10 00:40:02.378923 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:40:02.378892 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b52ed9_417f_456f_a759_b59c75763236.slice/crio-d02b722d36340574044d908339a4ae7164e4e45c28c097fd8f82826cd9163297 WatchSource:0}: Error finding container d02b722d36340574044d908339a4ae7164e4e45c28c097fd8f82826cd9163297: Status 404 returned error can't find the container with id d02b722d36340574044d908339a4ae7164e4e45c28c097fd8f82826cd9163297 Jul 10 00:40:03.235131 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:40:03.235071 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-57jts" event={"ID":"86b52ed9-417f-456f-a759-b59c75763236","Type":"ContainerStarted","Data":"59da2b45d5e0dfb9ba586fde39913cfaf1fdeeef08a74ad745e8cec45b56b754"} Jul 10 00:40:03.235519 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:40:03.235153 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-57jts" event={"ID":"86b52ed9-417f-456f-a759-b59c75763236","Type":"ContainerStarted","Data":"d02b722d36340574044d908339a4ae7164e4e45c28c097fd8f82826cd9163297"} Jul 10 00:40:03.235519 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:40:03.235181 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-57jts" Jul 10 00:40:03.255833 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:40:03.255776 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-57jts" podStartSLOduration=2.255757983 podStartE2EDuration="2.255757983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:40:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:40:03.254892405 +0000 UTC m=+1336.187976732" watchObservedRunningTime="2026-07-10 00:40:03.255757983 +0000 UTC m=+1336.188842294" Jul 10 00:40:14.242738 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:40:14.242697 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-57jts" Jul 10 00:42:47.591745 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:42:47.591615 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7zrr_20198eb1-6b3e-4e30-9adf-d4f13df4ee3b/ovn-acl-logging/0.log" Jul 10 00:42:47.595735 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:42:47.593632 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7zrr_20198eb1-6b3e-4e30-9adf-d4f13df4ee3b/ovn-acl-logging/0.log" Jul 10 00:46:36.719208 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.719099 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5"] Jul 10 00:46:36.723624 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.723488 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:36.725725 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.725702 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d4ecd23a-openshift-default-dockercfg-6qf7q\"" Jul 10 00:46:36.733528 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.733506 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5"] Jul 10 00:46:36.830707 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.830669 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-istio-envoy\") pod \"e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:36.830881 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.830727 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c644970-d688-4887-900f-3b6271ef7ea5-istiod-ca-cert\") pod \"e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:36.830881 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.830765 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c644970-d688-4887-900f-3b6271ef7ea5-istio-podinfo\") pod \"e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:36.830881 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.830816 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfsnn\" (UniqueName: \"kubernetes.io/projected/9c644970-d688-4887-900f-3b6271ef7ea5-kube-api-access-cfsnn\") pod \"e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:36.830881 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.830837 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-workload-socket\") pod \"e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:36.830881 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.830851 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-istio-data\") pod \"e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:36.831073 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.830888 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-workload-certs\") pod \"e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:36.831073 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.830984 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-credential-socket\") pod \"e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:36.831073 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.831023 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c644970-d688-4887-900f-3b6271ef7ea5-istio-token\") pod \"e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:36.931993 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.931956 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-istio-envoy\") pod \"e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:36.931993 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.931999 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c644970-d688-4887-900f-3b6271ef7ea5-istiod-ca-cert\") pod \"e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:36.932260 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.932037 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c644970-d688-4887-900f-3b6271ef7ea5-istio-podinfo\") pod \"e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:36.932260 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.932101 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfsnn\" (UniqueName: \"kubernetes.io/projected/9c644970-d688-4887-900f-3b6271ef7ea5-kube-api-access-cfsnn\") pod \"e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:36.932373 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.932255 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-workload-socket\") pod \"e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:36.932373 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.932297 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-istio-data\") pod \"e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:36.932373 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.932340 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-workload-certs\") pod \"e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:36.932373 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.932366 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-credential-socket\") pod \"e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:36.932602 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.932391 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c644970-d688-4887-900f-3b6271ef7ea5-istio-token\") pod \"e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:36.932699 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.932649 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-workload-socket\") pod \"e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:36.932699 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.932668 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-istio-data\") pod \"e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:36.932873 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.932757 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-workload-certs\") pod \"e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:36.932873 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.932819 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c644970-d688-4887-900f-3b6271ef7ea5-istiod-ca-cert\") pod \"e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:36.932991 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.932969 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-credential-socket\") pod \"e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:36.934287 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.934269 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-istio-envoy\") pod \"e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:36.934554 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.934537 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c644970-d688-4887-900f-3b6271ef7ea5-istio-podinfo\") pod \"e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:36.940282 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.940260 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfsnn\" (UniqueName: \"kubernetes.io/projected/9c644970-d688-4887-900f-3b6271ef7ea5-kube-api-access-cfsnn\") pod \"e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:36.940505 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:36.940489 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c644970-d688-4887-900f-3b6271ef7ea5-istio-token\") pod \"e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:37.036936 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:37.036907 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:37.171002 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:37.170969 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5"] Jul 10 00:46:37.172559 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:46:37.172532 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c644970_d688_4887_900f_3b6271ef7ea5.slice/crio-c101e78e722b35c21cbe92d68791158b7d0aa46c3b095bdec27b3453fa9bff88 WatchSource:0}: Error finding container c101e78e722b35c21cbe92d68791158b7d0aa46c3b095bdec27b3453fa9bff88: Status 404 returned error can't find the container with id c101e78e722b35c21cbe92d68791158b7d0aa46c3b095bdec27b3453fa9bff88 Jul 10 00:46:37.174636 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:37.174600 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:46:37.174707 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:37.174689 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:46:37.174755 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:37.174735 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:46:38.123423 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:38.123389 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" event={"ID":"9c644970-d688-4887-900f-3b6271ef7ea5","Type":"ContainerStarted","Data":"f90ee3cd41b22591fd63d53f4a45617b322d9f9f34d1db670d27cef1b5dbd23e"} Jul 10 00:46:38.123814 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:38.123441 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" event={"ID":"9c644970-d688-4887-900f-3b6271ef7ea5","Type":"ContainerStarted","Data":"c101e78e722b35c21cbe92d68791158b7d0aa46c3b095bdec27b3453fa9bff88"} Jul 10 00:46:38.144391 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:38.144343 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" podStartSLOduration=2.144325481 podStartE2EDuration="2.144325481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:46:38.141741858 +0000 UTC m=+1731.074826169" watchObservedRunningTime="2026-07-10 00:46:38.144325481 +0000 UTC m=+1731.077409789" Jul 10 00:46:39.037659 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:39.037621 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:39.042268 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:39.042243 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:39.128247 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:39.128218 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:39.129012 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:39.128983 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:43.461369 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:43.461324 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5"] Jul 10 00:46:43.461909 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:43.461572 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" podUID="9c644970-d688-4887-900f-3b6271ef7ea5" containerName="istio-proxy" containerID="cri-o://f90ee3cd41b22591fd63d53f4a45617b322d9f9f34d1db670d27cef1b5dbd23e" gracePeriod=30 Jul 10 00:46:49.178874 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.178794 2559 generic.go:356] "Generic (PLEG): container finished" podID="9c644970-d688-4887-900f-3b6271ef7ea5" containerID="f90ee3cd41b22591fd63d53f4a45617b322d9f9f34d1db670d27cef1b5dbd23e" exitCode=0 Jul 10 00:46:49.178874 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.178864 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" event={"ID":"9c644970-d688-4887-900f-3b6271ef7ea5","Type":"ContainerDied","Data":"f90ee3cd41b22591fd63d53f4a45617b322d9f9f34d1db670d27cef1b5dbd23e"} Jul 10 00:46:49.215382 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.215333 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:49.248321 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.248298 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-istio-envoy\") pod \"9c644970-d688-4887-900f-3b6271ef7ea5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " Jul 10 00:46:49.248459 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.248343 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c644970-d688-4887-900f-3b6271ef7ea5-istio-podinfo\") pod \"9c644970-d688-4887-900f-3b6271ef7ea5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " Jul 10 00:46:49.248459 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.248395 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-credential-socket\") pod \"9c644970-d688-4887-900f-3b6271ef7ea5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " Jul 10 00:46:49.248459 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.248426 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-istio-data\") pod \"9c644970-d688-4887-900f-3b6271ef7ea5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " Jul 10 00:46:49.248459 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.248446 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-workload-certs\") pod \"9c644970-d688-4887-900f-3b6271ef7ea5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " Jul 10 00:46:49.248657 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.248499 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c644970-d688-4887-900f-3b6271ef7ea5-istio-token\") pod \"9c644970-d688-4887-900f-3b6271ef7ea5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " Jul 10 00:46:49.248657 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.248530 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-workload-socket\") pod \"9c644970-d688-4887-900f-3b6271ef7ea5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " Jul 10 00:46:49.248657 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.248579 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c644970-d688-4887-900f-3b6271ef7ea5-istiod-ca-cert\") pod \"9c644970-d688-4887-900f-3b6271ef7ea5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " Jul 10 00:46:49.248657 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.248648 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfsnn\" (UniqueName: \"kubernetes.io/projected/9c644970-d688-4887-900f-3b6271ef7ea5-kube-api-access-cfsnn\") pod \"9c644970-d688-4887-900f-3b6271ef7ea5\" (UID: \"9c644970-d688-4887-900f-3b6271ef7ea5\") " Jul 10 00:46:49.248840 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.248695 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9c644970-d688-4887-900f-3b6271ef7ea5" (UID: "9c644970-d688-4887-900f-3b6271ef7ea5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:46:49.248907 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.248880 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9c644970-d688-4887-900f-3b6271ef7ea5" (UID: "9c644970-d688-4887-900f-3b6271ef7ea5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:46:49.248994 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.248974 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-credential-socket\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:46:49.249055 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.249000 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-workload-socket\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:46:49.249055 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.248997 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c644970-d688-4887-900f-3b6271ef7ea5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9c644970-d688-4887-900f-3b6271ef7ea5" (UID: "9c644970-d688-4887-900f-3b6271ef7ea5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:46:49.249055 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.249031 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9c644970-d688-4887-900f-3b6271ef7ea5" (UID: "9c644970-d688-4887-900f-3b6271ef7ea5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:46:49.249336 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.249310 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-istio-data" (OuterVolumeSpecName: "istio-data") pod "9c644970-d688-4887-900f-3b6271ef7ea5" (UID: "9c644970-d688-4887-900f-3b6271ef7ea5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:46:49.250818 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.250791 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c644970-d688-4887-900f-3b6271ef7ea5-istio-token" (OuterVolumeSpecName: "istio-token") pod "9c644970-d688-4887-900f-3b6271ef7ea5" (UID: "9c644970-d688-4887-900f-3b6271ef7ea5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:46:49.250949 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.250917 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c644970-d688-4887-900f-3b6271ef7ea5-kube-api-access-cfsnn" (OuterVolumeSpecName: "kube-api-access-cfsnn") pod "9c644970-d688-4887-900f-3b6271ef7ea5" (UID: "9c644970-d688-4887-900f-3b6271ef7ea5"). InnerVolumeSpecName "kube-api-access-cfsnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:46:49.250949 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.250932 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c644970-d688-4887-900f-3b6271ef7ea5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9c644970-d688-4887-900f-3b6271ef7ea5" (UID: "9c644970-d688-4887-900f-3b6271ef7ea5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 00:46:49.251072 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.250977 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9c644970-d688-4887-900f-3b6271ef7ea5" (UID: "9c644970-d688-4887-900f-3b6271ef7ea5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:46:49.350348 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.350275 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c644970-d688-4887-900f-3b6271ef7ea5-istiod-ca-cert\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:46:49.350348 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.350301 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfsnn\" (UniqueName: \"kubernetes.io/projected/9c644970-d688-4887-900f-3b6271ef7ea5-kube-api-access-cfsnn\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:46:49.350348 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.350311 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-istio-envoy\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:46:49.350348 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.350321 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c644970-d688-4887-900f-3b6271ef7ea5-istio-podinfo\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:46:49.350348 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.350329 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-istio-data\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:46:49.350348 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.350337 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c644970-d688-4887-900f-3b6271ef7ea5-workload-certs\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:46:49.350348 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.350344 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c644970-d688-4887-900f-3b6271ef7ea5-istio-token\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:46:49.675974 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.675946 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx"] Jul 10 00:46:49.676380 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.676358 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c644970-d688-4887-900f-3b6271ef7ea5" containerName="istio-proxy" Jul 10 00:46:49.676380 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.676378 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c644970-d688-4887-900f-3b6271ef7ea5" containerName="istio-proxy" Jul 10 00:46:49.676532 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.676451 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c644970-d688-4887-900f-3b6271ef7ea5" containerName="istio-proxy" Jul 10 00:46:49.678548 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.678521 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.680646 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.680621 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a1faa6f3-openshift-default-dockercfg-9fnn5\"" Jul 10 00:46:49.694563 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.694533 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx"] Jul 10 00:46:49.754234 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.754195 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-credential-socket\") pod \"e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.754234 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.754237 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-workload-certs\") pod \"e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.754428 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.754265 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4bb0875-01ac-4b4f-99db-1c22908eb221-istiod-ca-cert\") pod \"e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.754428 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.754360 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4bb0875-01ac-4b4f-99db-1c22908eb221-istio-podinfo\") pod \"e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.754428 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.754397 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbmdf\" (UniqueName: \"kubernetes.io/projected/e4bb0875-01ac-4b4f-99db-1c22908eb221-kube-api-access-nbmdf\") pod \"e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.754536 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.754437 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-istio-envoy\") pod \"e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.754536 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.754452 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4bb0875-01ac-4b4f-99db-1c22908eb221-istio-token\") pod \"e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.754536 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.754505 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-workload-socket\") pod \"e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.754536 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.754532 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-istio-data\") pod \"e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.855254 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.855222 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-workload-socket\") pod \"e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.855254 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.855253 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-istio-data\") pod \"e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.855457 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.855300 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-credential-socket\") pod \"e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.855457 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.855319 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-workload-certs\") pod \"e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.855457 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.855337 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4bb0875-01ac-4b4f-99db-1c22908eb221-istiod-ca-cert\") pod \"e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.855457 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.855402 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4bb0875-01ac-4b4f-99db-1c22908eb221-istio-podinfo\") pod \"e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.855457 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.855435 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbmdf\" (UniqueName: \"kubernetes.io/projected/e4bb0875-01ac-4b4f-99db-1c22908eb221-kube-api-access-nbmdf\") pod \"e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.855716 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.855495 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-istio-envoy\") pod \"e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.855716 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.855520 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4bb0875-01ac-4b4f-99db-1c22908eb221-istio-token\") pod \"e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.855716 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.855682 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-workload-socket\") pod \"e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.855716 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.855704 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-credential-socket\") pod \"e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.856055 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.856028 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4bb0875-01ac-4b4f-99db-1c22908eb221-istiod-ca-cert\") pod \"e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.856204 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.856087 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-istio-data\") pod \"e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.856204 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.856150 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-workload-certs\") pod \"e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.857855 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.857834 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4bb0875-01ac-4b4f-99db-1c22908eb221-istio-podinfo\") pod \"e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.857958 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.857942 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-istio-envoy\") pod \"e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.868898 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.868873 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbmdf\" (UniqueName: \"kubernetes.io/projected/e4bb0875-01ac-4b4f-99db-1c22908eb221-kube-api-access-nbmdf\") pod \"e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.868984 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.868949 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4bb0875-01ac-4b4f-99db-1c22908eb221-istio-token\") pod \"e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:49.990582 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:49.990514 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:50.123294 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:50.123265 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx"] Jul 10 00:46:50.124689 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:46:50.124658 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4bb0875_01ac_4b4f_99db_1c22908eb221.slice/crio-df1fed129e46fa4bd38c53135e516febf831602d5463ef46cfea9358e3edc1c6 WatchSource:0}: Error finding container df1fed129e46fa4bd38c53135e516febf831602d5463ef46cfea9358e3edc1c6: Status 404 returned error can't find the container with id df1fed129e46fa4bd38c53135e516febf831602d5463ef46cfea9358e3edc1c6 Jul 10 00:46:50.126829 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:50.126800 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:46:50.126934 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:50.126861 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:46:50.126934 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:50.126887 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 00:46:50.184930 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:50.184909 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" Jul 10 00:46:50.185316 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:50.184927 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5" event={"ID":"9c644970-d688-4887-900f-3b6271ef7ea5","Type":"ContainerDied","Data":"c101e78e722b35c21cbe92d68791158b7d0aa46c3b095bdec27b3453fa9bff88"} Jul 10 00:46:50.185316 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:50.184989 2559 scope.go:117] "RemoveContainer" containerID="f90ee3cd41b22591fd63d53f4a45617b322d9f9f34d1db670d27cef1b5dbd23e" Jul 10 00:46:50.187961 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:50.187934 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" event={"ID":"e4bb0875-01ac-4b4f-99db-1c22908eb221","Type":"ContainerStarted","Data":"df1fed129e46fa4bd38c53135e516febf831602d5463ef46cfea9358e3edc1c6"} Jul 10 00:46:50.200622 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:50.200601 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5"] Jul 10 00:46:50.204063 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:50.204043 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d4ecd23a-openshift-default-b59f5bc9f-njzc5"] Jul 10 00:46:51.194881 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:51.194848 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" event={"ID":"e4bb0875-01ac-4b4f-99db-1c22908eb221","Type":"ContainerStarted","Data":"f256cef6e032b232110fd65ff000d5c02210c980ff0842d12464cd98a7fe98f4"} Jul 10 00:46:51.218928 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:51.218879 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" podStartSLOduration=2.218861637 podStartE2EDuration="2.218861637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:46:51.216967965 +0000 UTC m=+1744.150052284" watchObservedRunningTime="2026-07-10 00:46:51.218861637 +0000 UTC m=+1744.151945945" Jul 10 00:46:51.649712 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:51.649674 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c644970-d688-4887-900f-3b6271ef7ea5" path="/var/lib/kubelet/pods/9c644970-d688-4887-900f-3b6271ef7ea5/volumes" Jul 10 00:46:51.991208 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:51.991093 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:51.996047 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:51.996020 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:52.200754 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:52.200723 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:46:52.201739 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:46:52.201719 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:47:02.547537 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.547507 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx"] Jul 10 00:47:02.547923 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.547700 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" podUID="e4bb0875-01ac-4b4f-99db-1c22908eb221" containerName="istio-proxy" containerID="cri-o://f256cef6e032b232110fd65ff000d5c02210c980ff0842d12464cd98a7fe98f4" gracePeriod=30 Jul 10 00:47:02.696267 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.696239 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:47:02.778949 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.778916 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbmdf\" (UniqueName: \"kubernetes.io/projected/e4bb0875-01ac-4b4f-99db-1c22908eb221-kube-api-access-nbmdf\") pod \"e4bb0875-01ac-4b4f-99db-1c22908eb221\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " Jul 10 00:47:02.779100 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.778963 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4bb0875-01ac-4b4f-99db-1c22908eb221-istio-token\") pod \"e4bb0875-01ac-4b4f-99db-1c22908eb221\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " Jul 10 00:47:02.779100 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.779008 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-workload-socket\") pod \"e4bb0875-01ac-4b4f-99db-1c22908eb221\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " Jul 10 00:47:02.779100 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.779043 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4bb0875-01ac-4b4f-99db-1c22908eb221-istio-podinfo\") pod \"e4bb0875-01ac-4b4f-99db-1c22908eb221\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " Jul 10 00:47:02.779100 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.779074 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-istio-envoy\") pod \"e4bb0875-01ac-4b4f-99db-1c22908eb221\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " Jul 10 00:47:02.779406 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.779139 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-workload-certs\") pod \"e4bb0875-01ac-4b4f-99db-1c22908eb221\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " Jul 10 00:47:02.779406 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.779183 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-istio-data\") pod \"e4bb0875-01ac-4b4f-99db-1c22908eb221\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " Jul 10 00:47:02.779406 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.779235 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-credential-socket\") pod \"e4bb0875-01ac-4b4f-99db-1c22908eb221\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " Jul 10 00:47:02.779406 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.779259 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4bb0875-01ac-4b4f-99db-1c22908eb221-istiod-ca-cert\") pod \"e4bb0875-01ac-4b4f-99db-1c22908eb221\" (UID: \"e4bb0875-01ac-4b4f-99db-1c22908eb221\") " Jul 10 00:47:02.779612 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.779584 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e4bb0875-01ac-4b4f-99db-1c22908eb221" (UID: "e4bb0875-01ac-4b4f-99db-1c22908eb221"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:47:02.779987 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.779696 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-workload-certs\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:47:02.780098 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.780002 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e4bb0875-01ac-4b4f-99db-1c22908eb221" (UID: "e4bb0875-01ac-4b4f-99db-1c22908eb221"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:47:02.780845 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.780810 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-istio-data" (OuterVolumeSpecName: "istio-data") pod "e4bb0875-01ac-4b4f-99db-1c22908eb221" (UID: "e4bb0875-01ac-4b4f-99db-1c22908eb221"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:47:02.781161 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.780977 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4bb0875-01ac-4b4f-99db-1c22908eb221-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e4bb0875-01ac-4b4f-99db-1c22908eb221" (UID: "e4bb0875-01ac-4b4f-99db-1c22908eb221"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:47:02.782971 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.781062 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e4bb0875-01ac-4b4f-99db-1c22908eb221" (UID: "e4bb0875-01ac-4b4f-99db-1c22908eb221"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:47:02.782971 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.781601 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4bb0875-01ac-4b4f-99db-1c22908eb221-istio-token" (OuterVolumeSpecName: "istio-token") pod "e4bb0875-01ac-4b4f-99db-1c22908eb221" (UID: "e4bb0875-01ac-4b4f-99db-1c22908eb221"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:47:02.782971 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.781817 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4bb0875-01ac-4b4f-99db-1c22908eb221-kube-api-access-nbmdf" (OuterVolumeSpecName: "kube-api-access-nbmdf") pod "e4bb0875-01ac-4b4f-99db-1c22908eb221" (UID: "e4bb0875-01ac-4b4f-99db-1c22908eb221"). InnerVolumeSpecName "kube-api-access-nbmdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:47:02.783179 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.782158 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e4bb0875-01ac-4b4f-99db-1c22908eb221" (UID: "e4bb0875-01ac-4b4f-99db-1c22908eb221"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:47:02.783275 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.783252 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4bb0875-01ac-4b4f-99db-1c22908eb221-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e4bb0875-01ac-4b4f-99db-1c22908eb221" (UID: "e4bb0875-01ac-4b4f-99db-1c22908eb221"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 00:47:02.880123 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.880086 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-istio-envoy\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:47:02.880123 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.880110 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-istio-data\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:47:02.880274 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.880137 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-credential-socket\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:47:02.880274 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.880152 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4bb0875-01ac-4b4f-99db-1c22908eb221-istiod-ca-cert\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:47:02.880274 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.880163 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbmdf\" (UniqueName: \"kubernetes.io/projected/e4bb0875-01ac-4b4f-99db-1c22908eb221-kube-api-access-nbmdf\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:47:02.880274 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.880172 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4bb0875-01ac-4b4f-99db-1c22908eb221-istio-token\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:47:02.880274 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.880179 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bb0875-01ac-4b4f-99db-1c22908eb221-workload-socket\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:47:02.880274 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:02.880187 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4bb0875-01ac-4b4f-99db-1c22908eb221-istio-podinfo\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:47:03.256130 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:03.256094 2559 generic.go:356] "Generic (PLEG): container finished" podID="e4bb0875-01ac-4b4f-99db-1c22908eb221" containerID="f256cef6e032b232110fd65ff000d5c02210c980ff0842d12464cd98a7fe98f4" exitCode=0 Jul 10 00:47:03.256298 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:03.256166 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" Jul 10 00:47:03.256298 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:03.256179 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" event={"ID":"e4bb0875-01ac-4b4f-99db-1c22908eb221","Type":"ContainerDied","Data":"f256cef6e032b232110fd65ff000d5c02210c980ff0842d12464cd98a7fe98f4"} Jul 10 00:47:03.256298 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:03.256241 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx" event={"ID":"e4bb0875-01ac-4b4f-99db-1c22908eb221","Type":"ContainerDied","Data":"df1fed129e46fa4bd38c53135e516febf831602d5463ef46cfea9358e3edc1c6"} Jul 10 00:47:03.256298 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:03.256271 2559 scope.go:117] "RemoveContainer" containerID="f256cef6e032b232110fd65ff000d5c02210c980ff0842d12464cd98a7fe98f4" Jul 10 00:47:03.265282 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:03.265263 2559 scope.go:117] "RemoveContainer" containerID="f256cef6e032b232110fd65ff000d5c02210c980ff0842d12464cd98a7fe98f4" Jul 10 00:47:03.265557 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:47:03.265535 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f256cef6e032b232110fd65ff000d5c02210c980ff0842d12464cd98a7fe98f4\": container with ID starting with f256cef6e032b232110fd65ff000d5c02210c980ff0842d12464cd98a7fe98f4 not found: ID does not exist" containerID="f256cef6e032b232110fd65ff000d5c02210c980ff0842d12464cd98a7fe98f4" Jul 10 00:47:03.265641 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:03.265563 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f256cef6e032b232110fd65ff000d5c02210c980ff0842d12464cd98a7fe98f4"} err="failed to get container status \"f256cef6e032b232110fd65ff000d5c02210c980ff0842d12464cd98a7fe98f4\": rpc error: code = NotFound desc = could not find container \"f256cef6e032b232110fd65ff000d5c02210c980ff0842d12464cd98a7fe98f4\": container with ID starting with f256cef6e032b232110fd65ff000d5c02210c980ff0842d12464cd98a7fe98f4 not found: ID does not exist" Jul 10 00:47:03.277021 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:03.276997 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx"] Jul 10 00:47:03.280531 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:03.280510 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a1faa6f3-openshift-default-b7cd974b9-lgvtx"] Jul 10 00:47:03.649933 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:03.649857 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4bb0875-01ac-4b4f-99db-1c22908eb221" path="/var/lib/kubelet/pods/e4bb0875-01ac-4b4f-99db-1c22908eb221/volumes" Jul 10 00:47:09.930624 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:09.930591 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd"] Jul 10 00:47:09.931066 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:09.931045 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4bb0875-01ac-4b4f-99db-1c22908eb221" containerName="istio-proxy" Jul 10 00:47:09.931128 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:09.931069 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4bb0875-01ac-4b4f-99db-1c22908eb221" containerName="istio-proxy" Jul 10 00:47:09.931233 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:09.931220 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4bb0875-01ac-4b4f-99db-1c22908eb221" containerName="istio-proxy" Jul 10 00:47:09.934246 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:09.934225 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd" Jul 10 00:47:09.936203 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:09.936180 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-64a0d649-serving-cert\"" Jul 10 00:47:09.944895 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:09.944874 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd"] Jul 10 00:47:10.053943 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:10.053912 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dw26\" (UniqueName: \"kubernetes.io/projected/6382c997-edb3-4130-b50d-f2b9faa6dfdc-kube-api-access-4dw26\") pod \"maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd\" (UID: \"6382c997-edb3-4130-b50d-f2b9faa6dfdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd" Jul 10 00:47:10.054092 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:10.053994 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6382c997-edb3-4130-b50d-f2b9faa6dfdc-maas-api-tls\") pod \"maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd\" (UID: \"6382c997-edb3-4130-b50d-f2b9faa6dfdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd" Jul 10 00:47:10.054092 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:10.054013 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6382c997-edb3-4130-b50d-f2b9faa6dfdc-openshift-service-ca\") pod \"maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd\" (UID: \"6382c997-edb3-4130-b50d-f2b9faa6dfdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd" Jul 10 00:47:10.155138 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:10.155089 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6382c997-edb3-4130-b50d-f2b9faa6dfdc-maas-api-tls\") pod \"maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd\" (UID: \"6382c997-edb3-4130-b50d-f2b9faa6dfdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd" Jul 10 00:47:10.155309 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:10.155152 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6382c997-edb3-4130-b50d-f2b9faa6dfdc-openshift-service-ca\") pod \"maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd\" (UID: \"6382c997-edb3-4130-b50d-f2b9faa6dfdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd" Jul 10 00:47:10.155309 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:10.155231 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dw26\" (UniqueName: \"kubernetes.io/projected/6382c997-edb3-4130-b50d-f2b9faa6dfdc-kube-api-access-4dw26\") pod \"maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd\" (UID: \"6382c997-edb3-4130-b50d-f2b9faa6dfdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd" Jul 10 00:47:10.155906 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:10.155879 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6382c997-edb3-4130-b50d-f2b9faa6dfdc-openshift-service-ca\") pod \"maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd\" (UID: \"6382c997-edb3-4130-b50d-f2b9faa6dfdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd" Jul 10 00:47:10.157473 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:10.157454 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6382c997-edb3-4130-b50d-f2b9faa6dfdc-maas-api-tls\") pod \"maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd\" (UID: \"6382c997-edb3-4130-b50d-f2b9faa6dfdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd" Jul 10 00:47:10.163438 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:10.163412 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dw26\" (UniqueName: \"kubernetes.io/projected/6382c997-edb3-4130-b50d-f2b9faa6dfdc-kube-api-access-4dw26\") pod \"maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd\" (UID: \"6382c997-edb3-4130-b50d-f2b9faa6dfdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd" Jul 10 00:47:10.245799 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:10.245735 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd" Jul 10 00:47:10.376197 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:10.376165 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd"] Jul 10 00:47:10.378684 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:47:10.378655 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6382c997_edb3_4130_b50d_f2b9faa6dfdc.slice/crio-096d4aa01111572b3f33bb43cd26752ee777039b5f4b782e801eb4a53255cbfc WatchSource:0}: Error finding container 096d4aa01111572b3f33bb43cd26752ee777039b5f4b782e801eb4a53255cbfc: Status 404 returned error can't find the container with id 096d4aa01111572b3f33bb43cd26752ee777039b5f4b782e801eb4a53255cbfc Jul 10 00:47:10.380307 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:10.380285 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 00:47:11.298223 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:11.298189 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd" event={"ID":"6382c997-edb3-4130-b50d-f2b9faa6dfdc","Type":"ContainerStarted","Data":"b8caaeec7d45ea2e578880ca6a22229b49b9b76425ff9a339988c1c9f6b55454"} Jul 10 00:47:11.298610 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:11.298240 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd" event={"ID":"6382c997-edb3-4130-b50d-f2b9faa6dfdc","Type":"ContainerStarted","Data":"096d4aa01111572b3f33bb43cd26752ee777039b5f4b782e801eb4a53255cbfc"} Jul 10 00:47:11.298610 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:11.298325 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd" Jul 10 00:47:11.317508 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:11.317464 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd" podStartSLOduration=1.988469338 podStartE2EDuration="2.317451472s" totalImagesPullingTime="328.982134ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:47:11.315996238 +0000 UTC m=+1764.249080546" watchObservedRunningTime="2026-07-10 00:47:11.317451472 +0000 UTC m=+1764.250535781" Jul 10 00:47:14.855881 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:14.855848 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd"] Jul 10 00:47:14.856416 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:14.856036 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd" podUID="6382c997-edb3-4130-b50d-f2b9faa6dfdc" containerName="maas-api" containerID="cri-o://b8caaeec7d45ea2e578880ca6a22229b49b9b76425ff9a339988c1c9f6b55454" gracePeriod=30 Jul 10 00:47:15.109559 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:15.109504 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd" Jul 10 00:47:15.199346 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:15.199317 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6382c997-edb3-4130-b50d-f2b9faa6dfdc-maas-api-tls\") pod \"6382c997-edb3-4130-b50d-f2b9faa6dfdc\" (UID: \"6382c997-edb3-4130-b50d-f2b9faa6dfdc\") " Jul 10 00:47:15.199487 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:15.199392 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6382c997-edb3-4130-b50d-f2b9faa6dfdc-openshift-service-ca\") pod \"6382c997-edb3-4130-b50d-f2b9faa6dfdc\" (UID: \"6382c997-edb3-4130-b50d-f2b9faa6dfdc\") " Jul 10 00:47:15.199487 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:15.199466 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dw26\" (UniqueName: \"kubernetes.io/projected/6382c997-edb3-4130-b50d-f2b9faa6dfdc-kube-api-access-4dw26\") pod \"6382c997-edb3-4130-b50d-f2b9faa6dfdc\" (UID: \"6382c997-edb3-4130-b50d-f2b9faa6dfdc\") " Jul 10 00:47:15.199772 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:15.199748 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6382c997-edb3-4130-b50d-f2b9faa6dfdc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6382c997-edb3-4130-b50d-f2b9faa6dfdc" (UID: "6382c997-edb3-4130-b50d-f2b9faa6dfdc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:47:15.201433 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:15.201411 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6382c997-edb3-4130-b50d-f2b9faa6dfdc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6382c997-edb3-4130-b50d-f2b9faa6dfdc" (UID: "6382c997-edb3-4130-b50d-f2b9faa6dfdc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:47:15.201553 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:15.201533 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6382c997-edb3-4130-b50d-f2b9faa6dfdc-kube-api-access-4dw26" (OuterVolumeSpecName: "kube-api-access-4dw26") pod "6382c997-edb3-4130-b50d-f2b9faa6dfdc" (UID: "6382c997-edb3-4130-b50d-f2b9faa6dfdc"). InnerVolumeSpecName "kube-api-access-4dw26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:47:15.300310 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:15.300280 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6382c997-edb3-4130-b50d-f2b9faa6dfdc-maas-api-tls\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:47:15.300310 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:15.300304 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6382c997-edb3-4130-b50d-f2b9faa6dfdc-openshift-service-ca\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:47:15.300310 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:15.300314 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dw26\" (UniqueName: \"kubernetes.io/projected/6382c997-edb3-4130-b50d-f2b9faa6dfdc-kube-api-access-4dw26\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:47:15.320432 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:15.320407 2559 generic.go:356] "Generic (PLEG): container finished" podID="6382c997-edb3-4130-b50d-f2b9faa6dfdc" containerID="b8caaeec7d45ea2e578880ca6a22229b49b9b76425ff9a339988c1c9f6b55454" exitCode=0 Jul 10 00:47:15.320537 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:15.320453 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd" Jul 10 00:47:15.320537 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:15.320477 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd" event={"ID":"6382c997-edb3-4130-b50d-f2b9faa6dfdc","Type":"ContainerDied","Data":"b8caaeec7d45ea2e578880ca6a22229b49b9b76425ff9a339988c1c9f6b55454"} Jul 10 00:47:15.320537 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:15.320516 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd" event={"ID":"6382c997-edb3-4130-b50d-f2b9faa6dfdc","Type":"ContainerDied","Data":"096d4aa01111572b3f33bb43cd26752ee777039b5f4b782e801eb4a53255cbfc"} Jul 10 00:47:15.320537 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:15.320538 2559 scope.go:117] "RemoveContainer" containerID="b8caaeec7d45ea2e578880ca6a22229b49b9b76425ff9a339988c1c9f6b55454" Jul 10 00:47:15.335272 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:15.335251 2559 scope.go:117] "RemoveContainer" containerID="b8caaeec7d45ea2e578880ca6a22229b49b9b76425ff9a339988c1c9f6b55454" Jul 10 00:47:15.335517 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:47:15.335498 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8caaeec7d45ea2e578880ca6a22229b49b9b76425ff9a339988c1c9f6b55454\": container with ID starting with b8caaeec7d45ea2e578880ca6a22229b49b9b76425ff9a339988c1c9f6b55454 not found: ID does not exist" containerID="b8caaeec7d45ea2e578880ca6a22229b49b9b76425ff9a339988c1c9f6b55454" Jul 10 00:47:15.335559 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:15.335526 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8caaeec7d45ea2e578880ca6a22229b49b9b76425ff9a339988c1c9f6b55454"} err="failed to get container status \"b8caaeec7d45ea2e578880ca6a22229b49b9b76425ff9a339988c1c9f6b55454\": rpc error: code = NotFound desc = could not find container \"b8caaeec7d45ea2e578880ca6a22229b49b9b76425ff9a339988c1c9f6b55454\": container with ID starting with b8caaeec7d45ea2e578880ca6a22229b49b9b76425ff9a339988c1c9f6b55454 not found: ID does not exist" Jul 10 00:47:15.343395 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:15.343373 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd"] Jul 10 00:47:15.347575 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:15.347554 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-64a0d649-6546c57c8d-gbctd"] Jul 10 00:47:15.649614 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:15.649586 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6382c997-edb3-4130-b50d-f2b9faa6dfdc" path="/var/lib/kubelet/pods/6382c997-edb3-4130-b50d-f2b9faa6dfdc/volumes" Jul 10 00:47:47.622571 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:47.622542 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7zrr_20198eb1-6b3e-4e30-9adf-d4f13df4ee3b/ovn-acl-logging/0.log" Jul 10 00:47:47.629368 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:47:47.629346 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7zrr_20198eb1-6b3e-4e30-9adf-d4f13df4ee3b/ovn-acl-logging/0.log" Jul 10 00:48:59.387651 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:48:59.387607 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg"] Jul 10 00:48:59.388369 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:48:59.388295 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6382c997-edb3-4130-b50d-f2b9faa6dfdc" containerName="maas-api" Jul 10 00:48:59.388369 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:48:59.388318 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6382c997-edb3-4130-b50d-f2b9faa6dfdc" containerName="maas-api" Jul 10 00:48:59.388575 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:48:59.388434 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6382c997-edb3-4130-b50d-f2b9faa6dfdc" containerName="maas-api" Jul 10 00:48:59.391306 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:48:59.391284 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg" Jul 10 00:48:59.393513 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:48:59.393491 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-23f04d4e-serving-cert\"" Jul 10 00:48:59.400729 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:48:59.400708 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg"] Jul 10 00:48:59.498841 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:48:59.498802 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm568\" (UniqueName: \"kubernetes.io/projected/44dab127-b8c5-42be-8c1a-4f10753998cb-kube-api-access-sm568\") pod \"maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg\" (UID: \"44dab127-b8c5-42be-8c1a-4f10753998cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg" Jul 10 00:48:59.499007 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:48:59.498921 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44dab127-b8c5-42be-8c1a-4f10753998cb-maas-api-tls\") pod \"maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg\" (UID: \"44dab127-b8c5-42be-8c1a-4f10753998cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg" Jul 10 00:48:59.499007 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:48:59.498976 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44dab127-b8c5-42be-8c1a-4f10753998cb-openshift-service-ca\") pod \"maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg\" (UID: \"44dab127-b8c5-42be-8c1a-4f10753998cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg" Jul 10 00:48:59.599867 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:48:59.599833 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm568\" (UniqueName: \"kubernetes.io/projected/44dab127-b8c5-42be-8c1a-4f10753998cb-kube-api-access-sm568\") pod \"maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg\" (UID: \"44dab127-b8c5-42be-8c1a-4f10753998cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg" Jul 10 00:48:59.600029 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:48:59.599888 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44dab127-b8c5-42be-8c1a-4f10753998cb-maas-api-tls\") pod \"maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg\" (UID: \"44dab127-b8c5-42be-8c1a-4f10753998cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg" Jul 10 00:48:59.600085 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:48:59.600018 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44dab127-b8c5-42be-8c1a-4f10753998cb-openshift-service-ca\") pod \"maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg\" (UID: \"44dab127-b8c5-42be-8c1a-4f10753998cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg" Jul 10 00:48:59.600710 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:48:59.600688 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44dab127-b8c5-42be-8c1a-4f10753998cb-openshift-service-ca\") pod \"maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg\" (UID: \"44dab127-b8c5-42be-8c1a-4f10753998cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg" Jul 10 00:48:59.602345 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:48:59.602321 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44dab127-b8c5-42be-8c1a-4f10753998cb-maas-api-tls\") pod \"maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg\" (UID: \"44dab127-b8c5-42be-8c1a-4f10753998cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg" Jul 10 00:48:59.607584 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:48:59.607565 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm568\" (UniqueName: \"kubernetes.io/projected/44dab127-b8c5-42be-8c1a-4f10753998cb-kube-api-access-sm568\") pod \"maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg\" (UID: \"44dab127-b8c5-42be-8c1a-4f10753998cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg" Jul 10 00:48:59.703415 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:48:59.703347 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg" Jul 10 00:48:59.842546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:48:59.842525 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg"] Jul 10 00:48:59.844700 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:48:59.844672 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44dab127_b8c5_42be_8c1a_4f10753998cb.slice/crio-a46e69e97b01a1c577c38955ddf442e77afe2f0836721c0c902ed159aa482f49 WatchSource:0}: Error finding container a46e69e97b01a1c577c38955ddf442e77afe2f0836721c0c902ed159aa482f49: Status 404 returned error can't find the container with id a46e69e97b01a1c577c38955ddf442e77afe2f0836721c0c902ed159aa482f49 Jul 10 00:49:00.799416 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:00.799376 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg" event={"ID":"44dab127-b8c5-42be-8c1a-4f10753998cb","Type":"ContainerStarted","Data":"e7e56c7dbd9ce55cdbfde3e870f350f825038ebb8dc60998df6817c5dfd8c234"} Jul 10 00:49:00.799823 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:00.799431 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg" event={"ID":"44dab127-b8c5-42be-8c1a-4f10753998cb","Type":"ContainerStarted","Data":"a46e69e97b01a1c577c38955ddf442e77afe2f0836721c0c902ed159aa482f49"} Jul 10 00:49:00.799823 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:00.799519 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg" Jul 10 00:49:00.815052 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:00.815009 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg" podStartSLOduration=1.285740807 podStartE2EDuration="1.814993874s" totalImagesPullingTime="529.253067ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:48:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:49:00.813489746 +0000 UTC m=+1873.746574055" watchObservedRunningTime="2026-07-10 00:49:00.814993874 +0000 UTC m=+1873.748078182" Jul 10 00:49:06.810481 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:06.810451 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg" Jul 10 00:49:11.433176 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:11.433093 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg"] Jul 10 00:49:11.433553 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:11.433343 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg" podUID="44dab127-b8c5-42be-8c1a-4f10753998cb" containerName="maas-api" containerID="cri-o://e7e56c7dbd9ce55cdbfde3e870f350f825038ebb8dc60998df6817c5dfd8c234" gracePeriod=30 Jul 10 00:49:11.684110 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:11.684048 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg" Jul 10 00:49:11.706965 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:11.706937 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44dab127-b8c5-42be-8c1a-4f10753998cb-maas-api-tls\") pod \"44dab127-b8c5-42be-8c1a-4f10753998cb\" (UID: \"44dab127-b8c5-42be-8c1a-4f10753998cb\") " Jul 10 00:49:11.707094 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:11.706993 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44dab127-b8c5-42be-8c1a-4f10753998cb-openshift-service-ca\") pod \"44dab127-b8c5-42be-8c1a-4f10753998cb\" (UID: \"44dab127-b8c5-42be-8c1a-4f10753998cb\") " Jul 10 00:49:11.707094 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:11.707062 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sm568\" (UniqueName: \"kubernetes.io/projected/44dab127-b8c5-42be-8c1a-4f10753998cb-kube-api-access-sm568\") pod \"44dab127-b8c5-42be-8c1a-4f10753998cb\" (UID: \"44dab127-b8c5-42be-8c1a-4f10753998cb\") " Jul 10 00:49:11.707405 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:11.707381 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44dab127-b8c5-42be-8c1a-4f10753998cb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "44dab127-b8c5-42be-8c1a-4f10753998cb" (UID: "44dab127-b8c5-42be-8c1a-4f10753998cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:49:11.709130 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:11.709088 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44dab127-b8c5-42be-8c1a-4f10753998cb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "44dab127-b8c5-42be-8c1a-4f10753998cb" (UID: "44dab127-b8c5-42be-8c1a-4f10753998cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:49:11.709223 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:11.709191 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44dab127-b8c5-42be-8c1a-4f10753998cb-kube-api-access-sm568" (OuterVolumeSpecName: "kube-api-access-sm568") pod "44dab127-b8c5-42be-8c1a-4f10753998cb" (UID: "44dab127-b8c5-42be-8c1a-4f10753998cb"). InnerVolumeSpecName "kube-api-access-sm568". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:49:11.808639 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:11.808610 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44dab127-b8c5-42be-8c1a-4f10753998cb-maas-api-tls\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:49:11.808780 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:11.808654 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44dab127-b8c5-42be-8c1a-4f10753998cb-openshift-service-ca\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:49:11.808780 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:11.808664 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm568\" (UniqueName: \"kubernetes.io/projected/44dab127-b8c5-42be-8c1a-4f10753998cb-kube-api-access-sm568\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:49:11.854830 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:11.854800 2559 generic.go:356] "Generic (PLEG): container finished" podID="44dab127-b8c5-42be-8c1a-4f10753998cb" containerID="e7e56c7dbd9ce55cdbfde3e870f350f825038ebb8dc60998df6817c5dfd8c234" exitCode=0 Jul 10 00:49:11.854968 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:11.854847 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg" Jul 10 00:49:11.854968 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:11.854882 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg" event={"ID":"44dab127-b8c5-42be-8c1a-4f10753998cb","Type":"ContainerDied","Data":"e7e56c7dbd9ce55cdbfde3e870f350f825038ebb8dc60998df6817c5dfd8c234"} Jul 10 00:49:11.854968 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:11.854930 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg" event={"ID":"44dab127-b8c5-42be-8c1a-4f10753998cb","Type":"ContainerDied","Data":"a46e69e97b01a1c577c38955ddf442e77afe2f0836721c0c902ed159aa482f49"} Jul 10 00:49:11.854968 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:11.854952 2559 scope.go:117] "RemoveContainer" containerID="e7e56c7dbd9ce55cdbfde3e870f350f825038ebb8dc60998df6817c5dfd8c234" Jul 10 00:49:11.864631 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:11.864616 2559 scope.go:117] "RemoveContainer" containerID="e7e56c7dbd9ce55cdbfde3e870f350f825038ebb8dc60998df6817c5dfd8c234" Jul 10 00:49:11.864869 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:49:11.864853 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7e56c7dbd9ce55cdbfde3e870f350f825038ebb8dc60998df6817c5dfd8c234\": container with ID starting with e7e56c7dbd9ce55cdbfde3e870f350f825038ebb8dc60998df6817c5dfd8c234 not found: ID does not exist" containerID="e7e56c7dbd9ce55cdbfde3e870f350f825038ebb8dc60998df6817c5dfd8c234" Jul 10 00:49:11.864908 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:11.864878 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7e56c7dbd9ce55cdbfde3e870f350f825038ebb8dc60998df6817c5dfd8c234"} err="failed to get container status \"e7e56c7dbd9ce55cdbfde3e870f350f825038ebb8dc60998df6817c5dfd8c234\": rpc error: code = NotFound desc = could not find container \"e7e56c7dbd9ce55cdbfde3e870f350f825038ebb8dc60998df6817c5dfd8c234\": container with ID starting with e7e56c7dbd9ce55cdbfde3e870f350f825038ebb8dc60998df6817c5dfd8c234 not found: ID does not exist" Jul 10 00:49:11.875632 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:11.875613 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg"] Jul 10 00:49:11.878719 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:11.878696 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-23f04d4e-7d5d4bfdc4-mn7wg"] Jul 10 00:49:13.649767 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:13.649736 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44dab127-b8c5-42be-8c1a-4f10753998cb" path="/var/lib/kubelet/pods/44dab127-b8c5-42be-8c1a-4f10753998cb/volumes" Jul 10 00:49:40.436700 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:40.436641 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs"] Jul 10 00:49:40.437649 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:40.437623 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44dab127-b8c5-42be-8c1a-4f10753998cb" containerName="maas-api" Jul 10 00:49:40.437649 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:40.437650 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="44dab127-b8c5-42be-8c1a-4f10753998cb" containerName="maas-api" Jul 10 00:49:40.437899 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:40.437883 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="44dab127-b8c5-42be-8c1a-4f10753998cb" containerName="maas-api" Jul 10 00:49:40.442815 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:40.442693 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs" Jul 10 00:49:40.445594 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:40.445574 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-de57c55c-serving-cert\"" Jul 10 00:49:40.448339 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:40.448316 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs"] Jul 10 00:49:40.547869 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:40.547841 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2b97fe0-760f-4e37-ac92-26540cde93c9-maas-api-tls\") pod \"maas-api-e2e-mt-de57c55c-698f56955-jcdhs\" (UID: \"e2b97fe0-760f-4e37-ac92-26540cde93c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs" Jul 10 00:49:40.548026 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:40.547905 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2b97fe0-760f-4e37-ac92-26540cde93c9-openshift-service-ca\") pod \"maas-api-e2e-mt-de57c55c-698f56955-jcdhs\" (UID: \"e2b97fe0-760f-4e37-ac92-26540cde93c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs" Jul 10 00:49:40.548026 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:40.547951 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4bz7\" (UniqueName: \"kubernetes.io/projected/e2b97fe0-760f-4e37-ac92-26540cde93c9-kube-api-access-p4bz7\") pod \"maas-api-e2e-mt-de57c55c-698f56955-jcdhs\" (UID: \"e2b97fe0-760f-4e37-ac92-26540cde93c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs" Jul 10 00:49:40.648475 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:40.648435 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4bz7\" (UniqueName: \"kubernetes.io/projected/e2b97fe0-760f-4e37-ac92-26540cde93c9-kube-api-access-p4bz7\") pod \"maas-api-e2e-mt-de57c55c-698f56955-jcdhs\" (UID: \"e2b97fe0-760f-4e37-ac92-26540cde93c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs" Jul 10 00:49:40.648627 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:40.648508 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2b97fe0-760f-4e37-ac92-26540cde93c9-maas-api-tls\") pod \"maas-api-e2e-mt-de57c55c-698f56955-jcdhs\" (UID: \"e2b97fe0-760f-4e37-ac92-26540cde93c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs" Jul 10 00:49:40.648627 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:40.648576 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2b97fe0-760f-4e37-ac92-26540cde93c9-openshift-service-ca\") pod \"maas-api-e2e-mt-de57c55c-698f56955-jcdhs\" (UID: \"e2b97fe0-760f-4e37-ac92-26540cde93c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs" Jul 10 00:49:40.648709 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:49:40.648661 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-serving-cert: secret "maas-api-e2e-mt-de57c55c-serving-cert" not found Jul 10 00:49:40.648743 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:49:40.648736 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2b97fe0-760f-4e37-ac92-26540cde93c9-maas-api-tls podName:e2b97fe0-760f-4e37-ac92-26540cde93c9 nodeName:}" failed. No retries permitted until 2026-07-10 00:49:41.14871786 +0000 UTC m=+1914.081802155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e2b97fe0-760f-4e37-ac92-26540cde93c9-maas-api-tls") pod "maas-api-e2e-mt-de57c55c-698f56955-jcdhs" (UID: "e2b97fe0-760f-4e37-ac92-26540cde93c9") : secret "maas-api-e2e-mt-de57c55c-serving-cert" not found Jul 10 00:49:40.649218 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:40.649202 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2b97fe0-760f-4e37-ac92-26540cde93c9-openshift-service-ca\") pod \"maas-api-e2e-mt-de57c55c-698f56955-jcdhs\" (UID: \"e2b97fe0-760f-4e37-ac92-26540cde93c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs" Jul 10 00:49:40.656023 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:40.656003 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4bz7\" (UniqueName: \"kubernetes.io/projected/e2b97fe0-760f-4e37-ac92-26540cde93c9-kube-api-access-p4bz7\") pod \"maas-api-e2e-mt-de57c55c-698f56955-jcdhs\" (UID: \"e2b97fe0-760f-4e37-ac92-26540cde93c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs" Jul 10 00:49:41.153399 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:41.153367 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2b97fe0-760f-4e37-ac92-26540cde93c9-maas-api-tls\") pod \"maas-api-e2e-mt-de57c55c-698f56955-jcdhs\" (UID: \"e2b97fe0-760f-4e37-ac92-26540cde93c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs" Jul 10 00:49:41.156005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:41.155975 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2b97fe0-760f-4e37-ac92-26540cde93c9-maas-api-tls\") pod \"maas-api-e2e-mt-de57c55c-698f56955-jcdhs\" (UID: \"e2b97fe0-760f-4e37-ac92-26540cde93c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs" Jul 10 00:49:41.354609 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:41.354574 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs" Jul 10 00:49:41.489205 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:41.489179 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs"] Jul 10 00:49:41.491288 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:49:41.491248 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b97fe0_760f_4e37_ac92_26540cde93c9.slice/crio-0b4bf2ddcb33a8b7ec0fc494535363ed7df7e822fbc600e14a3d036773b77b5c WatchSource:0}: Error finding container 0b4bf2ddcb33a8b7ec0fc494535363ed7df7e822fbc600e14a3d036773b77b5c: Status 404 returned error can't find the container with id 0b4bf2ddcb33a8b7ec0fc494535363ed7df7e822fbc600e14a3d036773b77b5c Jul 10 00:49:41.996674 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:41.996636 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs" event={"ID":"e2b97fe0-760f-4e37-ac92-26540cde93c9","Type":"ContainerStarted","Data":"62ef78619ae9d8d377aaf4c454f0d50ada82532db9af08420ecd3a5a8c7eed78"} Jul 10 00:49:41.996830 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:41.996703 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs" event={"ID":"e2b97fe0-760f-4e37-ac92-26540cde93c9","Type":"ContainerStarted","Data":"0b4bf2ddcb33a8b7ec0fc494535363ed7df7e822fbc600e14a3d036773b77b5c"} Jul 10 00:49:41.996830 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:41.996731 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs" Jul 10 00:49:42.012495 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:42.012447 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs" podStartSLOduration=1.648789485 podStartE2EDuration="2.012429817s" totalImagesPullingTime="363.640332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:49:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:49:42.010108269 +0000 UTC m=+1914.943192592" watchObservedRunningTime="2026-07-10 00:49:42.012429817 +0000 UTC m=+1914.945514125" Jul 10 00:49:48.007200 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:49:48.007160 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs" Jul 10 00:50:33.023072 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:50:33.023036 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs"] Jul 10 00:50:33.023744 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:50:33.023258 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs" podUID="e2b97fe0-760f-4e37-ac92-26540cde93c9" containerName="maas-api" containerID="cri-o://62ef78619ae9d8d377aaf4c454f0d50ada82532db9af08420ecd3a5a8c7eed78" gracePeriod=30 Jul 10 00:50:33.228982 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:50:33.228953 2559 generic.go:356] "Generic (PLEG): container finished" podID="e2b97fe0-760f-4e37-ac92-26540cde93c9" containerID="62ef78619ae9d8d377aaf4c454f0d50ada82532db9af08420ecd3a5a8c7eed78" exitCode=0 Jul 10 00:50:33.229134 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:50:33.229032 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs" event={"ID":"e2b97fe0-760f-4e37-ac92-26540cde93c9","Type":"ContainerDied","Data":"62ef78619ae9d8d377aaf4c454f0d50ada82532db9af08420ecd3a5a8c7eed78"} Jul 10 00:50:33.268308 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:50:33.268284 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs" Jul 10 00:50:33.401690 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:50:33.401613 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2b97fe0-760f-4e37-ac92-26540cde93c9-openshift-service-ca\") pod \"e2b97fe0-760f-4e37-ac92-26540cde93c9\" (UID: \"e2b97fe0-760f-4e37-ac92-26540cde93c9\") " Jul 10 00:50:33.401830 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:50:33.401700 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4bz7\" (UniqueName: \"kubernetes.io/projected/e2b97fe0-760f-4e37-ac92-26540cde93c9-kube-api-access-p4bz7\") pod \"e2b97fe0-760f-4e37-ac92-26540cde93c9\" (UID: \"e2b97fe0-760f-4e37-ac92-26540cde93c9\") " Jul 10 00:50:33.401830 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:50:33.401729 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2b97fe0-760f-4e37-ac92-26540cde93c9-maas-api-tls\") pod \"e2b97fe0-760f-4e37-ac92-26540cde93c9\" (UID: \"e2b97fe0-760f-4e37-ac92-26540cde93c9\") " Jul 10 00:50:33.401953 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:50:33.401930 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2b97fe0-760f-4e37-ac92-26540cde93c9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e2b97fe0-760f-4e37-ac92-26540cde93c9" (UID: "e2b97fe0-760f-4e37-ac92-26540cde93c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:50:33.403836 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:50:33.403805 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b97fe0-760f-4e37-ac92-26540cde93c9-kube-api-access-p4bz7" (OuterVolumeSpecName: "kube-api-access-p4bz7") pod "e2b97fe0-760f-4e37-ac92-26540cde93c9" (UID: "e2b97fe0-760f-4e37-ac92-26540cde93c9"). InnerVolumeSpecName "kube-api-access-p4bz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:50:33.403963 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:50:33.403951 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2b97fe0-760f-4e37-ac92-26540cde93c9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e2b97fe0-760f-4e37-ac92-26540cde93c9" (UID: "e2b97fe0-760f-4e37-ac92-26540cde93c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:50:33.502530 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:50:33.502496 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2b97fe0-760f-4e37-ac92-26540cde93c9-maas-api-tls\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:50:33.502530 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:50:33.502526 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2b97fe0-760f-4e37-ac92-26540cde93c9-openshift-service-ca\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:50:33.502723 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:50:33.502542 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4bz7\" (UniqueName: \"kubernetes.io/projected/e2b97fe0-760f-4e37-ac92-26540cde93c9-kube-api-access-p4bz7\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:50:34.235040 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:50:34.235006 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs" event={"ID":"e2b97fe0-760f-4e37-ac92-26540cde93c9","Type":"ContainerDied","Data":"0b4bf2ddcb33a8b7ec0fc494535363ed7df7e822fbc600e14a3d036773b77b5c"} Jul 10 00:50:34.235040 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:50:34.235035 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs" Jul 10 00:50:34.235507 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:50:34.235068 2559 scope.go:117] "RemoveContainer" containerID="62ef78619ae9d8d377aaf4c454f0d50ada82532db9af08420ecd3a5a8c7eed78" Jul 10 00:50:34.250656 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:50:34.250633 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs"] Jul 10 00:50:34.256136 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:50:34.256105 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-de57c55c-698f56955-jcdhs"] Jul 10 00:50:35.649539 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:50:35.649508 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2b97fe0-760f-4e37-ac92-26540cde93c9" path="/var/lib/kubelet/pods/e2b97fe0-760f-4e37-ac92-26540cde93c9/volumes" Jul 10 00:52:24.343791 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:24.343707 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4"] Jul 10 00:52:24.346031 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:24.344229 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2b97fe0-760f-4e37-ac92-26540cde93c9" containerName="maas-api" Jul 10 00:52:24.346031 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:24.344244 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2b97fe0-760f-4e37-ac92-26540cde93c9" containerName="maas-api" Jul 10 00:52:24.346031 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:24.344352 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2b97fe0-760f-4e37-ac92-26540cde93c9" containerName="maas-api" Jul 10 00:52:24.347442 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:24.347418 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4" Jul 10 00:52:24.349675 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:24.349659 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3ea0d8-serving-cert\"" Jul 10 00:52:24.357569 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:24.357548 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4"] Jul 10 00:52:24.491391 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:24.491353 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp9kd\" (UniqueName: \"kubernetes.io/projected/31690c34-bdb3-4112-99c6-6f95b657b863-kube-api-access-dp9kd\") pod \"maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4\" (UID: \"31690c34-bdb3-4112-99c6-6f95b657b863\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4" Jul 10 00:52:24.491597 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:24.491411 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31690c34-bdb3-4112-99c6-6f95b657b863-maas-api-tls\") pod \"maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4\" (UID: \"31690c34-bdb3-4112-99c6-6f95b657b863\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4" Jul 10 00:52:24.491597 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:24.491430 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31690c34-bdb3-4112-99c6-6f95b657b863-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4\" (UID: \"31690c34-bdb3-4112-99c6-6f95b657b863\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4" Jul 10 00:52:24.592698 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:24.592664 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp9kd\" (UniqueName: \"kubernetes.io/projected/31690c34-bdb3-4112-99c6-6f95b657b863-kube-api-access-dp9kd\") pod \"maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4\" (UID: \"31690c34-bdb3-4112-99c6-6f95b657b863\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4" Jul 10 00:52:24.592887 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:24.592719 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31690c34-bdb3-4112-99c6-6f95b657b863-maas-api-tls\") pod \"maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4\" (UID: \"31690c34-bdb3-4112-99c6-6f95b657b863\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4" Jul 10 00:52:24.592887 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:24.592739 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31690c34-bdb3-4112-99c6-6f95b657b863-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4\" (UID: \"31690c34-bdb3-4112-99c6-6f95b657b863\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4" Jul 10 00:52:24.592887 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:52:24.592875 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-serving-cert: secret "maas-api-e2e-inf-a-3ea0d8-serving-cert" not found Jul 10 00:52:24.593054 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:52:24.592954 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31690c34-bdb3-4112-99c6-6f95b657b863-maas-api-tls podName:31690c34-bdb3-4112-99c6-6f95b657b863 nodeName:}" failed. No retries permitted until 2026-07-10 00:52:25.09293672 +0000 UTC m=+2078.026021007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/31690c34-bdb3-4112-99c6-6f95b657b863-maas-api-tls") pod "maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4" (UID: "31690c34-bdb3-4112-99c6-6f95b657b863") : secret "maas-api-e2e-inf-a-3ea0d8-serving-cert" not found Jul 10 00:52:24.593318 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:24.593302 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31690c34-bdb3-4112-99c6-6f95b657b863-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4\" (UID: \"31690c34-bdb3-4112-99c6-6f95b657b863\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4" Jul 10 00:52:24.601535 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:24.601485 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp9kd\" (UniqueName: \"kubernetes.io/projected/31690c34-bdb3-4112-99c6-6f95b657b863-kube-api-access-dp9kd\") pod \"maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4\" (UID: \"31690c34-bdb3-4112-99c6-6f95b657b863\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4" Jul 10 00:52:25.097922 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:25.097889 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31690c34-bdb3-4112-99c6-6f95b657b863-maas-api-tls\") pod \"maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4\" (UID: \"31690c34-bdb3-4112-99c6-6f95b657b863\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4" Jul 10 00:52:25.100510 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:25.100478 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31690c34-bdb3-4112-99c6-6f95b657b863-maas-api-tls\") pod \"maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4\" (UID: \"31690c34-bdb3-4112-99c6-6f95b657b863\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4" Jul 10 00:52:25.260576 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:25.260542 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4" Jul 10 00:52:25.396562 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:25.396537 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4"] Jul 10 00:52:25.398726 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:52:25.398702 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31690c34_bdb3_4112_99c6_6f95b657b863.slice/crio-66f586d5ec9b9513033bed198196a977535c72c442cd81785ac9e06e6a64ac9f WatchSource:0}: Error finding container 66f586d5ec9b9513033bed198196a977535c72c442cd81785ac9e06e6a64ac9f: Status 404 returned error can't find the container with id 66f586d5ec9b9513033bed198196a977535c72c442cd81785ac9e06e6a64ac9f Jul 10 00:52:25.399991 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:25.399975 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 00:52:25.752240 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:25.752153 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4" event={"ID":"31690c34-bdb3-4112-99c6-6f95b657b863","Type":"ContainerStarted","Data":"66f586d5ec9b9513033bed198196a977535c72c442cd81785ac9e06e6a64ac9f"} Jul 10 00:52:26.758439 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:26.758393 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4" event={"ID":"31690c34-bdb3-4112-99c6-6f95b657b863","Type":"ContainerStarted","Data":"b11d34f2b46e34c343c4c3bd10530ca4f9bee9d8e75d024f92492e3585d4ac6b"} Jul 10 00:52:26.758826 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:26.758524 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4" Jul 10 00:52:26.772342 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:26.772277 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4" podStartSLOduration=2.063550807 podStartE2EDuration="2.77225524s" totalImagesPullingTime="708.704433ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:52:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:52:26.771893745 +0000 UTC m=+2079.704978056" watchObservedRunningTime="2026-07-10 00:52:26.77225524 +0000 UTC m=+2079.705339607" Jul 10 00:52:32.321846 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:32.321783 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-50269d-8645c458c5-t797g"] Jul 10 00:52:32.325533 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:32.325510 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50269d-8645c458c5-t797g" Jul 10 00:52:32.327829 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:32.327805 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-50269d-serving-cert\"" Jul 10 00:52:32.337027 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:32.336861 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-50269d-8645c458c5-t797g"] Jul 10 00:52:32.468188 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:32.468152 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1647192-7905-465c-abc5-a429969e8a39-openshift-service-ca\") pod \"maas-api-e2e-inf-b-50269d-8645c458c5-t797g\" (UID: \"f1647192-7905-465c-abc5-a429969e8a39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50269d-8645c458c5-t797g" Jul 10 00:52:32.468188 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:32.468191 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2td5f\" (UniqueName: \"kubernetes.io/projected/f1647192-7905-465c-abc5-a429969e8a39-kube-api-access-2td5f\") pod \"maas-api-e2e-inf-b-50269d-8645c458c5-t797g\" (UID: \"f1647192-7905-465c-abc5-a429969e8a39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50269d-8645c458c5-t797g" Jul 10 00:52:32.468405 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:32.468253 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1647192-7905-465c-abc5-a429969e8a39-maas-api-tls\") pod \"maas-api-e2e-inf-b-50269d-8645c458c5-t797g\" (UID: \"f1647192-7905-465c-abc5-a429969e8a39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50269d-8645c458c5-t797g" Jul 10 00:52:32.569061 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:32.569026 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1647192-7905-465c-abc5-a429969e8a39-openshift-service-ca\") pod \"maas-api-e2e-inf-b-50269d-8645c458c5-t797g\" (UID: \"f1647192-7905-465c-abc5-a429969e8a39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50269d-8645c458c5-t797g" Jul 10 00:52:32.569061 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:32.569062 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2td5f\" (UniqueName: \"kubernetes.io/projected/f1647192-7905-465c-abc5-a429969e8a39-kube-api-access-2td5f\") pod \"maas-api-e2e-inf-b-50269d-8645c458c5-t797g\" (UID: \"f1647192-7905-465c-abc5-a429969e8a39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50269d-8645c458c5-t797g" Jul 10 00:52:32.569289 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:32.569107 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1647192-7905-465c-abc5-a429969e8a39-maas-api-tls\") pod \"maas-api-e2e-inf-b-50269d-8645c458c5-t797g\" (UID: \"f1647192-7905-465c-abc5-a429969e8a39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50269d-8645c458c5-t797g" Jul 10 00:52:32.569715 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:32.569695 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1647192-7905-465c-abc5-a429969e8a39-openshift-service-ca\") pod \"maas-api-e2e-inf-b-50269d-8645c458c5-t797g\" (UID: \"f1647192-7905-465c-abc5-a429969e8a39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50269d-8645c458c5-t797g" Jul 10 00:52:32.571541 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:32.571520 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1647192-7905-465c-abc5-a429969e8a39-maas-api-tls\") pod \"maas-api-e2e-inf-b-50269d-8645c458c5-t797g\" (UID: \"f1647192-7905-465c-abc5-a429969e8a39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50269d-8645c458c5-t797g" Jul 10 00:52:32.577056 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:32.576985 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2td5f\" (UniqueName: \"kubernetes.io/projected/f1647192-7905-465c-abc5-a429969e8a39-kube-api-access-2td5f\") pod \"maas-api-e2e-inf-b-50269d-8645c458c5-t797g\" (UID: \"f1647192-7905-465c-abc5-a429969e8a39\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50269d-8645c458c5-t797g" Jul 10 00:52:32.637430 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:32.637405 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50269d-8645c458c5-t797g" Jul 10 00:52:32.772172 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:32.769452 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-50269d-8645c458c5-t797g"] Jul 10 00:52:32.776016 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:32.774430 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4" Jul 10 00:52:32.788346 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:32.788320 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50269d-8645c458c5-t797g" event={"ID":"f1647192-7905-465c-abc5-a429969e8a39","Type":"ContainerStarted","Data":"96055e4a28d195f8756c98d89355dacb2f2be14f45d5e4152965d00e2cf8e274"} Jul 10 00:52:33.794123 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:33.794075 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50269d-8645c458c5-t797g" event={"ID":"f1647192-7905-465c-abc5-a429969e8a39","Type":"ContainerStarted","Data":"d6a0e587378907767176c2e638caadd0c2054de3a21a06cf21d2a77197009d9c"} Jul 10 00:52:33.794522 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:33.794199 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50269d-8645c458c5-t797g" Jul 10 00:52:33.809961 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:33.809916 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50269d-8645c458c5-t797g" podStartSLOduration=1.4330329370000001 podStartE2EDuration="1.809902683s" totalImagesPullingTime="376.869746ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:52:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:52:33.807706593 +0000 UTC m=+2086.740790921" watchObservedRunningTime="2026-07-10 00:52:33.809902683 +0000 UTC m=+2086.742986992" Jul 10 00:52:39.804765 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:39.804735 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50269d-8645c458c5-t797g" Jul 10 00:52:47.650859 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:47.650827 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7zrr_20198eb1-6b3e-4e30-9adf-d4f13df4ee3b/ovn-acl-logging/0.log" Jul 10 00:52:47.659800 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:52:47.659780 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7zrr_20198eb1-6b3e-4e30-9adf-d4f13df4ee3b/ovn-acl-logging/0.log" Jul 10 00:53:29.685095 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:29.685046 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz"] Jul 10 00:53:29.689981 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:29.689950 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" Jul 10 00:53:29.693220 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:29.693187 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-50269d\"/\"default-dockercfg-nlx48\"" Jul 10 00:53:29.693531 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:29.693236 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-50269d\"/\"test-model-50269d-kserve-self-signed-certs\"" Jul 10 00:53:29.693531 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:29.693285 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-50269d\"/\"kube-root-ca.crt\"" Jul 10 00:53:29.693680 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:29.693342 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-50269d\"/\"openshift-service-ca.crt\"" Jul 10 00:53:29.696004 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:29.695982 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz"] Jul 10 00:53:29.769005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:29.768979 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7229f20-9307-4839-ae04-e8733b3bd9f3-tmp-dir\") pod \"test-model-50269d-kserve-6c5866f9c-gqlgz\" (UID: \"a7229f20-9307-4839-ae04-e8733b3bd9f3\") " pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" Jul 10 00:53:29.769148 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:29.769008 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7229f20-9307-4839-ae04-e8733b3bd9f3-model-cache\") pod \"test-model-50269d-kserve-6c5866f9c-gqlgz\" (UID: \"a7229f20-9307-4839-ae04-e8733b3bd9f3\") " pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" Jul 10 00:53:29.769148 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:29.769025 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7229f20-9307-4839-ae04-e8733b3bd9f3-tls-certs\") pod \"test-model-50269d-kserve-6c5866f9c-gqlgz\" (UID: \"a7229f20-9307-4839-ae04-e8733b3bd9f3\") " pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" Jul 10 00:53:29.769148 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:29.769073 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7229f20-9307-4839-ae04-e8733b3bd9f3-home\") pod \"test-model-50269d-kserve-6c5866f9c-gqlgz\" (UID: \"a7229f20-9307-4839-ae04-e8733b3bd9f3\") " pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" Jul 10 00:53:29.769148 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:29.769139 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bmq8\" (UniqueName: \"kubernetes.io/projected/a7229f20-9307-4839-ae04-e8733b3bd9f3-kube-api-access-5bmq8\") pod \"test-model-50269d-kserve-6c5866f9c-gqlgz\" (UID: \"a7229f20-9307-4839-ae04-e8733b3bd9f3\") " pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" Jul 10 00:53:29.769291 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:29.769180 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7229f20-9307-4839-ae04-e8733b3bd9f3-dshm\") pod \"test-model-50269d-kserve-6c5866f9c-gqlgz\" (UID: \"a7229f20-9307-4839-ae04-e8733b3bd9f3\") " pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" Jul 10 00:53:29.869730 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:29.869698 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7229f20-9307-4839-ae04-e8733b3bd9f3-home\") pod \"test-model-50269d-kserve-6c5866f9c-gqlgz\" (UID: \"a7229f20-9307-4839-ae04-e8733b3bd9f3\") " pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" Jul 10 00:53:29.869869 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:29.869743 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bmq8\" (UniqueName: \"kubernetes.io/projected/a7229f20-9307-4839-ae04-e8733b3bd9f3-kube-api-access-5bmq8\") pod \"test-model-50269d-kserve-6c5866f9c-gqlgz\" (UID: \"a7229f20-9307-4839-ae04-e8733b3bd9f3\") " pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" Jul 10 00:53:29.869869 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:29.869788 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7229f20-9307-4839-ae04-e8733b3bd9f3-dshm\") pod \"test-model-50269d-kserve-6c5866f9c-gqlgz\" (UID: \"a7229f20-9307-4839-ae04-e8733b3bd9f3\") " pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" Jul 10 00:53:29.869955 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:29.869883 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7229f20-9307-4839-ae04-e8733b3bd9f3-tmp-dir\") pod \"test-model-50269d-kserve-6c5866f9c-gqlgz\" (UID: \"a7229f20-9307-4839-ae04-e8733b3bd9f3\") " pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" Jul 10 00:53:29.869955 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:29.869909 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7229f20-9307-4839-ae04-e8733b3bd9f3-model-cache\") pod \"test-model-50269d-kserve-6c5866f9c-gqlgz\" (UID: \"a7229f20-9307-4839-ae04-e8733b3bd9f3\") " pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" Jul 10 00:53:29.869955 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:29.869930 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7229f20-9307-4839-ae04-e8733b3bd9f3-tls-certs\") pod \"test-model-50269d-kserve-6c5866f9c-gqlgz\" (UID: \"a7229f20-9307-4839-ae04-e8733b3bd9f3\") " pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" Jul 10 00:53:29.870214 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:29.870193 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7229f20-9307-4839-ae04-e8733b3bd9f3-home\") pod \"test-model-50269d-kserve-6c5866f9c-gqlgz\" (UID: \"a7229f20-9307-4839-ae04-e8733b3bd9f3\") " pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" Jul 10 00:53:29.870301 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:29.870281 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7229f20-9307-4839-ae04-e8733b3bd9f3-tmp-dir\") pod \"test-model-50269d-kserve-6c5866f9c-gqlgz\" (UID: \"a7229f20-9307-4839-ae04-e8733b3bd9f3\") " pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" Jul 10 00:53:29.870362 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:29.870318 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7229f20-9307-4839-ae04-e8733b3bd9f3-model-cache\") pod \"test-model-50269d-kserve-6c5866f9c-gqlgz\" (UID: \"a7229f20-9307-4839-ae04-e8733b3bd9f3\") " pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" Jul 10 00:53:29.872105 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:29.872087 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7229f20-9307-4839-ae04-e8733b3bd9f3-dshm\") pod \"test-model-50269d-kserve-6c5866f9c-gqlgz\" (UID: \"a7229f20-9307-4839-ae04-e8733b3bd9f3\") " pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" Jul 10 00:53:29.872435 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:29.872419 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7229f20-9307-4839-ae04-e8733b3bd9f3-tls-certs\") pod \"test-model-50269d-kserve-6c5866f9c-gqlgz\" (UID: \"a7229f20-9307-4839-ae04-e8733b3bd9f3\") " pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" Jul 10 00:53:29.877968 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:29.877941 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bmq8\" (UniqueName: \"kubernetes.io/projected/a7229f20-9307-4839-ae04-e8733b3bd9f3-kube-api-access-5bmq8\") pod \"test-model-50269d-kserve-6c5866f9c-gqlgz\" (UID: \"a7229f20-9307-4839-ae04-e8733b3bd9f3\") " pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" Jul 10 00:53:30.001410 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:30.001351 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" Jul 10 00:53:30.129690 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:30.129665 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz"] Jul 10 00:53:30.131609 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:53:30.131577 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7229f20_9307_4839_ae04_e8733b3bd9f3.slice/crio-b560ebac4a16138be10b5037895efc227320115234ff2d34d414b6e22ed22b4a WatchSource:0}: Error finding container b560ebac4a16138be10b5037895efc227320115234ff2d34d414b6e22ed22b4a: Status 404 returned error can't find the container with id b560ebac4a16138be10b5037895efc227320115234ff2d34d414b6e22ed22b4a Jul 10 00:53:31.061179 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:31.061144 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" event={"ID":"a7229f20-9307-4839-ae04-e8733b3bd9f3","Type":"ContainerStarted","Data":"696426e1f13b370715a2ec234a40fb720f87ac7cb2c7aab93247be20653765ac"} Jul 10 00:53:31.061546 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:31.061189 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" event={"ID":"a7229f20-9307-4839-ae04-e8733b3bd9f3","Type":"ContainerStarted","Data":"b560ebac4a16138be10b5037895efc227320115234ff2d34d414b6e22ed22b4a"} Jul 10 00:53:31.076735 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:31.076691 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" podStartSLOduration=2.076673375 podStartE2EDuration="2.076673375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:53:31.075891176 +0000 UTC m=+2144.008975485" watchObservedRunningTime="2026-07-10 00:53:31.076673375 +0000 UTC m=+2144.009757684" Jul 10 00:53:40.002143 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:40.002051 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" Jul 10 00:53:40.002143 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:40.002134 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" Jul 10 00:53:40.014289 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:40.014265 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" Jul 10 00:53:40.114812 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:40.114782 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" Jul 10 00:53:51.684696 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:51.684661 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz"] Jul 10 00:53:52.568939 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:52.568894 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4"] Jul 10 00:53:52.569244 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:52.569180 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4" podUID="31690c34-bdb3-4112-99c6-6f95b657b863" containerName="maas-api" containerID="cri-o://b11d34f2b46e34c343c4c3bd10530ca4f9bee9d8e75d024f92492e3585d4ac6b" gracePeriod=30 Jul 10 00:53:52.763460 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:52.763417 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4" podUID="31690c34-bdb3-4112-99c6-6f95b657b863" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.48:8443/health\": dial tcp 10.133.0.48:8443: connect: connection refused" Jul 10 00:53:53.018400 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:53.018374 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4" Jul 10 00:53:53.072591 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:53.072555 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dp9kd\" (UniqueName: \"kubernetes.io/projected/31690c34-bdb3-4112-99c6-6f95b657b863-kube-api-access-dp9kd\") pod \"31690c34-bdb3-4112-99c6-6f95b657b863\" (UID: \"31690c34-bdb3-4112-99c6-6f95b657b863\") " Jul 10 00:53:53.072750 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:53.072601 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31690c34-bdb3-4112-99c6-6f95b657b863-maas-api-tls\") pod \"31690c34-bdb3-4112-99c6-6f95b657b863\" (UID: \"31690c34-bdb3-4112-99c6-6f95b657b863\") " Jul 10 00:53:53.072750 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:53.072658 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31690c34-bdb3-4112-99c6-6f95b657b863-openshift-service-ca\") pod \"31690c34-bdb3-4112-99c6-6f95b657b863\" (UID: \"31690c34-bdb3-4112-99c6-6f95b657b863\") " Jul 10 00:53:53.073055 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:53.073029 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31690c34-bdb3-4112-99c6-6f95b657b863-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "31690c34-bdb3-4112-99c6-6f95b657b863" (UID: "31690c34-bdb3-4112-99c6-6f95b657b863"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:53:53.074656 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:53.074633 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31690c34-bdb3-4112-99c6-6f95b657b863-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "31690c34-bdb3-4112-99c6-6f95b657b863" (UID: "31690c34-bdb3-4112-99c6-6f95b657b863"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:53:53.074752 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:53.074641 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31690c34-bdb3-4112-99c6-6f95b657b863-kube-api-access-dp9kd" (OuterVolumeSpecName: "kube-api-access-dp9kd") pod "31690c34-bdb3-4112-99c6-6f95b657b863" (UID: "31690c34-bdb3-4112-99c6-6f95b657b863"). InnerVolumeSpecName "kube-api-access-dp9kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:53:53.170829 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:53.170759 2559 generic.go:356] "Generic (PLEG): container finished" podID="31690c34-bdb3-4112-99c6-6f95b657b863" containerID="b11d34f2b46e34c343c4c3bd10530ca4f9bee9d8e75d024f92492e3585d4ac6b" exitCode=0 Jul 10 00:53:53.170829 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:53.170810 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4" Jul 10 00:53:53.170829 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:53.170821 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4" event={"ID":"31690c34-bdb3-4112-99c6-6f95b657b863","Type":"ContainerDied","Data":"b11d34f2b46e34c343c4c3bd10530ca4f9bee9d8e75d024f92492e3585d4ac6b"} Jul 10 00:53:53.171059 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:53.170875 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4" event={"ID":"31690c34-bdb3-4112-99c6-6f95b657b863","Type":"ContainerDied","Data":"66f586d5ec9b9513033bed198196a977535c72c442cd81785ac9e06e6a64ac9f"} Jul 10 00:53:53.171059 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:53.170906 2559 scope.go:117] "RemoveContainer" containerID="b11d34f2b46e34c343c4c3bd10530ca4f9bee9d8e75d024f92492e3585d4ac6b" Jul 10 00:53:53.173788 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:53.173760 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31690c34-bdb3-4112-99c6-6f95b657b863-maas-api-tls\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:53:53.173788 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:53.173785 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31690c34-bdb3-4112-99c6-6f95b657b863-openshift-service-ca\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:53:53.173914 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:53.173797 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp9kd\" (UniqueName: \"kubernetes.io/projected/31690c34-bdb3-4112-99c6-6f95b657b863-kube-api-access-dp9kd\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:53:53.180665 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:53.180650 2559 scope.go:117] "RemoveContainer" containerID="b11d34f2b46e34c343c4c3bd10530ca4f9bee9d8e75d024f92492e3585d4ac6b" Jul 10 00:53:53.180904 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:53:53.180885 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b11d34f2b46e34c343c4c3bd10530ca4f9bee9d8e75d024f92492e3585d4ac6b\": container with ID starting with b11d34f2b46e34c343c4c3bd10530ca4f9bee9d8e75d024f92492e3585d4ac6b not found: ID does not exist" containerID="b11d34f2b46e34c343c4c3bd10530ca4f9bee9d8e75d024f92492e3585d4ac6b" Jul 10 00:53:53.180970 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:53.180911 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b11d34f2b46e34c343c4c3bd10530ca4f9bee9d8e75d024f92492e3585d4ac6b"} err="failed to get container status \"b11d34f2b46e34c343c4c3bd10530ca4f9bee9d8e75d024f92492e3585d4ac6b\": rpc error: code = NotFound desc = could not find container \"b11d34f2b46e34c343c4c3bd10530ca4f9bee9d8e75d024f92492e3585d4ac6b\": container with ID starting with b11d34f2b46e34c343c4c3bd10530ca4f9bee9d8e75d024f92492e3585d4ac6b not found: ID does not exist" Jul 10 00:53:53.191827 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:53.191803 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4"] Jul 10 00:53:53.195161 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:53.195139 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3ea0d8-5c8f7b8bcc-28gl4"] Jul 10 00:53:53.649728 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:53:53.649696 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31690c34-bdb3-4112-99c6-6f95b657b863" path="/var/lib/kubelet/pods/31690c34-bdb3-4112-99c6-6f95b657b863/volumes" Jul 10 00:54:06.706359 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:06.706297 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" podUID="a7229f20-9307-4839-ae04-e8733b3bd9f3" containerName="main" containerID="cri-o://696426e1f13b370715a2ec234a40fb720f87ac7cb2c7aab93247be20653765ac" gracePeriod=45 Jul 10 00:54:06.953467 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:06.953440 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" Jul 10 00:54:07.103160 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.103129 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7229f20-9307-4839-ae04-e8733b3bd9f3-tmp-dir\") pod \"a7229f20-9307-4839-ae04-e8733b3bd9f3\" (UID: \"a7229f20-9307-4839-ae04-e8733b3bd9f3\") " Jul 10 00:54:07.103319 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.103233 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7229f20-9307-4839-ae04-e8733b3bd9f3-home\") pod \"a7229f20-9307-4839-ae04-e8733b3bd9f3\" (UID: \"a7229f20-9307-4839-ae04-e8733b3bd9f3\") " Jul 10 00:54:07.103319 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.103265 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7229f20-9307-4839-ae04-e8733b3bd9f3-dshm\") pod \"a7229f20-9307-4839-ae04-e8733b3bd9f3\" (UID: \"a7229f20-9307-4839-ae04-e8733b3bd9f3\") " Jul 10 00:54:07.103319 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.103301 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bmq8\" (UniqueName: \"kubernetes.io/projected/a7229f20-9307-4839-ae04-e8733b3bd9f3-kube-api-access-5bmq8\") pod \"a7229f20-9307-4839-ae04-e8733b3bd9f3\" (UID: \"a7229f20-9307-4839-ae04-e8733b3bd9f3\") " Jul 10 00:54:07.103478 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.103329 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7229f20-9307-4839-ae04-e8733b3bd9f3-tls-certs\") pod \"a7229f20-9307-4839-ae04-e8733b3bd9f3\" (UID: \"a7229f20-9307-4839-ae04-e8733b3bd9f3\") " Jul 10 00:54:07.103478 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.103358 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7229f20-9307-4839-ae04-e8733b3bd9f3-model-cache\") pod \"a7229f20-9307-4839-ae04-e8733b3bd9f3\" (UID: \"a7229f20-9307-4839-ae04-e8733b3bd9f3\") " Jul 10 00:54:07.103478 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.103411 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7229f20-9307-4839-ae04-e8733b3bd9f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a7229f20-9307-4839-ae04-e8733b3bd9f3" (UID: "a7229f20-9307-4839-ae04-e8733b3bd9f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:54:07.103648 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.103517 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7229f20-9307-4839-ae04-e8733b3bd9f3-home" (OuterVolumeSpecName: "home") pod "a7229f20-9307-4839-ae04-e8733b3bd9f3" (UID: "a7229f20-9307-4839-ae04-e8733b3bd9f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:54:07.103648 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.103641 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7229f20-9307-4839-ae04-e8733b3bd9f3-home\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:54:07.103778 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.103659 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7229f20-9307-4839-ae04-e8733b3bd9f3-tmp-dir\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:54:07.103778 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.103716 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7229f20-9307-4839-ae04-e8733b3bd9f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "a7229f20-9307-4839-ae04-e8733b3bd9f3" (UID: "a7229f20-9307-4839-ae04-e8733b3bd9f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:54:07.105509 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.105469 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7229f20-9307-4839-ae04-e8733b3bd9f3-kube-api-access-5bmq8" (OuterVolumeSpecName: "kube-api-access-5bmq8") pod "a7229f20-9307-4839-ae04-e8733b3bd9f3" (UID: "a7229f20-9307-4839-ae04-e8733b3bd9f3"). InnerVolumeSpecName "kube-api-access-5bmq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:54:07.105509 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.105503 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7229f20-9307-4839-ae04-e8733b3bd9f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a7229f20-9307-4839-ae04-e8733b3bd9f3" (UID: "a7229f20-9307-4839-ae04-e8733b3bd9f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:54:07.105659 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.105532 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7229f20-9307-4839-ae04-e8733b3bd9f3-dshm" (OuterVolumeSpecName: "dshm") pod "a7229f20-9307-4839-ae04-e8733b3bd9f3" (UID: "a7229f20-9307-4839-ae04-e8733b3bd9f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:54:07.204069 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.204043 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7229f20-9307-4839-ae04-e8733b3bd9f3-dshm\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:54:07.204069 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.204067 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bmq8\" (UniqueName: \"kubernetes.io/projected/a7229f20-9307-4839-ae04-e8733b3bd9f3-kube-api-access-5bmq8\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:54:07.204253 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.204078 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7229f20-9307-4839-ae04-e8733b3bd9f3-tls-certs\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:54:07.204253 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.204087 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7229f20-9307-4839-ae04-e8733b3bd9f3-model-cache\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:54:07.239549 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.239521 2559 generic.go:356] "Generic (PLEG): container finished" podID="a7229f20-9307-4839-ae04-e8733b3bd9f3" containerID="696426e1f13b370715a2ec234a40fb720f87ac7cb2c7aab93247be20653765ac" exitCode=0 Jul 10 00:54:07.239700 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.239557 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" event={"ID":"a7229f20-9307-4839-ae04-e8733b3bd9f3","Type":"ContainerDied","Data":"696426e1f13b370715a2ec234a40fb720f87ac7cb2c7aab93247be20653765ac"} Jul 10 00:54:07.239700 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.239559 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" Jul 10 00:54:07.239700 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.239595 2559 scope.go:117] "RemoveContainer" containerID="696426e1f13b370715a2ec234a40fb720f87ac7cb2c7aab93247be20653765ac" Jul 10 00:54:07.239700 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.239585 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz" event={"ID":"a7229f20-9307-4839-ae04-e8733b3bd9f3","Type":"ContainerDied","Data":"b560ebac4a16138be10b5037895efc227320115234ff2d34d414b6e22ed22b4a"} Jul 10 00:54:07.248466 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.248450 2559 scope.go:117] "RemoveContainer" containerID="696426e1f13b370715a2ec234a40fb720f87ac7cb2c7aab93247be20653765ac" Jul 10 00:54:07.248701 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:54:07.248684 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"696426e1f13b370715a2ec234a40fb720f87ac7cb2c7aab93247be20653765ac\": container with ID starting with 696426e1f13b370715a2ec234a40fb720f87ac7cb2c7aab93247be20653765ac not found: ID does not exist" containerID="696426e1f13b370715a2ec234a40fb720f87ac7cb2c7aab93247be20653765ac" Jul 10 00:54:07.248747 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.248709 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"696426e1f13b370715a2ec234a40fb720f87ac7cb2c7aab93247be20653765ac"} err="failed to get container status \"696426e1f13b370715a2ec234a40fb720f87ac7cb2c7aab93247be20653765ac\": rpc error: code = NotFound desc = could not find container \"696426e1f13b370715a2ec234a40fb720f87ac7cb2c7aab93247be20653765ac\": container with ID starting with 696426e1f13b370715a2ec234a40fb720f87ac7cb2c7aab93247be20653765ac not found: ID does not exist" Jul 10 00:54:07.259940 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.259917 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz"] Jul 10 00:54:07.263505 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.263480 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-50269d/test-model-50269d-kserve-6c5866f9c-gqlgz"] Jul 10 00:54:07.649799 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:07.649764 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7229f20-9307-4839-ae04-e8733b3bd9f3" path="/var/lib/kubelet/pods/a7229f20-9307-4839-ae04-e8733b3bd9f3/volumes" Jul 10 00:54:35.550499 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:35.550460 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-50269d-8645c458c5-t797g"] Jul 10 00:54:35.551038 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:35.550739 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50269d-8645c458c5-t797g" podUID="f1647192-7905-465c-abc5-a429969e8a39" containerName="maas-api" containerID="cri-o://d6a0e587378907767176c2e638caadd0c2054de3a21a06cf21d2a77197009d9c" gracePeriod=30 Jul 10 00:54:35.796283 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:35.796256 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50269d-8645c458c5-t797g" Jul 10 00:54:35.854148 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:35.854017 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2td5f\" (UniqueName: \"kubernetes.io/projected/f1647192-7905-465c-abc5-a429969e8a39-kube-api-access-2td5f\") pod \"f1647192-7905-465c-abc5-a429969e8a39\" (UID: \"f1647192-7905-465c-abc5-a429969e8a39\") " Jul 10 00:54:35.854148 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:35.854142 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1647192-7905-465c-abc5-a429969e8a39-openshift-service-ca\") pod \"f1647192-7905-465c-abc5-a429969e8a39\" (UID: \"f1647192-7905-465c-abc5-a429969e8a39\") " Jul 10 00:54:35.854384 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:35.854220 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1647192-7905-465c-abc5-a429969e8a39-maas-api-tls\") pod \"f1647192-7905-465c-abc5-a429969e8a39\" (UID: \"f1647192-7905-465c-abc5-a429969e8a39\") " Jul 10 00:54:35.854512 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:35.854486 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1647192-7905-465c-abc5-a429969e8a39-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f1647192-7905-465c-abc5-a429969e8a39" (UID: "f1647192-7905-465c-abc5-a429969e8a39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:54:35.856190 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:35.856160 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1647192-7905-465c-abc5-a429969e8a39-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f1647192-7905-465c-abc5-a429969e8a39" (UID: "f1647192-7905-465c-abc5-a429969e8a39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:54:35.856297 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:35.856232 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1647192-7905-465c-abc5-a429969e8a39-kube-api-access-2td5f" (OuterVolumeSpecName: "kube-api-access-2td5f") pod "f1647192-7905-465c-abc5-a429969e8a39" (UID: "f1647192-7905-465c-abc5-a429969e8a39"). InnerVolumeSpecName "kube-api-access-2td5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:54:35.955260 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:35.955228 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1647192-7905-465c-abc5-a429969e8a39-maas-api-tls\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:54:35.955260 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:35.955255 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2td5f\" (UniqueName: \"kubernetes.io/projected/f1647192-7905-465c-abc5-a429969e8a39-kube-api-access-2td5f\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:54:35.955260 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:35.955265 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1647192-7905-465c-abc5-a429969e8a39-openshift-service-ca\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:54:36.383213 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:36.383181 2559 generic.go:356] "Generic (PLEG): container finished" podID="f1647192-7905-465c-abc5-a429969e8a39" containerID="d6a0e587378907767176c2e638caadd0c2054de3a21a06cf21d2a77197009d9c" exitCode=0 Jul 10 00:54:36.383412 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:36.383220 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50269d-8645c458c5-t797g" event={"ID":"f1647192-7905-465c-abc5-a429969e8a39","Type":"ContainerDied","Data":"d6a0e587378907767176c2e638caadd0c2054de3a21a06cf21d2a77197009d9c"} Jul 10 00:54:36.383412 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:36.383226 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50269d-8645c458c5-t797g" Jul 10 00:54:36.383412 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:36.383262 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-50269d-8645c458c5-t797g" event={"ID":"f1647192-7905-465c-abc5-a429969e8a39","Type":"ContainerDied","Data":"96055e4a28d195f8756c98d89355dacb2f2be14f45d5e4152965d00e2cf8e274"} Jul 10 00:54:36.383412 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:36.383285 2559 scope.go:117] "RemoveContainer" containerID="d6a0e587378907767176c2e638caadd0c2054de3a21a06cf21d2a77197009d9c" Jul 10 00:54:36.398942 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:36.398919 2559 scope.go:117] "RemoveContainer" containerID="d6a0e587378907767176c2e638caadd0c2054de3a21a06cf21d2a77197009d9c" Jul 10 00:54:36.399276 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:54:36.399250 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6a0e587378907767176c2e638caadd0c2054de3a21a06cf21d2a77197009d9c\": container with ID starting with d6a0e587378907767176c2e638caadd0c2054de3a21a06cf21d2a77197009d9c not found: ID does not exist" containerID="d6a0e587378907767176c2e638caadd0c2054de3a21a06cf21d2a77197009d9c" Jul 10 00:54:36.399372 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:36.399286 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6a0e587378907767176c2e638caadd0c2054de3a21a06cf21d2a77197009d9c"} err="failed to get container status \"d6a0e587378907767176c2e638caadd0c2054de3a21a06cf21d2a77197009d9c\": rpc error: code = NotFound desc = could not find container \"d6a0e587378907767176c2e638caadd0c2054de3a21a06cf21d2a77197009d9c\": container with ID starting with d6a0e587378907767176c2e638caadd0c2054de3a21a06cf21d2a77197009d9c not found: ID does not exist" Jul 10 00:54:36.408938 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:36.408914 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-50269d-8645c458c5-t797g"] Jul 10 00:54:36.412169 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:36.412149 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-50269d-8645c458c5-t797g"] Jul 10 00:54:37.649701 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:37.649665 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1647192-7905-465c-abc5-a429969e8a39" path="/var/lib/kubelet/pods/f1647192-7905-465c-abc5-a429969e8a39/volumes" Jul 10 00:54:45.225050 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:45.225007 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv"] Jul 10 00:54:45.225653 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:45.225407 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31690c34-bdb3-4112-99c6-6f95b657b863" containerName="maas-api" Jul 10 00:54:45.225653 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:45.225419 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="31690c34-bdb3-4112-99c6-6f95b657b863" containerName="maas-api" Jul 10 00:54:45.225653 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:45.225428 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1647192-7905-465c-abc5-a429969e8a39" containerName="maas-api" Jul 10 00:54:45.225653 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:45.225434 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1647192-7905-465c-abc5-a429969e8a39" containerName="maas-api" Jul 10 00:54:45.225653 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:45.225450 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7229f20-9307-4839-ae04-e8733b3bd9f3" containerName="main" Jul 10 00:54:45.225653 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:45.225455 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7229f20-9307-4839-ae04-e8733b3bd9f3" containerName="main" Jul 10 00:54:45.225653 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:45.225505 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1647192-7905-465c-abc5-a429969e8a39" containerName="maas-api" Jul 10 00:54:45.225653 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:45.225514 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="31690c34-bdb3-4112-99c6-6f95b657b863" containerName="maas-api" Jul 10 00:54:45.225653 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:45.225520 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7229f20-9307-4839-ae04-e8733b3bd9f3" containerName="main" Jul 10 00:54:45.230305 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:45.230283 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv" Jul 10 00:54:45.232429 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:45.232411 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3f46d3-serving-cert\"" Jul 10 00:54:45.239825 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:45.239802 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv"] Jul 10 00:54:45.338974 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:45.338936 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b87jk\" (UniqueName: \"kubernetes.io/projected/b777e3bd-9915-472e-85d1-25602f154d04-kube-api-access-b87jk\") pod \"maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv\" (UID: \"b777e3bd-9915-472e-85d1-25602f154d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv" Jul 10 00:54:45.339193 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:45.339003 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b777e3bd-9915-472e-85d1-25602f154d04-maas-api-tls\") pod \"maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv\" (UID: \"b777e3bd-9915-472e-85d1-25602f154d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv" Jul 10 00:54:45.339193 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:45.339068 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b777e3bd-9915-472e-85d1-25602f154d04-openshift-service-ca\") pod \"maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv\" (UID: \"b777e3bd-9915-472e-85d1-25602f154d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv" Jul 10 00:54:45.440037 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:45.440004 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b87jk\" (UniqueName: \"kubernetes.io/projected/b777e3bd-9915-472e-85d1-25602f154d04-kube-api-access-b87jk\") pod \"maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv\" (UID: \"b777e3bd-9915-472e-85d1-25602f154d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv" Jul 10 00:54:45.440238 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:45.440070 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b777e3bd-9915-472e-85d1-25602f154d04-maas-api-tls\") pod \"maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv\" (UID: \"b777e3bd-9915-472e-85d1-25602f154d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv" Jul 10 00:54:45.440238 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:45.440111 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b777e3bd-9915-472e-85d1-25602f154d04-openshift-service-ca\") pod \"maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv\" (UID: \"b777e3bd-9915-472e-85d1-25602f154d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv" Jul 10 00:54:45.440238 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:54:45.440228 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-serving-cert: secret "maas-api-e2e-api-a-3f46d3-serving-cert" not found Jul 10 00:54:45.440431 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:54:45.440318 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b777e3bd-9915-472e-85d1-25602f154d04-maas-api-tls podName:b777e3bd-9915-472e-85d1-25602f154d04 nodeName:}" failed. No retries permitted until 2026-07-10 00:54:45.94029644 +0000 UTC m=+2218.873380749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b777e3bd-9915-472e-85d1-25602f154d04-maas-api-tls") pod "maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv" (UID: "b777e3bd-9915-472e-85d1-25602f154d04") : secret "maas-api-e2e-api-a-3f46d3-serving-cert" not found Jul 10 00:54:45.440780 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:45.440762 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b777e3bd-9915-472e-85d1-25602f154d04-openshift-service-ca\") pod \"maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv\" (UID: \"b777e3bd-9915-472e-85d1-25602f154d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv" Jul 10 00:54:45.448158 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:45.448134 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b87jk\" (UniqueName: \"kubernetes.io/projected/b777e3bd-9915-472e-85d1-25602f154d04-kube-api-access-b87jk\") pod \"maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv\" (UID: \"b777e3bd-9915-472e-85d1-25602f154d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv" Jul 10 00:54:45.944958 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:45.944915 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b777e3bd-9915-472e-85d1-25602f154d04-maas-api-tls\") pod \"maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv\" (UID: \"b777e3bd-9915-472e-85d1-25602f154d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv" Jul 10 00:54:45.945187 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:54:45.945062 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-serving-cert: secret "maas-api-e2e-api-a-3f46d3-serving-cert" not found Jul 10 00:54:45.945187 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:54:45.945150 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b777e3bd-9915-472e-85d1-25602f154d04-maas-api-tls podName:b777e3bd-9915-472e-85d1-25602f154d04 nodeName:}" failed. No retries permitted until 2026-07-10 00:54:46.945128984 +0000 UTC m=+2219.878213275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b777e3bd-9915-472e-85d1-25602f154d04-maas-api-tls") pod "maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv" (UID: "b777e3bd-9915-472e-85d1-25602f154d04") : secret "maas-api-e2e-api-a-3f46d3-serving-cert" not found Jul 10 00:54:46.954326 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:46.954297 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b777e3bd-9915-472e-85d1-25602f154d04-maas-api-tls\") pod \"maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv\" (UID: \"b777e3bd-9915-472e-85d1-25602f154d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv" Jul 10 00:54:46.956724 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:46.956653 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b777e3bd-9915-472e-85d1-25602f154d04-maas-api-tls\") pod \"maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv\" (UID: \"b777e3bd-9915-472e-85d1-25602f154d04\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv" Jul 10 00:54:47.042479 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:47.042446 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv" Jul 10 00:54:47.172806 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:47.172784 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv"] Jul 10 00:54:47.174761 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:54:47.174730 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb777e3bd_9915_472e_85d1_25602f154d04.slice/crio-c96a5ad55b46226e75a66c93ece683b23a13e297292fcd4e3664934953393724 WatchSource:0}: Error finding container c96a5ad55b46226e75a66c93ece683b23a13e297292fcd4e3664934953393724: Status 404 returned error can't find the container with id c96a5ad55b46226e75a66c93ece683b23a13e297292fcd4e3664934953393724 Jul 10 00:54:47.437803 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:47.437771 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv" event={"ID":"b777e3bd-9915-472e-85d1-25602f154d04","Type":"ContainerStarted","Data":"c96a5ad55b46226e75a66c93ece683b23a13e297292fcd4e3664934953393724"} Jul 10 00:54:48.450392 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:48.450351 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv" event={"ID":"b777e3bd-9915-472e-85d1-25602f154d04","Type":"ContainerStarted","Data":"2723bd5c5d42b4e6e9faea0478b92805cd999529132f58aaf11c6b22f84434e3"} Jul 10 00:54:48.450823 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:48.450486 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv" Jul 10 00:54:48.469239 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:48.469191 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv" podStartSLOduration=3.0346797 podStartE2EDuration="3.469176604s" totalImagesPullingTime="434.496904ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:54:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:54:48.467846799 +0000 UTC m=+2221.400931132" watchObservedRunningTime="2026-07-10 00:54:48.469176604 +0000 UTC m=+2221.402260912" Jul 10 00:54:53.145788 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:53.145753 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-08071d-658bf8d774-n858b"] Jul 10 00:54:53.149750 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:53.149729 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08071d-658bf8d774-n858b" Jul 10 00:54:53.151753 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:53.151726 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-08071d-serving-cert\"" Jul 10 00:54:53.157682 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:53.157659 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-08071d-658bf8d774-n858b"] Jul 10 00:54:53.311683 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:53.311645 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/755f8085-9009-4bbe-95f4-3b1ee5e77151-maas-api-tls\") pod \"maas-api-e2e-api-b-08071d-658bf8d774-n858b\" (UID: \"755f8085-9009-4bbe-95f4-3b1ee5e77151\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08071d-658bf8d774-n858b" Jul 10 00:54:53.311885 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:53.311711 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jnvn\" (UniqueName: \"kubernetes.io/projected/755f8085-9009-4bbe-95f4-3b1ee5e77151-kube-api-access-2jnvn\") pod \"maas-api-e2e-api-b-08071d-658bf8d774-n858b\" (UID: \"755f8085-9009-4bbe-95f4-3b1ee5e77151\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08071d-658bf8d774-n858b" Jul 10 00:54:53.311885 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:53.311752 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/755f8085-9009-4bbe-95f4-3b1ee5e77151-openshift-service-ca\") pod \"maas-api-e2e-api-b-08071d-658bf8d774-n858b\" (UID: \"755f8085-9009-4bbe-95f4-3b1ee5e77151\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08071d-658bf8d774-n858b" Jul 10 00:54:53.413059 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:53.412967 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/755f8085-9009-4bbe-95f4-3b1ee5e77151-maas-api-tls\") pod \"maas-api-e2e-api-b-08071d-658bf8d774-n858b\" (UID: \"755f8085-9009-4bbe-95f4-3b1ee5e77151\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08071d-658bf8d774-n858b" Jul 10 00:54:53.413059 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:53.413005 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jnvn\" (UniqueName: \"kubernetes.io/projected/755f8085-9009-4bbe-95f4-3b1ee5e77151-kube-api-access-2jnvn\") pod \"maas-api-e2e-api-b-08071d-658bf8d774-n858b\" (UID: \"755f8085-9009-4bbe-95f4-3b1ee5e77151\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08071d-658bf8d774-n858b" Jul 10 00:54:53.413333 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:53.413162 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/755f8085-9009-4bbe-95f4-3b1ee5e77151-openshift-service-ca\") pod \"maas-api-e2e-api-b-08071d-658bf8d774-n858b\" (UID: \"755f8085-9009-4bbe-95f4-3b1ee5e77151\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08071d-658bf8d774-n858b" Jul 10 00:54:53.413777 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:53.413758 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/755f8085-9009-4bbe-95f4-3b1ee5e77151-openshift-service-ca\") pod \"maas-api-e2e-api-b-08071d-658bf8d774-n858b\" (UID: \"755f8085-9009-4bbe-95f4-3b1ee5e77151\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08071d-658bf8d774-n858b" Jul 10 00:54:53.415382 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:53.415360 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/755f8085-9009-4bbe-95f4-3b1ee5e77151-maas-api-tls\") pod \"maas-api-e2e-api-b-08071d-658bf8d774-n858b\" (UID: \"755f8085-9009-4bbe-95f4-3b1ee5e77151\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08071d-658bf8d774-n858b" Jul 10 00:54:53.420306 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:53.420286 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jnvn\" (UniqueName: \"kubernetes.io/projected/755f8085-9009-4bbe-95f4-3b1ee5e77151-kube-api-access-2jnvn\") pod \"maas-api-e2e-api-b-08071d-658bf8d774-n858b\" (UID: \"755f8085-9009-4bbe-95f4-3b1ee5e77151\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08071d-658bf8d774-n858b" Jul 10 00:54:53.463199 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:53.463176 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08071d-658bf8d774-n858b" Jul 10 00:54:53.607675 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:53.607650 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-08071d-658bf8d774-n858b"] Jul 10 00:54:53.609374 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:54:53.609343 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod755f8085_9009_4bbe_95f4_3b1ee5e77151.slice/crio-96fb9abcc826a9d2c071fda4dc7427aec96879ea4d87fb35044f143a834efa80 WatchSource:0}: Error finding container 96fb9abcc826a9d2c071fda4dc7427aec96879ea4d87fb35044f143a834efa80: Status 404 returned error can't find the container with id 96fb9abcc826a9d2c071fda4dc7427aec96879ea4d87fb35044f143a834efa80 Jul 10 00:54:54.460255 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:54.460181 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv" Jul 10 00:54:54.491485 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:54.491454 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08071d-658bf8d774-n858b" event={"ID":"755f8085-9009-4bbe-95f4-3b1ee5e77151","Type":"ContainerStarted","Data":"17291a60bbf825610aa42018a8f158e30946ac1838ebaffe67404d1302ddba12"} Jul 10 00:54:54.491622 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:54.491524 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08071d-658bf8d774-n858b" event={"ID":"755f8085-9009-4bbe-95f4-3b1ee5e77151","Type":"ContainerStarted","Data":"96fb9abcc826a9d2c071fda4dc7427aec96879ea4d87fb35044f143a834efa80"} Jul 10 00:54:54.491622 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:54.491546 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08071d-658bf8d774-n858b" Jul 10 00:54:54.524248 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:54:54.524203 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08071d-658bf8d774-n858b" podStartSLOduration=1.086552566 podStartE2EDuration="1.52418744s" totalImagesPullingTime="437.634874ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:54:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:54:54.521801635 +0000 UTC m=+2227.454885979" watchObservedRunningTime="2026-07-10 00:54:54.52418744 +0000 UTC m=+2227.457271776" Jul 10 00:55:00.501547 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:00.501521 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08071d-658bf8d774-n858b" Jul 10 00:55:06.548750 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:06.548715 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv"] Jul 10 00:55:06.549519 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:06.549483 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv" podUID="b777e3bd-9915-472e-85d1-25602f154d04" containerName="maas-api" containerID="cri-o://2723bd5c5d42b4e6e9faea0478b92805cd999529132f58aaf11c6b22f84434e3" gracePeriod=30 Jul 10 00:55:06.805397 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:06.805340 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv" Jul 10 00:55:06.823181 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:06.823154 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b777e3bd-9915-472e-85d1-25602f154d04-maas-api-tls\") pod \"b777e3bd-9915-472e-85d1-25602f154d04\" (UID: \"b777e3bd-9915-472e-85d1-25602f154d04\") " Jul 10 00:55:06.823316 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:06.823242 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b777e3bd-9915-472e-85d1-25602f154d04-openshift-service-ca\") pod \"b777e3bd-9915-472e-85d1-25602f154d04\" (UID: \"b777e3bd-9915-472e-85d1-25602f154d04\") " Jul 10 00:55:06.823392 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:06.823344 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b87jk\" (UniqueName: \"kubernetes.io/projected/b777e3bd-9915-472e-85d1-25602f154d04-kube-api-access-b87jk\") pod \"b777e3bd-9915-472e-85d1-25602f154d04\" (UID: \"b777e3bd-9915-472e-85d1-25602f154d04\") " Jul 10 00:55:06.823652 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:06.823619 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b777e3bd-9915-472e-85d1-25602f154d04-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b777e3bd-9915-472e-85d1-25602f154d04" (UID: "b777e3bd-9915-472e-85d1-25602f154d04"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:55:06.825422 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:06.825395 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b777e3bd-9915-472e-85d1-25602f154d04-kube-api-access-b87jk" (OuterVolumeSpecName: "kube-api-access-b87jk") pod "b777e3bd-9915-472e-85d1-25602f154d04" (UID: "b777e3bd-9915-472e-85d1-25602f154d04"). InnerVolumeSpecName "kube-api-access-b87jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:55:06.825422 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:06.825405 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b777e3bd-9915-472e-85d1-25602f154d04-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b777e3bd-9915-472e-85d1-25602f154d04" (UID: "b777e3bd-9915-472e-85d1-25602f154d04"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:55:06.924048 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:06.924015 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b87jk\" (UniqueName: \"kubernetes.io/projected/b777e3bd-9915-472e-85d1-25602f154d04-kube-api-access-b87jk\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:55:06.924048 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:06.924045 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b777e3bd-9915-472e-85d1-25602f154d04-maas-api-tls\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:55:06.924265 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:06.924058 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b777e3bd-9915-472e-85d1-25602f154d04-openshift-service-ca\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:55:07.556725 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:07.556693 2559 generic.go:356] "Generic (PLEG): container finished" podID="b777e3bd-9915-472e-85d1-25602f154d04" containerID="2723bd5c5d42b4e6e9faea0478b92805cd999529132f58aaf11c6b22f84434e3" exitCode=0 Jul 10 00:55:07.557157 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:07.556737 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv" Jul 10 00:55:07.557157 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:07.556784 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv" event={"ID":"b777e3bd-9915-472e-85d1-25602f154d04","Type":"ContainerDied","Data":"2723bd5c5d42b4e6e9faea0478b92805cd999529132f58aaf11c6b22f84434e3"} Jul 10 00:55:07.557157 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:07.556831 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv" event={"ID":"b777e3bd-9915-472e-85d1-25602f154d04","Type":"ContainerDied","Data":"c96a5ad55b46226e75a66c93ece683b23a13e297292fcd4e3664934953393724"} Jul 10 00:55:07.557157 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:07.556854 2559 scope.go:117] "RemoveContainer" containerID="2723bd5c5d42b4e6e9faea0478b92805cd999529132f58aaf11c6b22f84434e3" Jul 10 00:55:07.566147 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:07.566106 2559 scope.go:117] "RemoveContainer" containerID="2723bd5c5d42b4e6e9faea0478b92805cd999529132f58aaf11c6b22f84434e3" Jul 10 00:55:07.566386 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:55:07.566367 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2723bd5c5d42b4e6e9faea0478b92805cd999529132f58aaf11c6b22f84434e3\": container with ID starting with 2723bd5c5d42b4e6e9faea0478b92805cd999529132f58aaf11c6b22f84434e3 not found: ID does not exist" containerID="2723bd5c5d42b4e6e9faea0478b92805cd999529132f58aaf11c6b22f84434e3" Jul 10 00:55:07.566432 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:07.566395 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2723bd5c5d42b4e6e9faea0478b92805cd999529132f58aaf11c6b22f84434e3"} err="failed to get container status \"2723bd5c5d42b4e6e9faea0478b92805cd999529132f58aaf11c6b22f84434e3\": rpc error: code = NotFound desc = could not find container \"2723bd5c5d42b4e6e9faea0478b92805cd999529132f58aaf11c6b22f84434e3\": container with ID starting with 2723bd5c5d42b4e6e9faea0478b92805cd999529132f58aaf11c6b22f84434e3 not found: ID does not exist" Jul 10 00:55:07.578127 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:07.578087 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv"] Jul 10 00:55:07.581302 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:07.581282 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3f46d3-6b68c78987-q99nv"] Jul 10 00:55:07.649956 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:07.649935 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b777e3bd-9915-472e-85d1-25602f154d04" path="/var/lib/kubelet/pods/b777e3bd-9915-472e-85d1-25602f154d04/volumes" Jul 10 00:55:13.704698 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:13.704619 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-08071d-658bf8d774-n858b"] Jul 10 00:55:13.705177 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:13.704855 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08071d-658bf8d774-n858b" podUID="755f8085-9009-4bbe-95f4-3b1ee5e77151" containerName="maas-api" containerID="cri-o://17291a60bbf825610aa42018a8f158e30946ac1838ebaffe67404d1302ddba12" gracePeriod=30 Jul 10 00:55:13.949938 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:13.949915 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08071d-658bf8d774-n858b" Jul 10 00:55:13.983624 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:13.983502 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/755f8085-9009-4bbe-95f4-3b1ee5e77151-openshift-service-ca\") pod \"755f8085-9009-4bbe-95f4-3b1ee5e77151\" (UID: \"755f8085-9009-4bbe-95f4-3b1ee5e77151\") " Jul 10 00:55:13.983753 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:13.983641 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jnvn\" (UniqueName: \"kubernetes.io/projected/755f8085-9009-4bbe-95f4-3b1ee5e77151-kube-api-access-2jnvn\") pod \"755f8085-9009-4bbe-95f4-3b1ee5e77151\" (UID: \"755f8085-9009-4bbe-95f4-3b1ee5e77151\") " Jul 10 00:55:13.983753 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:13.983690 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/755f8085-9009-4bbe-95f4-3b1ee5e77151-maas-api-tls\") pod \"755f8085-9009-4bbe-95f4-3b1ee5e77151\" (UID: \"755f8085-9009-4bbe-95f4-3b1ee5e77151\") " Jul 10 00:55:13.984019 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:13.983993 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/755f8085-9009-4bbe-95f4-3b1ee5e77151-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "755f8085-9009-4bbe-95f4-3b1ee5e77151" (UID: "755f8085-9009-4bbe-95f4-3b1ee5e77151"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:55:13.985752 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:13.985726 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/755f8085-9009-4bbe-95f4-3b1ee5e77151-kube-api-access-2jnvn" (OuterVolumeSpecName: "kube-api-access-2jnvn") pod "755f8085-9009-4bbe-95f4-3b1ee5e77151" (UID: "755f8085-9009-4bbe-95f4-3b1ee5e77151"). InnerVolumeSpecName "kube-api-access-2jnvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:55:13.985843 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:13.985828 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/755f8085-9009-4bbe-95f4-3b1ee5e77151-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "755f8085-9009-4bbe-95f4-3b1ee5e77151" (UID: "755f8085-9009-4bbe-95f4-3b1ee5e77151"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:55:14.084727 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:14.084696 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/755f8085-9009-4bbe-95f4-3b1ee5e77151-openshift-service-ca\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:55:14.084727 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:14.084723 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jnvn\" (UniqueName: \"kubernetes.io/projected/755f8085-9009-4bbe-95f4-3b1ee5e77151-kube-api-access-2jnvn\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:55:14.084872 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:14.084737 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/755f8085-9009-4bbe-95f4-3b1ee5e77151-maas-api-tls\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:55:14.593415 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:14.593376 2559 generic.go:356] "Generic (PLEG): container finished" podID="755f8085-9009-4bbe-95f4-3b1ee5e77151" containerID="17291a60bbf825610aa42018a8f158e30946ac1838ebaffe67404d1302ddba12" exitCode=0 Jul 10 00:55:14.593589 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:14.593434 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08071d-658bf8d774-n858b" Jul 10 00:55:14.593589 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:14.593461 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08071d-658bf8d774-n858b" event={"ID":"755f8085-9009-4bbe-95f4-3b1ee5e77151","Type":"ContainerDied","Data":"17291a60bbf825610aa42018a8f158e30946ac1838ebaffe67404d1302ddba12"} Jul 10 00:55:14.593589 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:14.593549 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-08071d-658bf8d774-n858b" event={"ID":"755f8085-9009-4bbe-95f4-3b1ee5e77151","Type":"ContainerDied","Data":"96fb9abcc826a9d2c071fda4dc7427aec96879ea4d87fb35044f143a834efa80"} Jul 10 00:55:14.593589 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:14.593579 2559 scope.go:117] "RemoveContainer" containerID="17291a60bbf825610aa42018a8f158e30946ac1838ebaffe67404d1302ddba12" Jul 10 00:55:14.602958 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:14.602939 2559 scope.go:117] "RemoveContainer" containerID="17291a60bbf825610aa42018a8f158e30946ac1838ebaffe67404d1302ddba12" Jul 10 00:55:14.603234 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:55:14.603215 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17291a60bbf825610aa42018a8f158e30946ac1838ebaffe67404d1302ddba12\": container with ID starting with 17291a60bbf825610aa42018a8f158e30946ac1838ebaffe67404d1302ddba12 not found: ID does not exist" containerID="17291a60bbf825610aa42018a8f158e30946ac1838ebaffe67404d1302ddba12" Jul 10 00:55:14.603313 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:14.603242 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17291a60bbf825610aa42018a8f158e30946ac1838ebaffe67404d1302ddba12"} err="failed to get container status \"17291a60bbf825610aa42018a8f158e30946ac1838ebaffe67404d1302ddba12\": rpc error: code = NotFound desc = could not find container \"17291a60bbf825610aa42018a8f158e30946ac1838ebaffe67404d1302ddba12\": container with ID starting with 17291a60bbf825610aa42018a8f158e30946ac1838ebaffe67404d1302ddba12 not found: ID does not exist" Jul 10 00:55:14.614023 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:14.614002 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-08071d-658bf8d774-n858b"] Jul 10 00:55:14.617839 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:14.617820 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-08071d-658bf8d774-n858b"] Jul 10 00:55:15.649374 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:15.649341 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="755f8085-9009-4bbe-95f4-3b1ee5e77151" path="/var/lib/kubelet/pods/755f8085-9009-4bbe-95f4-3b1ee5e77151/volumes" Jul 10 00:55:23.339584 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:23.339535 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv"] Jul 10 00:55:23.340109 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:23.340091 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="755f8085-9009-4bbe-95f4-3b1ee5e77151" containerName="maas-api" Jul 10 00:55:23.340188 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:23.340130 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="755f8085-9009-4bbe-95f4-3b1ee5e77151" containerName="maas-api" Jul 10 00:55:23.340188 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:23.340148 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b777e3bd-9915-472e-85d1-25602f154d04" containerName="maas-api" Jul 10 00:55:23.340188 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:23.340157 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b777e3bd-9915-472e-85d1-25602f154d04" containerName="maas-api" Jul 10 00:55:23.340290 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:23.340275 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b777e3bd-9915-472e-85d1-25602f154d04" containerName="maas-api" Jul 10 00:55:23.340323 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:23.340290 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="755f8085-9009-4bbe-95f4-3b1ee5e77151" containerName="maas-api" Jul 10 00:55:23.343558 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:23.343538 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv" Jul 10 00:55:23.345699 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:23.345676 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-df95cf-serving-cert\"" Jul 10 00:55:23.351362 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:23.351297 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv"] Jul 10 00:55:23.463765 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:23.463728 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd8qz\" (UniqueName: \"kubernetes.io/projected/2044591c-baf1-40bb-b5e6-e3fd08f356ea-kube-api-access-wd8qz\") pod \"maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv\" (UID: \"2044591c-baf1-40bb-b5e6-e3fd08f356ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv" Jul 10 00:55:23.463925 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:23.463781 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2044591c-baf1-40bb-b5e6-e3fd08f356ea-openshift-service-ca\") pod \"maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv\" (UID: \"2044591c-baf1-40bb-b5e6-e3fd08f356ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv" Jul 10 00:55:23.463925 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:23.463881 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2044591c-baf1-40bb-b5e6-e3fd08f356ea-maas-api-tls\") pod \"maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv\" (UID: \"2044591c-baf1-40bb-b5e6-e3fd08f356ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv" Jul 10 00:55:23.564641 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:23.564604 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2044591c-baf1-40bb-b5e6-e3fd08f356ea-maas-api-tls\") pod \"maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv\" (UID: \"2044591c-baf1-40bb-b5e6-e3fd08f356ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv" Jul 10 00:55:23.564809 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:23.564662 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd8qz\" (UniqueName: \"kubernetes.io/projected/2044591c-baf1-40bb-b5e6-e3fd08f356ea-kube-api-access-wd8qz\") pod \"maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv\" (UID: \"2044591c-baf1-40bb-b5e6-e3fd08f356ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv" Jul 10 00:55:23.564809 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:23.564705 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2044591c-baf1-40bb-b5e6-e3fd08f356ea-openshift-service-ca\") pod \"maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv\" (UID: \"2044591c-baf1-40bb-b5e6-e3fd08f356ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv" Jul 10 00:55:23.565355 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:23.565337 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2044591c-baf1-40bb-b5e6-e3fd08f356ea-openshift-service-ca\") pod \"maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv\" (UID: \"2044591c-baf1-40bb-b5e6-e3fd08f356ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv" Jul 10 00:55:23.566976 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:23.566952 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2044591c-baf1-40bb-b5e6-e3fd08f356ea-maas-api-tls\") pod \"maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv\" (UID: \"2044591c-baf1-40bb-b5e6-e3fd08f356ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv" Jul 10 00:55:23.571557 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:23.571532 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd8qz\" (UniqueName: \"kubernetes.io/projected/2044591c-baf1-40bb-b5e6-e3fd08f356ea-kube-api-access-wd8qz\") pod \"maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv\" (UID: \"2044591c-baf1-40bb-b5e6-e3fd08f356ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv" Jul 10 00:55:23.657924 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:23.657857 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv" Jul 10 00:55:23.789249 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:23.789224 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv"] Jul 10 00:55:23.791169 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:55:23.791141 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2044591c_baf1_40bb_b5e6_e3fd08f356ea.slice/crio-98e4b0f4390a69dea867869c83ca243db39dc89b42e63fd53afe5952fb476d56 WatchSource:0}: Error finding container 98e4b0f4390a69dea867869c83ca243db39dc89b42e63fd53afe5952fb476d56: Status 404 returned error can't find the container with id 98e4b0f4390a69dea867869c83ca243db39dc89b42e63fd53afe5952fb476d56 Jul 10 00:55:24.643857 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:24.643822 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv" event={"ID":"2044591c-baf1-40bb-b5e6-e3fd08f356ea","Type":"ContainerStarted","Data":"9291dd5bf3fc44b6a0f0cc87b894973529b0fdb9cf6b6d7f696b2afbb52038f6"} Jul 10 00:55:24.644333 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:24.643881 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv" event={"ID":"2044591c-baf1-40bb-b5e6-e3fd08f356ea","Type":"ContainerStarted","Data":"98e4b0f4390a69dea867869c83ca243db39dc89b42e63fd53afe5952fb476d56"} Jul 10 00:55:24.644333 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:24.644019 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv" Jul 10 00:55:24.658966 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:24.658924 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv" podStartSLOduration=1.360513626 podStartE2EDuration="1.658911965s" totalImagesPullingTime="298.398339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:55:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:55:24.658377043 +0000 UTC m=+2257.591461365" watchObservedRunningTime="2026-07-10 00:55:24.658911965 +0000 UTC m=+2257.591996273" Jul 10 00:55:30.653703 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:30.653674 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv" Jul 10 00:55:31.523847 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:31.523811 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw"] Jul 10 00:55:31.528099 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:31.528073 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw" Jul 10 00:55:31.530097 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:31.530074 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-82df1b-serving-cert\"" Jul 10 00:55:31.537065 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:31.537042 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw"] Jul 10 00:55:31.630157 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:31.630110 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4539a33-b156-4604-acc7-6dfb7f51c769-maas-api-tls\") pod \"maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw\" (UID: \"d4539a33-b156-4604-acc7-6dfb7f51c769\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw" Jul 10 00:55:31.630342 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:31.630233 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4539a33-b156-4604-acc7-6dfb7f51c769-openshift-service-ca\") pod \"maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw\" (UID: \"d4539a33-b156-4604-acc7-6dfb7f51c769\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw" Jul 10 00:55:31.630342 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:31.630302 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjmhl\" (UniqueName: \"kubernetes.io/projected/d4539a33-b156-4604-acc7-6dfb7f51c769-kube-api-access-jjmhl\") pod \"maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw\" (UID: \"d4539a33-b156-4604-acc7-6dfb7f51c769\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw" Jul 10 00:55:31.731073 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:31.731041 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4539a33-b156-4604-acc7-6dfb7f51c769-openshift-service-ca\") pod \"maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw\" (UID: \"d4539a33-b156-4604-acc7-6dfb7f51c769\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw" Jul 10 00:55:31.731522 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:31.731096 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjmhl\" (UniqueName: \"kubernetes.io/projected/d4539a33-b156-4604-acc7-6dfb7f51c769-kube-api-access-jjmhl\") pod \"maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw\" (UID: \"d4539a33-b156-4604-acc7-6dfb7f51c769\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw" Jul 10 00:55:31.731522 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:31.731162 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4539a33-b156-4604-acc7-6dfb7f51c769-maas-api-tls\") pod \"maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw\" (UID: \"d4539a33-b156-4604-acc7-6dfb7f51c769\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw" Jul 10 00:55:31.731522 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:55:31.731279 2559 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-serving-cert: secret "maas-api-e2e-shared-b-82df1b-serving-cert" not found Jul 10 00:55:31.731522 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:55:31.731341 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4539a33-b156-4604-acc7-6dfb7f51c769-maas-api-tls podName:d4539a33-b156-4604-acc7-6dfb7f51c769 nodeName:}" failed. No retries permitted until 2026-07-10 00:55:32.231322109 +0000 UTC m=+2265.164406396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d4539a33-b156-4604-acc7-6dfb7f51c769-maas-api-tls") pod "maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw" (UID: "d4539a33-b156-4604-acc7-6dfb7f51c769") : secret "maas-api-e2e-shared-b-82df1b-serving-cert" not found Jul 10 00:55:31.731762 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:31.731737 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4539a33-b156-4604-acc7-6dfb7f51c769-openshift-service-ca\") pod \"maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw\" (UID: \"d4539a33-b156-4604-acc7-6dfb7f51c769\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw" Jul 10 00:55:31.739258 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:31.739233 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjmhl\" (UniqueName: \"kubernetes.io/projected/d4539a33-b156-4604-acc7-6dfb7f51c769-kube-api-access-jjmhl\") pod \"maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw\" (UID: \"d4539a33-b156-4604-acc7-6dfb7f51c769\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw" Jul 10 00:55:32.236087 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:32.236049 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4539a33-b156-4604-acc7-6dfb7f51c769-maas-api-tls\") pod \"maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw\" (UID: \"d4539a33-b156-4604-acc7-6dfb7f51c769\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw" Jul 10 00:55:32.238525 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:32.238497 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4539a33-b156-4604-acc7-6dfb7f51c769-maas-api-tls\") pod \"maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw\" (UID: \"d4539a33-b156-4604-acc7-6dfb7f51c769\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw" Jul 10 00:55:32.440063 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:32.440027 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw" Jul 10 00:55:32.578466 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:32.578437 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw"] Jul 10 00:55:32.580467 ip-10-0-132-30 kubenswrapper[2559]: W0710 00:55:32.580438 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4539a33_b156_4604_acc7_6dfb7f51c769.slice/crio-bd4ff667ac419c61b9a357945b283c8614cb3ad20b960aca5e0f41a33ffbef93 WatchSource:0}: Error finding container bd4ff667ac419c61b9a357945b283c8614cb3ad20b960aca5e0f41a33ffbef93: Status 404 returned error can't find the container with id bd4ff667ac419c61b9a357945b283c8614cb3ad20b960aca5e0f41a33ffbef93 Jul 10 00:55:32.681249 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:32.681212 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw" event={"ID":"d4539a33-b156-4604-acc7-6dfb7f51c769","Type":"ContainerStarted","Data":"bd4ff667ac419c61b9a357945b283c8614cb3ad20b960aca5e0f41a33ffbef93"} Jul 10 00:55:33.687363 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:33.687331 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw" event={"ID":"d4539a33-b156-4604-acc7-6dfb7f51c769","Type":"ContainerStarted","Data":"cda611551b872b1c13f1c74e02e576c7bd04886371a27a279deeefcedbaef55f"} Jul 10 00:55:33.687815 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:33.687455 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw" Jul 10 00:55:33.704860 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:33.704813 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw" podStartSLOduration=2.359471015 podStartE2EDuration="2.704797864s" totalImagesPullingTime="345.326849ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:55:33.703529707 +0000 UTC m=+2266.636614017" watchObservedRunningTime="2026-07-10 00:55:33.704797864 +0000 UTC m=+2266.637882172" Jul 10 00:55:39.697674 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:55:39.697644 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw" Jul 10 00:56:22.777108 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:22.777076 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn"] Jul 10 00:56:22.780686 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:22.780661 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" Jul 10 00:56:22.782780 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:22.782747 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-82df1b\"/\"openshift-service-ca.crt\"" Jul 10 00:56:22.783644 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:22.783613 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-82df1b\"/\"auth-test-model-82df1b-kserve-self-signed-certs\"" Jul 10 00:56:22.783760 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:22.783684 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-82df1b\"/\"kube-root-ca.crt\"" Jul 10 00:56:22.783760 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:22.783629 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-82df1b\"/\"default-dockercfg-f42sc\"" Jul 10 00:56:22.790108 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:22.790088 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn"] Jul 10 00:56:22.854686 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:22.854660 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-tmp-dir\") pod \"auth-test-model-82df1b-kserve-8549784d67-hnzfn\" (UID: \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\") " pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" Jul 10 00:56:22.854823 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:22.854706 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-home\") pod \"auth-test-model-82df1b-kserve-8549784d67-hnzfn\" (UID: \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\") " pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" Jul 10 00:56:22.854868 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:22.854839 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-model-cache\") pod \"auth-test-model-82df1b-kserve-8549784d67-hnzfn\" (UID: \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\") " pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" Jul 10 00:56:22.854904 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:22.854884 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqlpc\" (UniqueName: \"kubernetes.io/projected/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-kube-api-access-kqlpc\") pod \"auth-test-model-82df1b-kserve-8549784d67-hnzfn\" (UID: \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\") " pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" Jul 10 00:56:22.854946 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:22.854906 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-dshm\") pod \"auth-test-model-82df1b-kserve-8549784d67-hnzfn\" (UID: \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\") " pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" Jul 10 00:56:22.854992 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:22.854966 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-tls-certs\") pod \"auth-test-model-82df1b-kserve-8549784d67-hnzfn\" (UID: \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\") " pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" Jul 10 00:56:22.955971 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:22.955943 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-model-cache\") pod \"auth-test-model-82df1b-kserve-8549784d67-hnzfn\" (UID: \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\") " pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" Jul 10 00:56:22.956154 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:22.955988 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqlpc\" (UniqueName: \"kubernetes.io/projected/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-kube-api-access-kqlpc\") pod \"auth-test-model-82df1b-kserve-8549784d67-hnzfn\" (UID: \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\") " pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" Jul 10 00:56:22.956154 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:22.956014 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-dshm\") pod \"auth-test-model-82df1b-kserve-8549784d67-hnzfn\" (UID: \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\") " pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" Jul 10 00:56:22.956154 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:22.956041 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-tls-certs\") pod \"auth-test-model-82df1b-kserve-8549784d67-hnzfn\" (UID: \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\") " pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" Jul 10 00:56:22.956154 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:22.956065 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-tmp-dir\") pod \"auth-test-model-82df1b-kserve-8549784d67-hnzfn\" (UID: \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\") " pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" Jul 10 00:56:22.956154 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:22.956095 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-home\") pod \"auth-test-model-82df1b-kserve-8549784d67-hnzfn\" (UID: \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\") " pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" Jul 10 00:56:22.956473 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:22.956454 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-tmp-dir\") pod \"auth-test-model-82df1b-kserve-8549784d67-hnzfn\" (UID: \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\") " pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" Jul 10 00:56:22.956597 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:22.956579 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-model-cache\") pod \"auth-test-model-82df1b-kserve-8549784d67-hnzfn\" (UID: \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\") " pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" Jul 10 00:56:22.956656 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:22.956601 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-home\") pod \"auth-test-model-82df1b-kserve-8549784d67-hnzfn\" (UID: \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\") " pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" Jul 10 00:56:22.958426 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:22.958407 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-dshm\") pod \"auth-test-model-82df1b-kserve-8549784d67-hnzfn\" (UID: \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\") " pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" Jul 10 00:56:22.958545 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:22.958530 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-tls-certs\") pod \"auth-test-model-82df1b-kserve-8549784d67-hnzfn\" (UID: \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\") " pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" Jul 10 00:56:22.963003 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:22.962981 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqlpc\" (UniqueName: \"kubernetes.io/projected/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-kube-api-access-kqlpc\") pod \"auth-test-model-82df1b-kserve-8549784d67-hnzfn\" (UID: \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\") " pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" Jul 10 00:56:23.091291 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:23.091212 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" Jul 10 00:56:23.221622 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:23.219288 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn"] Jul 10 00:56:23.923594 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:23.923555 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" event={"ID":"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51","Type":"ContainerStarted","Data":"f04af877e5069f28725a87f9ef692e2cf433765c2cb28a0455a71cbb8d2da42e"} Jul 10 00:56:23.924005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:23.923606 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" event={"ID":"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51","Type":"ContainerStarted","Data":"ce892b4bc649028a418bf04aee810f27e79bd865e8e4f828d899bca7085f699b"} Jul 10 00:56:23.941676 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:23.941630 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" podStartSLOduration=1.941614544 podStartE2EDuration="1.941614544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:56:23.941153813 +0000 UTC m=+2316.874238147" watchObservedRunningTime="2026-07-10 00:56:23.941614544 +0000 UTC m=+2316.874698856" Jul 10 00:56:33.091387 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:33.091351 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" Jul 10 00:56:33.091387 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:33.091396 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" Jul 10 00:56:33.103960 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:33.103929 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" Jul 10 00:56:33.983842 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:56:33.983813 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" Jul 10 00:57:47.679321 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:57:47.679292 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7zrr_20198eb1-6b3e-4e30-9adf-d4f13df4ee3b/ovn-acl-logging/0.log" Jul 10 00:57:47.690376 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:57:47.690356 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7zrr_20198eb1-6b3e-4e30-9adf-d4f13df4ee3b/ovn-acl-logging/0.log" Jul 10 00:58:19.860859 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:19.860783 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn"] Jul 10 00:58:34.881420 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:34.881369 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" podUID="d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51" containerName="main" containerID="cri-o://f04af877e5069f28725a87f9ef692e2cf433765c2cb28a0455a71cbb8d2da42e" gracePeriod=45 Jul 10 00:58:35.131930 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.131873 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" Jul 10 00:58:35.223218 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.223189 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-home\") pod \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\" (UID: \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\") " Jul 10 00:58:35.223352 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.223229 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqlpc\" (UniqueName: \"kubernetes.io/projected/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-kube-api-access-kqlpc\") pod \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\" (UID: \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\") " Jul 10 00:58:35.223352 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.223254 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-model-cache\") pod \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\" (UID: \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\") " Jul 10 00:58:35.223352 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.223273 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-tls-certs\") pod \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\" (UID: \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\") " Jul 10 00:58:35.223352 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.223324 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-tmp-dir\") pod \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\" (UID: \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\") " Jul 10 00:58:35.223623 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.223373 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-dshm\") pod \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\" (UID: \"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51\") " Jul 10 00:58:35.223623 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.223489 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-home" (OuterVolumeSpecName: "home") pod "d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51" (UID: "d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:58:35.223623 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.223500 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-model-cache" (OuterVolumeSpecName: "model-cache") pod "d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51" (UID: "d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:58:35.223755 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.223639 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51" (UID: "d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:58:35.223755 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.223673 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-home\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:58:35.223755 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.223687 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-model-cache\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:58:35.225413 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.225382 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-dshm" (OuterVolumeSpecName: "dshm") pod "d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51" (UID: "d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:58:35.225534 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.225480 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51" (UID: "d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:58:35.225534 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.225513 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-kube-api-access-kqlpc" (OuterVolumeSpecName: "kube-api-access-kqlpc") pod "d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51" (UID: "d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51"). InnerVolumeSpecName "kube-api-access-kqlpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:58:35.325086 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.325049 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-tmp-dir\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:58:35.325086 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.325082 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-dshm\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:58:35.325086 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.325091 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqlpc\" (UniqueName: \"kubernetes.io/projected/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-kube-api-access-kqlpc\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:58:35.325298 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.325101 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51-tls-certs\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 00:58:35.549574 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.549541 2559 generic.go:356] "Generic (PLEG): container finished" podID="d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51" containerID="f04af877e5069f28725a87f9ef692e2cf433765c2cb28a0455a71cbb8d2da42e" exitCode=0 Jul 10 00:58:35.549737 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.549588 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" Jul 10 00:58:35.549737 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.549632 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" event={"ID":"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51","Type":"ContainerDied","Data":"f04af877e5069f28725a87f9ef692e2cf433765c2cb28a0455a71cbb8d2da42e"} Jul 10 00:58:35.549737 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.549673 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn" event={"ID":"d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51","Type":"ContainerDied","Data":"ce892b4bc649028a418bf04aee810f27e79bd865e8e4f828d899bca7085f699b"} Jul 10 00:58:35.549737 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.549691 2559 scope.go:117] "RemoveContainer" containerID="f04af877e5069f28725a87f9ef692e2cf433765c2cb28a0455a71cbb8d2da42e" Jul 10 00:58:35.558707 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.558690 2559 scope.go:117] "RemoveContainer" containerID="f04af877e5069f28725a87f9ef692e2cf433765c2cb28a0455a71cbb8d2da42e" Jul 10 00:58:35.558965 ip-10-0-132-30 kubenswrapper[2559]: E0710 00:58:35.558949 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f04af877e5069f28725a87f9ef692e2cf433765c2cb28a0455a71cbb8d2da42e\": container with ID starting with f04af877e5069f28725a87f9ef692e2cf433765c2cb28a0455a71cbb8d2da42e not found: ID does not exist" containerID="f04af877e5069f28725a87f9ef692e2cf433765c2cb28a0455a71cbb8d2da42e" Jul 10 00:58:35.559005 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.558973 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f04af877e5069f28725a87f9ef692e2cf433765c2cb28a0455a71cbb8d2da42e"} err="failed to get container status \"f04af877e5069f28725a87f9ef692e2cf433765c2cb28a0455a71cbb8d2da42e\": rpc error: code = NotFound desc = could not find container \"f04af877e5069f28725a87f9ef692e2cf433765c2cb28a0455a71cbb8d2da42e\": container with ID starting with f04af877e5069f28725a87f9ef692e2cf433765c2cb28a0455a71cbb8d2da42e not found: ID does not exist" Jul 10 00:58:35.572145 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.571056 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn"] Jul 10 00:58:35.578297 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.578271 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-82df1b/auth-test-model-82df1b-kserve-8549784d67-hnzfn"] Jul 10 00:58:35.649600 ip-10-0-132-30 kubenswrapper[2559]: I0710 00:58:35.649574 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51" path="/var/lib/kubelet/pods/d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51/volumes" Jul 10 01:00:58.073409 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.073371 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms"] Jul 10 01:00:58.074302 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.074274 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51" containerName="main" Jul 10 01:00:58.074302 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.074301 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51" containerName="main" Jul 10 01:00:58.074512 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.074423 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0cbc1ed-f844-4d5b-9fc3-bfb51bf49b51" containerName="main" Jul 10 01:00:58.077571 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.077552 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" Jul 10 01:00:58.080575 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.080556 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-82df1b\"/\"rate-test-model-82df1b-kserve-self-signed-certs\"" Jul 10 01:00:58.080668 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.080574 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-82df1b\"/\"default-dockercfg-f42sc\"" Jul 10 01:00:58.080668 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.080602 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-82df1b\"/\"openshift-service-ca.crt\"" Jul 10 01:00:58.080668 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.080641 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-82df1b\"/\"kube-root-ca.crt\"" Jul 10 01:00:58.086821 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.086802 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms"] Jul 10 01:00:58.175379 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.175348 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-dshm\") pod \"rate-test-model-82df1b-kserve-64c8cfb7-46nms\" (UID: \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\") " pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" Jul 10 01:00:58.175550 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.175398 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-tls-certs\") pod \"rate-test-model-82df1b-kserve-64c8cfb7-46nms\" (UID: \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\") " pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" Jul 10 01:00:58.175550 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.175446 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-home\") pod \"rate-test-model-82df1b-kserve-64c8cfb7-46nms\" (UID: \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\") " pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" Jul 10 01:00:58.175550 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.175511 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p24m\" (UniqueName: \"kubernetes.io/projected/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-kube-api-access-2p24m\") pod \"rate-test-model-82df1b-kserve-64c8cfb7-46nms\" (UID: \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\") " pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" Jul 10 01:00:58.175550 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.175542 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-tmp-dir\") pod \"rate-test-model-82df1b-kserve-64c8cfb7-46nms\" (UID: \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\") " pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" Jul 10 01:00:58.175705 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.175612 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-model-cache\") pod \"rate-test-model-82df1b-kserve-64c8cfb7-46nms\" (UID: \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\") " pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" Jul 10 01:00:58.276856 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.276823 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-dshm\") pod \"rate-test-model-82df1b-kserve-64c8cfb7-46nms\" (UID: \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\") " pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" Jul 10 01:00:58.277024 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.276869 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-tls-certs\") pod \"rate-test-model-82df1b-kserve-64c8cfb7-46nms\" (UID: \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\") " pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" Jul 10 01:00:58.277024 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.276890 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-home\") pod \"rate-test-model-82df1b-kserve-64c8cfb7-46nms\" (UID: \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\") " pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" Jul 10 01:00:58.277024 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.276953 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p24m\" (UniqueName: \"kubernetes.io/projected/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-kube-api-access-2p24m\") pod \"rate-test-model-82df1b-kserve-64c8cfb7-46nms\" (UID: \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\") " pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" Jul 10 01:00:58.277024 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.276989 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-tmp-dir\") pod \"rate-test-model-82df1b-kserve-64c8cfb7-46nms\" (UID: \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\") " pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" Jul 10 01:00:58.277292 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.277036 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-model-cache\") pod \"rate-test-model-82df1b-kserve-64c8cfb7-46nms\" (UID: \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\") " pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" Jul 10 01:00:58.277479 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.277421 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-home\") pod \"rate-test-model-82df1b-kserve-64c8cfb7-46nms\" (UID: \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\") " pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" Jul 10 01:00:58.277479 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.277467 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-model-cache\") pod \"rate-test-model-82df1b-kserve-64c8cfb7-46nms\" (UID: \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\") " pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" Jul 10 01:00:58.277634 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.277512 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-tmp-dir\") pod \"rate-test-model-82df1b-kserve-64c8cfb7-46nms\" (UID: \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\") " pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" Jul 10 01:00:58.279446 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.279421 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-dshm\") pod \"rate-test-model-82df1b-kserve-64c8cfb7-46nms\" (UID: \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\") " pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" Jul 10 01:00:58.279591 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.279570 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-tls-certs\") pod \"rate-test-model-82df1b-kserve-64c8cfb7-46nms\" (UID: \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\") " pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" Jul 10 01:00:58.284951 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.284928 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p24m\" (UniqueName: \"kubernetes.io/projected/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-kube-api-access-2p24m\") pod \"rate-test-model-82df1b-kserve-64c8cfb7-46nms\" (UID: \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\") " pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" Jul 10 01:00:58.387781 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.387703 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" Jul 10 01:00:58.518233 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:58.518198 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms"] Jul 10 01:00:59.226948 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:59.226911 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" event={"ID":"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0","Type":"ContainerStarted","Data":"b9258eebe483f38e9436efc670687962256b0acedad3233b04396ad1f40ca87e"} Jul 10 01:00:59.227348 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:59.226956 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" event={"ID":"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0","Type":"ContainerStarted","Data":"f8957f9f028941ec493e6e68473e2313fbe714b335d5697b23d8a96d195c85a4"} Jul 10 01:00:59.242506 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:00:59.242458 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" podStartSLOduration=1.242442818 podStartE2EDuration="1.242442818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 01:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 01:00:59.241329399 +0000 UTC m=+2592.174413709" watchObservedRunningTime="2026-07-10 01:00:59.242442818 +0000 UTC m=+2592.175527126" Jul 10 01:01:08.388286 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:01:08.388244 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" Jul 10 01:01:08.388892 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:01:08.388309 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" Jul 10 01:01:08.401013 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:01:08.400988 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" Jul 10 01:01:09.286142 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:01:09.286045 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" Jul 10 01:01:49.856945 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:01:49.856916 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms"] Jul 10 01:02:04.876697 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:04.876649 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" podUID="1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0" containerName="main" containerID="cri-o://b9258eebe483f38e9436efc670687962256b0acedad3233b04396ad1f40ca87e" gracePeriod=45 Jul 10 01:02:05.122997 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.122972 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" Jul 10 01:02:05.275195 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.275165 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-tmp-dir\") pod \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\" (UID: \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\") " Jul 10 01:02:05.275372 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.275208 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2p24m\" (UniqueName: \"kubernetes.io/projected/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-kube-api-access-2p24m\") pod \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\" (UID: \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\") " Jul 10 01:02:05.275372 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.275240 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-dshm\") pod \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\" (UID: \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\") " Jul 10 01:02:05.275372 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.275254 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-home\") pod \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\" (UID: \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\") " Jul 10 01:02:05.275372 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.275294 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-model-cache\") pod \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\" (UID: \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\") " Jul 10 01:02:05.275372 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.275323 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-tls-certs\") pod \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\" (UID: \"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0\") " Jul 10 01:02:05.275631 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.275510 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0" (UID: "1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:02:05.275631 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.275560 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-home" (OuterVolumeSpecName: "home") pod "1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0" (UID: "1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:02:05.275631 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.275575 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0" (UID: "1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:02:05.275802 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.275679 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-home\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 01:02:05.275802 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.275699 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-model-cache\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 01:02:05.275802 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.275713 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-tmp-dir\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 01:02:05.277422 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.277391 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-dshm" (OuterVolumeSpecName: "dshm") pod "1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0" (UID: "1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:02:05.277422 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.277414 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-kube-api-access-2p24m" (OuterVolumeSpecName: "kube-api-access-2p24m") pod "1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0" (UID: "1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0"). InnerVolumeSpecName "kube-api-access-2p24m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:02:05.277565 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.277459 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0" (UID: "1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 01:02:05.376895 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.376865 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p24m\" (UniqueName: \"kubernetes.io/projected/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-kube-api-access-2p24m\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 01:02:05.376895 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.376891 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-dshm\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 01:02:05.377067 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.376901 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0-tls-certs\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 01:02:05.561077 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.560992 2559 generic.go:356] "Generic (PLEG): container finished" podID="1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0" containerID="b9258eebe483f38e9436efc670687962256b0acedad3233b04396ad1f40ca87e" exitCode=0 Jul 10 01:02:05.561077 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.561033 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" Jul 10 01:02:05.561286 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.561079 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" event={"ID":"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0","Type":"ContainerDied","Data":"b9258eebe483f38e9436efc670687962256b0acedad3233b04396ad1f40ca87e"} Jul 10 01:02:05.561286 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.561139 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms" event={"ID":"1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0","Type":"ContainerDied","Data":"f8957f9f028941ec493e6e68473e2313fbe714b335d5697b23d8a96d195c85a4"} Jul 10 01:02:05.561286 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.561164 2559 scope.go:117] "RemoveContainer" containerID="b9258eebe483f38e9436efc670687962256b0acedad3233b04396ad1f40ca87e" Jul 10 01:02:05.570244 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.570227 2559 scope.go:117] "RemoveContainer" containerID="b9258eebe483f38e9436efc670687962256b0acedad3233b04396ad1f40ca87e" Jul 10 01:02:05.570494 ip-10-0-132-30 kubenswrapper[2559]: E0710 01:02:05.570476 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9258eebe483f38e9436efc670687962256b0acedad3233b04396ad1f40ca87e\": container with ID starting with b9258eebe483f38e9436efc670687962256b0acedad3233b04396ad1f40ca87e not found: ID does not exist" containerID="b9258eebe483f38e9436efc670687962256b0acedad3233b04396ad1f40ca87e" Jul 10 01:02:05.570554 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.570506 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9258eebe483f38e9436efc670687962256b0acedad3233b04396ad1f40ca87e"} err="failed to get container status \"b9258eebe483f38e9436efc670687962256b0acedad3233b04396ad1f40ca87e\": rpc error: code = NotFound desc = could not find container \"b9258eebe483f38e9436efc670687962256b0acedad3233b04396ad1f40ca87e\": container with ID starting with b9258eebe483f38e9436efc670687962256b0acedad3233b04396ad1f40ca87e not found: ID does not exist" Jul 10 01:02:05.580850 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.580827 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms"] Jul 10 01:02:05.584807 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.584786 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-82df1b/rate-test-model-82df1b-kserve-64c8cfb7-46nms"] Jul 10 01:02:05.650527 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:05.650499 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0" path="/var/lib/kubelet/pods/1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0/volumes" Jul 10 01:02:09.967205 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:09.967172 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv"] Jul 10 01:02:09.967722 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:09.967395 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv" podUID="2044591c-baf1-40bb-b5e6-e3fd08f356ea" containerName="maas-api" containerID="cri-o://9291dd5bf3fc44b6a0f0cc87b894973529b0fdb9cf6b6d7f696b2afbb52038f6" gracePeriod=30 Jul 10 01:02:10.415814 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:10.415791 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv" Jul 10 01:02:10.424643 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:10.424612 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2044591c-baf1-40bb-b5e6-e3fd08f356ea-maas-api-tls\") pod \"2044591c-baf1-40bb-b5e6-e3fd08f356ea\" (UID: \"2044591c-baf1-40bb-b5e6-e3fd08f356ea\") " Jul 10 01:02:10.424769 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:10.424697 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2044591c-baf1-40bb-b5e6-e3fd08f356ea-openshift-service-ca\") pod \"2044591c-baf1-40bb-b5e6-e3fd08f356ea\" (UID: \"2044591c-baf1-40bb-b5e6-e3fd08f356ea\") " Jul 10 01:02:10.425085 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:10.425061 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2044591c-baf1-40bb-b5e6-e3fd08f356ea-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2044591c-baf1-40bb-b5e6-e3fd08f356ea" (UID: "2044591c-baf1-40bb-b5e6-e3fd08f356ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 01:02:10.426643 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:10.426620 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2044591c-baf1-40bb-b5e6-e3fd08f356ea-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2044591c-baf1-40bb-b5e6-e3fd08f356ea" (UID: "2044591c-baf1-40bb-b5e6-e3fd08f356ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 01:02:10.525628 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:10.525600 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wd8qz\" (UniqueName: \"kubernetes.io/projected/2044591c-baf1-40bb-b5e6-e3fd08f356ea-kube-api-access-wd8qz\") pod \"2044591c-baf1-40bb-b5e6-e3fd08f356ea\" (UID: \"2044591c-baf1-40bb-b5e6-e3fd08f356ea\") " Jul 10 01:02:10.525972 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:10.525954 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2044591c-baf1-40bb-b5e6-e3fd08f356ea-maas-api-tls\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 01:02:10.526043 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:10.525977 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2044591c-baf1-40bb-b5e6-e3fd08f356ea-openshift-service-ca\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 01:02:10.527617 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:10.527590 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2044591c-baf1-40bb-b5e6-e3fd08f356ea-kube-api-access-wd8qz" (OuterVolumeSpecName: "kube-api-access-wd8qz") pod "2044591c-baf1-40bb-b5e6-e3fd08f356ea" (UID: "2044591c-baf1-40bb-b5e6-e3fd08f356ea"). InnerVolumeSpecName "kube-api-access-wd8qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:02:10.587900 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:10.587869 2559 generic.go:356] "Generic (PLEG): container finished" podID="2044591c-baf1-40bb-b5e6-e3fd08f356ea" containerID="9291dd5bf3fc44b6a0f0cc87b894973529b0fdb9cf6b6d7f696b2afbb52038f6" exitCode=0 Jul 10 01:02:10.588059 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:10.587909 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv" event={"ID":"2044591c-baf1-40bb-b5e6-e3fd08f356ea","Type":"ContainerDied","Data":"9291dd5bf3fc44b6a0f0cc87b894973529b0fdb9cf6b6d7f696b2afbb52038f6"} Jul 10 01:02:10.588059 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:10.587914 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv" Jul 10 01:02:10.588059 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:10.587949 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv" event={"ID":"2044591c-baf1-40bb-b5e6-e3fd08f356ea","Type":"ContainerDied","Data":"98e4b0f4390a69dea867869c83ca243db39dc89b42e63fd53afe5952fb476d56"} Jul 10 01:02:10.588059 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:10.587971 2559 scope.go:117] "RemoveContainer" containerID="9291dd5bf3fc44b6a0f0cc87b894973529b0fdb9cf6b6d7f696b2afbb52038f6" Jul 10 01:02:10.598704 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:10.598680 2559 scope.go:117] "RemoveContainer" containerID="9291dd5bf3fc44b6a0f0cc87b894973529b0fdb9cf6b6d7f696b2afbb52038f6" Jul 10 01:02:10.598963 ip-10-0-132-30 kubenswrapper[2559]: E0710 01:02:10.598940 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9291dd5bf3fc44b6a0f0cc87b894973529b0fdb9cf6b6d7f696b2afbb52038f6\": container with ID starting with 9291dd5bf3fc44b6a0f0cc87b894973529b0fdb9cf6b6d7f696b2afbb52038f6 not found: ID does not exist" containerID="9291dd5bf3fc44b6a0f0cc87b894973529b0fdb9cf6b6d7f696b2afbb52038f6" Jul 10 01:02:10.599018 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:10.598972 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9291dd5bf3fc44b6a0f0cc87b894973529b0fdb9cf6b6d7f696b2afbb52038f6"} err="failed to get container status \"9291dd5bf3fc44b6a0f0cc87b894973529b0fdb9cf6b6d7f696b2afbb52038f6\": rpc error: code = NotFound desc = could not find container \"9291dd5bf3fc44b6a0f0cc87b894973529b0fdb9cf6b6d7f696b2afbb52038f6\": container with ID starting with 9291dd5bf3fc44b6a0f0cc87b894973529b0fdb9cf6b6d7f696b2afbb52038f6 not found: ID does not exist" Jul 10 01:02:10.612496 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:10.612016 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv"] Jul 10 01:02:10.617244 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:10.617220 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-df95cf-578f8ffc99-sbhxv"] Jul 10 01:02:10.626494 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:10.626472 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd8qz\" (UniqueName: \"kubernetes.io/projected/2044591c-baf1-40bb-b5e6-e3fd08f356ea-kube-api-access-wd8qz\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 01:02:11.650391 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:11.650360 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2044591c-baf1-40bb-b5e6-e3fd08f356ea" path="/var/lib/kubelet/pods/2044591c-baf1-40bb-b5e6-e3fd08f356ea/volumes" Jul 10 01:02:16.967895 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:16.967858 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw"] Jul 10 01:02:16.968331 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:16.968108 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw" podUID="d4539a33-b156-4604-acc7-6dfb7f51c769" containerName="maas-api" containerID="cri-o://cda611551b872b1c13f1c74e02e576c7bd04886371a27a279deeefcedbaef55f" gracePeriod=30 Jul 10 01:02:17.416582 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:17.416560 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw" Jul 10 01:02:17.476998 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:17.476964 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4539a33-b156-4604-acc7-6dfb7f51c769-openshift-service-ca\") pod \"d4539a33-b156-4604-acc7-6dfb7f51c769\" (UID: \"d4539a33-b156-4604-acc7-6dfb7f51c769\") " Jul 10 01:02:17.477170 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:17.477016 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4539a33-b156-4604-acc7-6dfb7f51c769-maas-api-tls\") pod \"d4539a33-b156-4604-acc7-6dfb7f51c769\" (UID: \"d4539a33-b156-4604-acc7-6dfb7f51c769\") " Jul 10 01:02:17.477170 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:17.477084 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjmhl\" (UniqueName: \"kubernetes.io/projected/d4539a33-b156-4604-acc7-6dfb7f51c769-kube-api-access-jjmhl\") pod \"d4539a33-b156-4604-acc7-6dfb7f51c769\" (UID: \"d4539a33-b156-4604-acc7-6dfb7f51c769\") " Jul 10 01:02:17.477403 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:17.477380 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4539a33-b156-4604-acc7-6dfb7f51c769-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d4539a33-b156-4604-acc7-6dfb7f51c769" (UID: "d4539a33-b156-4604-acc7-6dfb7f51c769"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 01:02:17.479660 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:17.479619 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4539a33-b156-4604-acc7-6dfb7f51c769-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d4539a33-b156-4604-acc7-6dfb7f51c769" (UID: "d4539a33-b156-4604-acc7-6dfb7f51c769"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 01:02:17.479761 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:17.479628 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4539a33-b156-4604-acc7-6dfb7f51c769-kube-api-access-jjmhl" (OuterVolumeSpecName: "kube-api-access-jjmhl") pod "d4539a33-b156-4604-acc7-6dfb7f51c769" (UID: "d4539a33-b156-4604-acc7-6dfb7f51c769"). InnerVolumeSpecName "kube-api-access-jjmhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:02:17.578643 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:17.578618 2559 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4539a33-b156-4604-acc7-6dfb7f51c769-openshift-service-ca\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 01:02:17.578643 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:17.578644 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4539a33-b156-4604-acc7-6dfb7f51c769-maas-api-tls\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 01:02:17.578643 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:17.578654 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjmhl\" (UniqueName: \"kubernetes.io/projected/d4539a33-b156-4604-acc7-6dfb7f51c769-kube-api-access-jjmhl\") on node \"ip-10-0-132-30.ec2.internal\" DevicePath \"\"" Jul 10 01:02:17.623816 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:17.623786 2559 generic.go:356] "Generic (PLEG): container finished" podID="d4539a33-b156-4604-acc7-6dfb7f51c769" containerID="cda611551b872b1c13f1c74e02e576c7bd04886371a27a279deeefcedbaef55f" exitCode=0 Jul 10 01:02:17.623955 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:17.623829 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw" Jul 10 01:02:17.623955 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:17.623876 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw" event={"ID":"d4539a33-b156-4604-acc7-6dfb7f51c769","Type":"ContainerDied","Data":"cda611551b872b1c13f1c74e02e576c7bd04886371a27a279deeefcedbaef55f"} Jul 10 01:02:17.623955 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:17.623934 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw" event={"ID":"d4539a33-b156-4604-acc7-6dfb7f51c769","Type":"ContainerDied","Data":"bd4ff667ac419c61b9a357945b283c8614cb3ad20b960aca5e0f41a33ffbef93"} Jul 10 01:02:17.624136 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:17.623963 2559 scope.go:117] "RemoveContainer" containerID="cda611551b872b1c13f1c74e02e576c7bd04886371a27a279deeefcedbaef55f" Jul 10 01:02:17.632824 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:17.632793 2559 scope.go:117] "RemoveContainer" containerID="cda611551b872b1c13f1c74e02e576c7bd04886371a27a279deeefcedbaef55f" Jul 10 01:02:17.633081 ip-10-0-132-30 kubenswrapper[2559]: E0710 01:02:17.633063 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cda611551b872b1c13f1c74e02e576c7bd04886371a27a279deeefcedbaef55f\": container with ID starting with cda611551b872b1c13f1c74e02e576c7bd04886371a27a279deeefcedbaef55f not found: ID does not exist" containerID="cda611551b872b1c13f1c74e02e576c7bd04886371a27a279deeefcedbaef55f" Jul 10 01:02:17.633162 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:17.633090 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda611551b872b1c13f1c74e02e576c7bd04886371a27a279deeefcedbaef55f"} err="failed to get container status \"cda611551b872b1c13f1c74e02e576c7bd04886371a27a279deeefcedbaef55f\": rpc error: code = NotFound desc = could not find container \"cda611551b872b1c13f1c74e02e576c7bd04886371a27a279deeefcedbaef55f\": container with ID starting with cda611551b872b1c13f1c74e02e576c7bd04886371a27a279deeefcedbaef55f not found: ID does not exist" Jul 10 01:02:17.649371 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:17.649350 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw"] Jul 10 01:02:17.649475 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:17.649376 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-82df1b-595bfc9566-szdxw"] Jul 10 01:02:19.649850 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:19.649818 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4539a33-b156-4604-acc7-6dfb7f51c769" path="/var/lib/kubelet/pods/d4539a33-b156-4604-acc7-6dfb7f51c769/volumes" Jul 10 01:02:29.047416 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:29.047384 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5fc87c469-xw8qr_4d94fc25-7cec-4ffa-9d26-ae8d3de15848/manager/0.log" Jul 10 01:02:29.710405 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:29.710377 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85d875657f-z2vng_5af46045-6974-437d-83ca-80cb4bcefa05/copy-manifests/0.log" Jul 10 01:02:29.717350 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:29.717326 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85d875657f-z2vng_5af46045-6974-437d-83ca-80cb4bcefa05/manager/0.log" Jul 10 01:02:30.070976 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:30.070943 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6886d9bcc8-g665s_349020b7-43cd-4927-8058-1103e4e6f20a/maas-api/0.log" Jul 10 01:02:30.195467 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:30.195442 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-42kf5_eaffba7c-7ac3-4b24-90f1-4ab01fd79f9c/postgres/0.log" Jul 10 01:02:31.817583 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:31.817552 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-vtg5w_62fd95d9-a931-4eb7-a4db-2cceb02cbd7b/manager/0.log" Jul 10 01:02:32.060637 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:32.060609 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-cljqj_2f80f79a-3264-4b3e-8fa7-a6166013b333/registry-server/0.log" Jul 10 01:02:32.197089 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:32.197007 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-57jts_86b52ed9-417f-456f-a759-b59c75763236/manager/0.log" Jul 10 01:02:32.840953 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:32.840923 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4xnd87_84663b2e-6507-4aac-8ad4-db9fa1621614/istio-proxy/0.log" Jul 10 01:02:33.344817 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:33.344774 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pfqlk_cf310fe5-e414-4b36-97d9-c99cb1da8844/istio-proxy/1.log" Jul 10 01:02:33.470777 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:33.470747 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-56gc5_541871dd-0ab5-41ed-87c7-b688171167b0/payload-pre-processing/0.log" Jul 10 01:02:33.592668 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:33.592627 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-6bgxv_ebc9f72c-30bf-4071-ae1c-928caeb9d033/payload-processing/0.log" Jul 10 01:02:34.196264 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:34.196233 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-r9qfv_1739f22c-f99b-4680-bdc2-f71c855d6c84/main/0.log" Jul 10 01:02:34.437900 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:34.437871 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbqg97p_22823e39-0e4a-45ec-91b5-5cec70045e28/main/0.log" Jul 10 01:02:34.551139 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:34.551095 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-jnbgn_f69258ad-d5af-4af9-bd35-a2a631670d5d/main/0.log" Jul 10 01:02:39.381529 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:39.381454 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6cvbh/must-gather-vxk5g"] Jul 10 01:02:39.381880 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:39.381797 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0" containerName="main" Jul 10 01:02:39.381880 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:39.381807 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0" containerName="main" Jul 10 01:02:39.381880 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:39.381826 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2044591c-baf1-40bb-b5e6-e3fd08f356ea" containerName="maas-api" Jul 10 01:02:39.381880 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:39.381832 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2044591c-baf1-40bb-b5e6-e3fd08f356ea" containerName="maas-api" Jul 10 01:02:39.381880 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:39.381842 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4539a33-b156-4604-acc7-6dfb7f51c769" containerName="maas-api" Jul 10 01:02:39.381880 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:39.381847 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4539a33-b156-4604-acc7-6dfb7f51c769" containerName="maas-api" Jul 10 01:02:39.382073 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:39.381900 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2044591c-baf1-40bb-b5e6-e3fd08f356ea" containerName="maas-api" Jul 10 01:02:39.382073 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:39.381909 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4539a33-b156-4604-acc7-6dfb7f51c769" containerName="maas-api" Jul 10 01:02:39.382073 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:39.381916 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b1e1ec5-d56a-4fe1-b66e-2a217bc6c1d0" containerName="main" Jul 10 01:02:39.384800 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:39.384784 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6cvbh/must-gather-vxk5g" Jul 10 01:02:39.386988 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:39.386967 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6cvbh\"/\"openshift-service-ca.crt\"" Jul 10 01:02:39.387138 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:39.386971 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6cvbh\"/\"default-dockercfg-njqvb\"" Jul 10 01:02:39.387138 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:39.387062 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6cvbh\"/\"kube-root-ca.crt\"" Jul 10 01:02:39.400391 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:39.397397 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6cvbh/must-gather-vxk5g"] Jul 10 01:02:39.470812 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:39.470784 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d339b3d7-1561-46f1-9d24-071f68437af2-must-gather-output\") pod \"must-gather-vxk5g\" (UID: \"d339b3d7-1561-46f1-9d24-071f68437af2\") " pod="openshift-must-gather-6cvbh/must-gather-vxk5g" Jul 10 01:02:39.470930 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:39.470835 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmg9r\" (UniqueName: \"kubernetes.io/projected/d339b3d7-1561-46f1-9d24-071f68437af2-kube-api-access-dmg9r\") pod \"must-gather-vxk5g\" (UID: \"d339b3d7-1561-46f1-9d24-071f68437af2\") " pod="openshift-must-gather-6cvbh/must-gather-vxk5g" Jul 10 01:02:39.571332 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:39.571310 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d339b3d7-1561-46f1-9d24-071f68437af2-must-gather-output\") pod \"must-gather-vxk5g\" (UID: \"d339b3d7-1561-46f1-9d24-071f68437af2\") " pod="openshift-must-gather-6cvbh/must-gather-vxk5g" Jul 10 01:02:39.571436 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:39.571354 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmg9r\" (UniqueName: \"kubernetes.io/projected/d339b3d7-1561-46f1-9d24-071f68437af2-kube-api-access-dmg9r\") pod \"must-gather-vxk5g\" (UID: \"d339b3d7-1561-46f1-9d24-071f68437af2\") " pod="openshift-must-gather-6cvbh/must-gather-vxk5g" Jul 10 01:02:39.571641 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:39.571621 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d339b3d7-1561-46f1-9d24-071f68437af2-must-gather-output\") pod \"must-gather-vxk5g\" (UID: \"d339b3d7-1561-46f1-9d24-071f68437af2\") " pod="openshift-must-gather-6cvbh/must-gather-vxk5g" Jul 10 01:02:39.579301 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:39.579274 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmg9r\" (UniqueName: \"kubernetes.io/projected/d339b3d7-1561-46f1-9d24-071f68437af2-kube-api-access-dmg9r\") pod \"must-gather-vxk5g\" (UID: \"d339b3d7-1561-46f1-9d24-071f68437af2\") " pod="openshift-must-gather-6cvbh/must-gather-vxk5g" Jul 10 01:02:39.695329 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:39.695267 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6cvbh/must-gather-vxk5g" Jul 10 01:02:39.826509 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:39.826478 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6cvbh/must-gather-vxk5g"] Jul 10 01:02:39.829892 ip-10-0-132-30 kubenswrapper[2559]: W0710 01:02:39.829865 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd339b3d7_1561_46f1_9d24_071f68437af2.slice/crio-0ab11cadcd2dde726c9e9ec523c2b21fd32c3eb875f23717b50c133c7758cb33 WatchSource:0}: Error finding container 0ab11cadcd2dde726c9e9ec523c2b21fd32c3eb875f23717b50c133c7758cb33: Status 404 returned error can't find the container with id 0ab11cadcd2dde726c9e9ec523c2b21fd32c3eb875f23717b50c133c7758cb33 Jul 10 01:02:39.832212 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:39.832196 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 01:02:40.737806 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:40.737778 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6cvbh/must-gather-vxk5g" event={"ID":"d339b3d7-1561-46f1-9d24-071f68437af2","Type":"ContainerStarted","Data":"5e26dcce5277054cb86a8559f6639b67da7b14ad887253ac09b8e57dc2353a15"} Jul 10 01:02:40.738197 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:40.737833 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6cvbh/must-gather-vxk5g" event={"ID":"d339b3d7-1561-46f1-9d24-071f68437af2","Type":"ContainerStarted","Data":"64b565cbab9fd78da6e29f65d920a414d0b983ac265bb69280d4e726c992ab48"} Jul 10 01:02:40.738197 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:40.737851 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6cvbh/must-gather-vxk5g" event={"ID":"d339b3d7-1561-46f1-9d24-071f68437af2","Type":"ContainerStarted","Data":"0ab11cadcd2dde726c9e9ec523c2b21fd32c3eb875f23717b50c133c7758cb33"} Jul 10 01:02:40.752157 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:40.752096 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6cvbh/must-gather-vxk5g" podStartSLOduration=1.035290594 podStartE2EDuration="1.752077797s" totalImagesPullingTime="716.787203ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 01:02:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 01:02:40.750628864 +0000 UTC m=+2693.683713172" watchObservedRunningTime="2026-07-10 01:02:40.752077797 +0000 UTC m=+2693.685162107" Jul 10 01:02:42.109539 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:42.109505 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zqzc6_b1c3645d-2300-4b32-b0ef-e373064e5aad/global-pull-secret-syncer/0.log" Jul 10 01:02:42.194178 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:42.194146 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-k2hs8_f59c94d4-93a7-4602-bf06-b2e0a4b2e312/konnectivity-agent/0.log" Jul 10 01:02:42.252109 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:42.252081 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-30.ec2.internal_fea6ac5a21d75add74f7c2411fe7019d/haproxy/0.log" Jul 10 01:02:47.055051 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:47.055004 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-vtg5w_62fd95d9-a931-4eb7-a4db-2cceb02cbd7b/manager/0.log" Jul 10 01:02:47.121686 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:47.121648 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-cljqj_2f80f79a-3264-4b3e-8fa7-a6166013b333/registry-server/0.log" Jul 10 01:02:47.254737 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:47.254703 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-57jts_86b52ed9-417f-456f-a759-b59c75763236/manager/0.log" Jul 10 01:02:47.734230 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:47.733428 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7zrr_20198eb1-6b3e-4e30-9adf-d4f13df4ee3b/ovn-acl-logging/0.log" Jul 10 01:02:47.741134 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:47.740173 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7zrr_20198eb1-6b3e-4e30-9adf-d4f13df4ee3b/ovn-acl-logging/0.log" Jul 10 01:02:48.963455 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:48.963423 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f96592fc-0ecc-40f4-be2a-d5c713e1041e/alertmanager/0.log" Jul 10 01:02:48.990685 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:48.990656 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f96592fc-0ecc-40f4-be2a-d5c713e1041e/config-reloader/0.log" Jul 10 01:02:49.011629 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:49.011582 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f96592fc-0ecc-40f4-be2a-d5c713e1041e/kube-rbac-proxy-web/0.log" Jul 10 01:02:49.040874 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:49.040836 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f96592fc-0ecc-40f4-be2a-d5c713e1041e/kube-rbac-proxy/0.log" Jul 10 01:02:49.060344 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:49.060316 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f96592fc-0ecc-40f4-be2a-d5c713e1041e/kube-rbac-proxy-metric/0.log" Jul 10 01:02:49.085458 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:49.085433 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f96592fc-0ecc-40f4-be2a-d5c713e1041e/prom-label-proxy/0.log" Jul 10 01:02:49.106736 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:49.106694 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f96592fc-0ecc-40f4-be2a-d5c713e1041e/init-config-reloader/0.log" Jul 10 01:02:49.300519 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:49.300487 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-dx5j6_f089d09e-5f9a-4818-b10c-b1926646d89e/monitoring-plugin/0.log" Jul 10 01:02:49.404701 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:49.404671 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4szx5_04dd1d7f-86d8-49a0-89aa-e15bc09ac11f/node-exporter/0.log" Jul 10 01:02:49.426701 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:49.426638 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4szx5_04dd1d7f-86d8-49a0-89aa-e15bc09ac11f/kube-rbac-proxy/0.log" Jul 10 01:02:49.451320 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:49.451289 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4szx5_04dd1d7f-86d8-49a0-89aa-e15bc09ac11f/init-textfile/0.log" Jul 10 01:02:52.495148 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:52.495101 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6cvbh/perf-node-gather-daemonset-kkqxp"] Jul 10 01:02:52.500852 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:52.500826 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6cvbh/perf-node-gather-daemonset-kkqxp" Jul 10 01:02:52.507518 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:52.507488 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6cvbh/perf-node-gather-daemonset-kkqxp"] Jul 10 01:02:52.590303 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:52.590052 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-nxhzz_f699436c-1006-49b2-8dc4-061253a6b2b2/download-server/0.log" Jul 10 01:02:52.619533 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:52.619499 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9-podres\") pod \"perf-node-gather-daemonset-kkqxp\" (UID: \"5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9\") " pod="openshift-must-gather-6cvbh/perf-node-gather-daemonset-kkqxp" Jul 10 01:02:52.619727 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:52.619616 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9-sys\") pod \"perf-node-gather-daemonset-kkqxp\" (UID: \"5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9\") " pod="openshift-must-gather-6cvbh/perf-node-gather-daemonset-kkqxp" Jul 10 01:02:52.619727 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:52.619661 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9-lib-modules\") pod \"perf-node-gather-daemonset-kkqxp\" (UID: \"5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9\") " pod="openshift-must-gather-6cvbh/perf-node-gather-daemonset-kkqxp" Jul 10 01:02:52.619727 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:52.619698 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9-proc\") pod \"perf-node-gather-daemonset-kkqxp\" (UID: \"5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9\") " pod="openshift-must-gather-6cvbh/perf-node-gather-daemonset-kkqxp" Jul 10 01:02:52.619894 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:52.619753 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7ppp\" (UniqueName: \"kubernetes.io/projected/5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9-kube-api-access-x7ppp\") pod \"perf-node-gather-daemonset-kkqxp\" (UID: \"5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9\") " pod="openshift-must-gather-6cvbh/perf-node-gather-daemonset-kkqxp" Jul 10 01:02:52.720603 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:52.720563 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9-podres\") pod \"perf-node-gather-daemonset-kkqxp\" (UID: \"5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9\") " pod="openshift-must-gather-6cvbh/perf-node-gather-daemonset-kkqxp" Jul 10 01:02:52.720781 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:52.720685 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9-sys\") pod \"perf-node-gather-daemonset-kkqxp\" (UID: \"5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9\") " pod="openshift-must-gather-6cvbh/perf-node-gather-daemonset-kkqxp" Jul 10 01:02:52.720781 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:52.720712 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9-lib-modules\") pod \"perf-node-gather-daemonset-kkqxp\" (UID: \"5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9\") " pod="openshift-must-gather-6cvbh/perf-node-gather-daemonset-kkqxp" Jul 10 01:02:52.720781 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:52.720761 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9-podres\") pod \"perf-node-gather-daemonset-kkqxp\" (UID: \"5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9\") " pod="openshift-must-gather-6cvbh/perf-node-gather-daemonset-kkqxp" Jul 10 01:02:52.720957 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:52.720797 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9-sys\") pod \"perf-node-gather-daemonset-kkqxp\" (UID: \"5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9\") " pod="openshift-must-gather-6cvbh/perf-node-gather-daemonset-kkqxp" Jul 10 01:02:52.720957 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:52.720835 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9-lib-modules\") pod \"perf-node-gather-daemonset-kkqxp\" (UID: \"5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9\") " pod="openshift-must-gather-6cvbh/perf-node-gather-daemonset-kkqxp" Jul 10 01:02:52.720957 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:52.720845 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9-proc\") pod \"perf-node-gather-daemonset-kkqxp\" (UID: \"5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9\") " pod="openshift-must-gather-6cvbh/perf-node-gather-daemonset-kkqxp" Jul 10 01:02:52.720957 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:52.720882 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7ppp\" (UniqueName: \"kubernetes.io/projected/5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9-kube-api-access-x7ppp\") pod \"perf-node-gather-daemonset-kkqxp\" (UID: \"5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9\") " pod="openshift-must-gather-6cvbh/perf-node-gather-daemonset-kkqxp" Jul 10 01:02:52.721170 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:52.720962 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9-proc\") pod \"perf-node-gather-daemonset-kkqxp\" (UID: \"5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9\") " pod="openshift-must-gather-6cvbh/perf-node-gather-daemonset-kkqxp" Jul 10 01:02:52.728832 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:52.728802 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7ppp\" (UniqueName: \"kubernetes.io/projected/5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9-kube-api-access-x7ppp\") pod \"perf-node-gather-daemonset-kkqxp\" (UID: \"5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9\") " pod="openshift-must-gather-6cvbh/perf-node-gather-daemonset-kkqxp" Jul 10 01:02:52.817037 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:52.816989 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6cvbh/perf-node-gather-daemonset-kkqxp" Jul 10 01:02:53.002888 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:53.002429 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6cvbh/perf-node-gather-daemonset-kkqxp"] Jul 10 01:02:53.004797 ip-10-0-132-30 kubenswrapper[2559]: W0710 01:02:53.004736 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5d9aae2f_fbcb_46b2_8678_3e4c6e24f0d9.slice/crio-fca631d7361d2de76f4b90aaffb4d46f7d10519a9fde33ef4dfa1b03c15db0da WatchSource:0}: Error finding container fca631d7361d2de76f4b90aaffb4d46f7d10519a9fde33ef4dfa1b03c15db0da: Status 404 returned error can't find the container with id fca631d7361d2de76f4b90aaffb4d46f7d10519a9fde33ef4dfa1b03c15db0da Jul 10 01:02:53.835134 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:53.835077 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6cvbh/perf-node-gather-daemonset-kkqxp" event={"ID":"5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9","Type":"ContainerStarted","Data":"55dcc9ae02483bdacf5ad07c7c2a7a2f8c81fc071722bb6fb430887b7e14fff0"} Jul 10 01:02:53.835612 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:53.835168 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6cvbh/perf-node-gather-daemonset-kkqxp" event={"ID":"5d9aae2f-fbcb-46b2-8678-3e4c6e24f0d9","Type":"ContainerStarted","Data":"fca631d7361d2de76f4b90aaffb4d46f7d10519a9fde33ef4dfa1b03c15db0da"} Jul 10 01:02:53.835612 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:53.835393 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6cvbh/perf-node-gather-daemonset-kkqxp" Jul 10 01:02:53.850174 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:53.850101 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6cvbh/perf-node-gather-daemonset-kkqxp" podStartSLOduration=1.8500835690000001 podStartE2EDuration="1.850083569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 01:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 01:02:53.849379276 +0000 UTC m=+2706.782463578" watchObservedRunningTime="2026-07-10 01:02:53.850083569 +0000 UTC m=+2706.783167879" Jul 10 01:02:54.228791 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:54.228716 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ds66q_0654b471-0fff-4b5c-a6fc-d3e972163d20/dns/0.log" Jul 10 01:02:54.252049 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:54.251945 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ds66q_0654b471-0fff-4b5c-a6fc-d3e972163d20/kube-rbac-proxy/0.log" Jul 10 01:02:54.314258 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:54.314233 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gzg2p_733f8b73-939c-414d-9d90-7791508e5b84/dns-node-resolver/0.log" Jul 10 01:02:54.932862 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:54.932828 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jdrfw_613ac839-60fd-4f20-a127-a072d8a574bb/node-ca/0.log" Jul 10 01:02:55.857534 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:55.857494 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4xnd87_84663b2e-6507-4aac-8ad4-db9fa1621614/istio-proxy/0.log" Jul 10 01:02:56.134772 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:56.134672 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pfqlk_cf310fe5-e414-4b36-97d9-c99cb1da8844/istio-proxy/0.log" Jul 10 01:02:56.171533 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:56.171506 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pfqlk_cf310fe5-e414-4b36-97d9-c99cb1da8844/istio-proxy/1.log" Jul 10 01:02:56.201789 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:56.201758 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-56gc5_541871dd-0ab5-41ed-87c7-b688171167b0/payload-pre-processing/0.log" Jul 10 01:02:56.231416 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:56.231373 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-6bgxv_ebc9f72c-30bf-4071-ae1c-928caeb9d033/payload-processing/0.log" Jul 10 01:02:56.951307 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:56.951261 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-q8pzq_5918e467-096f-431c-a997-d13e6ddd04e0/serve-healthcheck-canary/0.log" Jul 10 01:02:57.561085 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:57.561057 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-khzbs_373f4170-5007-4ba6-8bc9-f109677f4f49/kube-rbac-proxy/0.log" Jul 10 01:02:57.581385 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:57.581352 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-khzbs_373f4170-5007-4ba6-8bc9-f109677f4f49/exporter/0.log" Jul 10 01:02:57.601830 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:57.601803 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-khzbs_373f4170-5007-4ba6-8bc9-f109677f4f49/extractor/0.log" Jul 10 01:02:59.853867 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:02:59.853839 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6cvbh/perf-node-gather-daemonset-kkqxp" Jul 10 01:03:00.002057 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:03:00.002020 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5fc87c469-xw8qr_4d94fc25-7cec-4ffa-9d26-ae8d3de15848/manager/0.log" Jul 10 01:03:00.330302 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:03:00.330269 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85d875657f-z2vng_5af46045-6974-437d-83ca-80cb4bcefa05/manager/0.log" Jul 10 01:03:00.349452 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:03:00.349429 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85d875657f-z2vng_5af46045-6974-437d-83ca-80cb4bcefa05/copy-manifests/0.log" Jul 10 01:03:01.697708 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:03:01.697600 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-56d4fb96db-zq65z_3ad3bac2-deb5-4191-8edc-28e5cf1ac514/manager/0.log" Jul 10 01:03:02.219252 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:03:02.219156 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4xnd87_84663b2e-6507-4aac-8ad4-db9fa1621614/istio-proxy/0.log" Jul 10 01:03:02.457082 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:03:02.457002 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pfqlk_cf310fe5-e414-4b36-97d9-c99cb1da8844/istio-proxy/0.log" Jul 10 01:03:02.499470 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:03:02.499399 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pfqlk_cf310fe5-e414-4b36-97d9-c99cb1da8844/istio-proxy/1.log" Jul 10 01:03:02.521812 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:03:02.521787 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-56gc5_541871dd-0ab5-41ed-87c7-b688171167b0/payload-pre-processing/0.log" Jul 10 01:03:02.543502 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:03:02.543480 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-6bgxv_ebc9f72c-30bf-4071-ae1c-928caeb9d033/payload-processing/0.log" Jul 10 01:03:03.642910 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:03:03.642882 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4xnd87_84663b2e-6507-4aac-8ad4-db9fa1621614/istio-proxy/0.log" Jul 10 01:03:03.896865 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:03:03.896785 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pfqlk_cf310fe5-e414-4b36-97d9-c99cb1da8844/istio-proxy/0.log" Jul 10 01:03:03.940283 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:03:03.940251 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-pfqlk_cf310fe5-e414-4b36-97d9-c99cb1da8844/istio-proxy/1.log" Jul 10 01:03:03.969289 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:03:03.969260 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-56gc5_541871dd-0ab5-41ed-87c7-b688171167b0/payload-pre-processing/0.log" Jul 10 01:03:03.996692 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:03:03.996570 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-f9b8d9b9b-6bgxv_ebc9f72c-30bf-4071-ae1c-928caeb9d033/payload-processing/0.log" Jul 10 01:03:08.368220 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:03:08.368193 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8hp4r_d642e8f4-ca88-46f0-ba74-e17a26ec2f7f/kube-multus-additional-cni-plugins/0.log" Jul 10 01:03:08.388614 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:03:08.388590 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8hp4r_d642e8f4-ca88-46f0-ba74-e17a26ec2f7f/egress-router-binary-copy/0.log" Jul 10 01:03:08.408675 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:03:08.408655 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8hp4r_d642e8f4-ca88-46f0-ba74-e17a26ec2f7f/cni-plugins/0.log" Jul 10 01:03:08.429077 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:03:08.429058 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8hp4r_d642e8f4-ca88-46f0-ba74-e17a26ec2f7f/bond-cni-plugin/0.log" Jul 10 01:03:08.448909 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:03:08.448888 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8hp4r_d642e8f4-ca88-46f0-ba74-e17a26ec2f7f/routeoverride-cni/0.log" Jul 10 01:03:08.470535 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:03:08.470514 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8hp4r_d642e8f4-ca88-46f0-ba74-e17a26ec2f7f/whereabouts-cni-bincopy/0.log" Jul 10 01:03:08.491033 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:03:08.491012 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8hp4r_d642e8f4-ca88-46f0-ba74-e17a26ec2f7f/whereabouts-cni/0.log" Jul 10 01:03:08.700646 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:03:08.700579 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hf6sh_de6e5b48-34f6-43bb-8b9d-d1e651860753/kube-multus/0.log" Jul 10 01:03:08.756731 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:03:08.756710 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ms9mf_376630cb-0d38-46f3-a311-7cf9af6f2d10/network-metrics-daemon/0.log" Jul 10 01:03:08.774835 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:03:08.774816 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ms9mf_376630cb-0d38-46f3-a311-7cf9af6f2d10/kube-rbac-proxy/0.log" Jul 10 01:03:09.705707 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:03:09.705663 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7zrr_20198eb1-6b3e-4e30-9adf-d4f13df4ee3b/ovn-controller/0.log" Jul 10 01:03:09.721600 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:03:09.721574 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7zrr_20198eb1-6b3e-4e30-9adf-d4f13df4ee3b/ovn-acl-logging/1.log" Jul 10 01:03:09.721754 ip-10-0-132-30 kubenswrapper[2559]: I0710 01:03:09.721737 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-p7zrr_20198eb1-6b3e-4e30-9adf-d4f13df4ee3b/ovn-acl-logging/0.log"