Jul 08 18:37:46.446303 ip-10-0-131-102 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 08 18:37:46.446316 ip-10-0-131-102 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 08 18:37:46.446326 ip-10-0-131-102 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 08 18:37:46.446651 ip-10-0-131-102 systemd[1]: Failed to start Kubernetes Kubelet. Jul 08 18:37:56.535802 ip-10-0-131-102 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 08 18:37:56.535822 ip-10-0-131-102 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot fab6fd160623493992813a4437e3577e -- Jul 08 18:39:49.037648 ip-10-0-131-102 systemd[1]: Starting Kubernetes Kubelet... Jul 08 18:39:49.471440 ip-10-0-131-102 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 18:39:49.471440 ip-10-0-131-102 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 08 18:39:49.471440 ip-10-0-131-102 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 18:39:49.471440 ip-10-0-131-102 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 08 18:39:49.471440 ip-10-0-131-102 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 18:39:49.471440 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.467718 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 08 18:39:49.473777 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473671 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 18:39:49.473777 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473687 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 18:39:49.473777 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473691 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 18:39:49.473777 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473694 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 18:39:49.473777 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473697 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 18:39:49.473777 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473700 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 18:39:49.473777 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473702 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 18:39:49.473777 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473705 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 18:39:49.473777 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473708 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 18:39:49.473777 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473710 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 18:39:49.473777 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473713 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 18:39:49.473777 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473716 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 18:39:49.473777 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473720 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 18:39:49.473777 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473722 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 18:39:49.473777 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473725 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 18:39:49.473777 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473728 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 18:39:49.473777 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473730 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 18:39:49.473777 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473733 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 18:39:49.473777 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473735 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 18:39:49.473777 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473738 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 18:39:49.554951 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473740 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 18:39:49.554951 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473743 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 18:39:49.554951 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473752 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 18:39:49.554951 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473755 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 18:39:49.554951 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473758 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 18:39:49.554951 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473760 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 18:39:49.554951 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473765 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 18:39:49.554951 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473770 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 18:39:49.554951 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473773 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 18:39:49.554951 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473776 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 18:39:49.554951 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473779 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 18:39:49.554951 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473782 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 18:39:49.554951 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473784 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 18:39:49.554951 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473787 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 18:39:49.554951 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473790 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 18:39:49.554951 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473793 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 18:39:49.554951 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473795 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 18:39:49.554951 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473798 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 18:39:49.554951 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473800 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 18:39:49.549578 ip-10-0-131-102 systemd[1]: Started Kubernetes Kubelet. Jul 08 18:39:49.556052 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473803 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 18:39:49.556052 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473805 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 18:39:49.556052 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473808 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 18:39:49.556052 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473810 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 18:39:49.556052 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473813 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 18:39:49.556052 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473816 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 18:39:49.556052 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473818 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 18:39:49.556052 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473822 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 18:39:49.556052 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473824 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 18:39:49.556052 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473827 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 18:39:49.556052 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473830 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 18:39:49.556052 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473834 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 18:39:49.556052 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473836 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 18:39:49.556052 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473839 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 18:39:49.556052 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473842 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 18:39:49.556052 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473844 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 18:39:49.556052 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473847 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 18:39:49.556052 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473850 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 18:39:49.556052 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473852 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 18:39:49.556052 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473857 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 18:39:49.559147 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473860 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 18:39:49.559147 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473863 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 18:39:49.559147 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473867 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 18:39:49.559147 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473871 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 18:39:49.559147 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473873 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 18:39:49.559147 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473876 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 18:39:49.559147 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473878 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 18:39:49.559147 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473881 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 18:39:49.559147 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473883 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 18:39:49.559147 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473886 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 18:39:49.559147 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473888 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 18:39:49.559147 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473891 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 18:39:49.559147 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473894 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 18:39:49.559147 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473896 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 18:39:49.559147 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473899 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 18:39:49.559147 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473901 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 18:39:49.559147 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473904 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 18:39:49.559147 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473906 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 18:39:49.559147 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473909 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 18:39:49.560035 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473912 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 18:39:49.560035 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473914 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 18:39:49.560035 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473919 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 18:39:49.560035 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473921 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 18:39:49.560035 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473924 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 18:39:49.560035 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473926 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 18:39:49.560035 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473929 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 18:39:49.560035 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473931 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 18:39:49.560035 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473934 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 18:39:49.560035 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473937 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 18:39:49.560035 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473939 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 18:39:49.560035 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473942 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 18:39:49.560035 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473946 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 18:39:49.560035 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473948 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 18:39:49.560035 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473951 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 18:39:49.560035 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473953 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 18:39:49.560035 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473956 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 18:39:49.560035 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473958 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 18:39:49.560035 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473961 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 08 18:39:49.560035 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473963 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 18:39:49.563055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.473966 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 18:39:49.563055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474711 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 18:39:49.563055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474718 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 18:39:49.563055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474721 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 18:39:49.563055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474724 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 18:39:49.563055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474726 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 18:39:49.563055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474730 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 18:39:49.563055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474733 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 18:39:49.563055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474736 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 18:39:49.563055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474740 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 18:39:49.563055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474746 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 18:39:49.563055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474750 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 18:39:49.563055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474753 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 18:39:49.563055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474756 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 18:39:49.563055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474758 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 18:39:49.563055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474761 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 18:39:49.563055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474764 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 18:39:49.563055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474767 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 18:39:49.563055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474769 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 18:39:49.563055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474772 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 18:39:49.564137 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474775 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 18:39:49.564137 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474778 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 18:39:49.564137 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474780 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 18:39:49.564137 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474782 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 18:39:49.564137 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474785 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 18:39:49.564137 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474788 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 18:39:49.564137 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474790 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 18:39:49.564137 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474793 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 18:39:49.564137 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474795 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 08 18:39:49.564137 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474798 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 18:39:49.564137 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474801 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 18:39:49.564137 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474803 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 18:39:49.564137 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474805 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 18:39:49.564137 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474808 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 18:39:49.564137 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474810 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 18:39:49.564137 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474812 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 18:39:49.564137 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474815 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 18:39:49.564137 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474817 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 18:39:49.564137 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474820 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 18:39:49.564137 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474823 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 18:39:49.566702 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474825 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 18:39:49.566702 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474828 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 18:39:49.566702 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474831 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 18:39:49.566702 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474834 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 18:39:49.566702 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474836 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 18:39:49.566702 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474839 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 18:39:49.566702 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474841 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 18:39:49.566702 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474844 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 18:39:49.566702 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474846 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 18:39:49.566702 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474849 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 18:39:49.566702 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474852 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 18:39:49.566702 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474855 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 18:39:49.566702 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474857 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 18:39:49.566702 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474863 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 18:39:49.566702 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474866 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 18:39:49.566702 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474869 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 18:39:49.566702 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474872 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 18:39:49.566702 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474875 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 18:39:49.566702 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474877 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 18:39:49.566702 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474880 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 18:39:49.567782 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474883 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 18:39:49.567782 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474887 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 18:39:49.567782 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474890 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 18:39:49.567782 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474893 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 18:39:49.567782 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474896 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 18:39:49.567782 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474899 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 18:39:49.567782 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474902 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 18:39:49.567782 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474905 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 18:39:49.567782 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474908 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 18:39:49.567782 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474911 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 18:39:49.567782 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474913 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 18:39:49.567782 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474917 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 18:39:49.567782 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474920 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 18:39:49.567782 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474923 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 18:39:49.567782 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474926 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 18:39:49.567782 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474928 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 18:39:49.567782 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474931 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 18:39:49.567782 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474934 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 18:39:49.567782 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474936 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 18:39:49.569925 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474939 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 18:39:49.569925 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474942 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 18:39:49.569925 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474944 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 18:39:49.569925 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474947 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 18:39:49.569925 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474950 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 18:39:49.569925 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474952 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 18:39:49.569925 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474955 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 18:39:49.569925 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474957 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 18:39:49.569925 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474960 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 18:39:49.569925 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474962 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 18:39:49.569925 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474965 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 18:39:49.569925 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474967 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 18:39:49.569925 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474970 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 18:39:49.569925 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474973 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 18:39:49.569925 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474975 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 18:39:49.569925 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474978 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 18:39:49.569925 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474980 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 18:39:49.569925 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474983 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 18:39:49.569925 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474985 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 18:39:49.569925 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474988 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 18:39:49.571024 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.474990 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 18:39:49.571024 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475428 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 18:39:49.571024 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475434 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 18:39:49.571024 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475438 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 18:39:49.571024 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475441 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 18:39:49.571024 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475444 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 18:39:49.571024 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475447 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 18:39:49.571024 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475450 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 18:39:49.571024 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475454 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 18:39:49.571024 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475457 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 18:39:49.571024 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475461 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 18:39:49.571024 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475464 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 18:39:49.571024 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475467 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 18:39:49.571024 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475470 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 18:39:49.571024 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475472 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 18:39:49.571024 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475475 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 18:39:49.571024 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475478 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 18:39:49.571024 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475480 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 18:39:49.571024 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475483 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 18:39:49.574105 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475486 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 18:39:49.574105 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475490 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 18:39:49.574105 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475493 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 18:39:49.574105 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475495 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 18:39:49.574105 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475498 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 18:39:49.574105 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475500 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 18:39:49.574105 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475503 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 18:39:49.574105 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475505 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 18:39:49.574105 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475508 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 18:39:49.574105 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475510 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 18:39:49.574105 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475513 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 18:39:49.574105 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475515 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 18:39:49.574105 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475518 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 18:39:49.574105 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475520 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 18:39:49.574105 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475523 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 18:39:49.574105 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475525 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 18:39:49.574105 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475528 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 18:39:49.574105 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475531 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 18:39:49.574105 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475533 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 18:39:49.574105 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475537 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 18:39:49.575200 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475541 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 18:39:49.575200 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475545 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 18:39:49.575200 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475547 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 18:39:49.575200 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475550 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 18:39:49.575200 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475554 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 18:39:49.575200 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475557 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 18:39:49.575200 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475559 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 18:39:49.575200 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475562 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 18:39:49.575200 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475565 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 18:39:49.575200 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475568 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 18:39:49.575200 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475570 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 18:39:49.575200 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475573 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 18:39:49.575200 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475576 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 18:39:49.575200 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475578 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 18:39:49.575200 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475581 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 18:39:49.575200 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475583 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 18:39:49.575200 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475586 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 18:39:49.575200 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475588 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 18:39:49.575200 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475591 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 18:39:49.575200 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475593 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 18:39:49.578650 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475596 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 18:39:49.578650 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475598 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 18:39:49.578650 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475600 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 18:39:49.578650 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475603 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 18:39:49.578650 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475606 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 18:39:49.578650 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475610 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 18:39:49.578650 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475614 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 18:39:49.578650 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475621 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 18:39:49.578650 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475624 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 18:39:49.578650 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475627 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 18:39:49.578650 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475630 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 18:39:49.578650 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475632 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 18:39:49.578650 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475637 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 18:39:49.578650 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475640 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 18:39:49.578650 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475643 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 18:39:49.578650 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475646 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 18:39:49.578650 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475648 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 18:39:49.578650 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475651 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 18:39:49.578650 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475653 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 18:39:49.579739 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475656 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 18:39:49.579739 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475658 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 18:39:49.579739 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475661 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 18:39:49.579739 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475663 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 18:39:49.579739 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475666 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 18:39:49.579739 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475668 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 18:39:49.579739 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475671 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 18:39:49.579739 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475673 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 18:39:49.579739 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475676 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 18:39:49.579739 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475678 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 08 18:39:49.579739 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475681 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 18:39:49.579739 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475683 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 18:39:49.579739 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475686 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 18:39:49.579739 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475688 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 18:39:49.579739 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475690 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 18:39:49.579739 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475693 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 18:39:49.579739 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475695 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 18:39:49.579739 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475698 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 18:39:49.579739 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475700 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 18:39:49.579739 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475703 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 18:39:49.580618 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475707 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 18:39:49.580618 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.475709 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 18:39:49.580618 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475789 2575 flags.go:64] FLAG: --address="0.0.0.0" Jul 08 18:39:49.580618 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475800 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 08 18:39:49.580618 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475811 2575 flags.go:64] FLAG: --anonymous-auth="true" Jul 08 18:39:49.580618 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475819 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 08 18:39:49.580618 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475826 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 08 18:39:49.580618 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475831 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 08 18:39:49.580618 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475837 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 08 18:39:49.580618 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475842 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 08 18:39:49.580618 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475845 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 08 18:39:49.580618 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475848 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 08 18:39:49.580618 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475852 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 08 18:39:49.580618 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475855 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 08 18:39:49.580618 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475858 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 08 18:39:49.580618 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475861 2575 flags.go:64] FLAG: --cgroup-root="" Jul 08 18:39:49.580618 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475864 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 08 18:39:49.580618 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475867 2575 flags.go:64] FLAG: --client-ca-file="" Jul 08 18:39:49.580618 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475870 2575 flags.go:64] FLAG: --cloud-provider="external" Jul 08 18:39:49.580618 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475873 2575 flags.go:64] FLAG: --cluster-dns="[]" Jul 08 18:39:49.580618 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475877 2575 flags.go:64] FLAG: --cluster-domain="" Jul 08 18:39:49.580618 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475880 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 08 18:39:49.580618 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475883 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 08 18:39:49.580618 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475886 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 08 18:39:49.584039 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475890 2575 flags.go:64] FLAG: --container-log-max-files="5" Jul 08 18:39:49.584039 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475894 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 08 18:39:49.584039 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475897 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 08 18:39:49.584039 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475900 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 08 18:39:49.584039 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475903 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 08 18:39:49.584039 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475906 2575 flags.go:64] FLAG: --contention-profiling="false" Jul 08 18:39:49.584039 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475909 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 08 18:39:49.584039 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475912 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 08 18:39:49.584039 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475917 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 08 18:39:49.584039 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475920 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 08 18:39:49.584039 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475925 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 08 18:39:49.584039 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475928 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 08 18:39:49.584039 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475931 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 08 18:39:49.584039 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475936 2575 flags.go:64] FLAG: --enable-load-reader="false" Jul 08 18:39:49.584039 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475939 2575 flags.go:64] FLAG: --enable-server="true" Jul 08 18:39:49.584039 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475942 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 08 18:39:49.584039 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475947 2575 flags.go:64] FLAG: --event-burst="100" Jul 08 18:39:49.584039 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475950 2575 flags.go:64] FLAG: --event-qps="50" Jul 08 18:39:49.584039 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475953 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 08 18:39:49.584039 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475956 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 08 18:39:49.584039 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475959 2575 flags.go:64] FLAG: --eviction-hard="" Jul 08 18:39:49.584039 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475963 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 08 18:39:49.584039 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475966 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 08 18:39:49.584039 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475969 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 08 18:39:49.584039 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475971 2575 flags.go:64] FLAG: --eviction-soft="" Jul 08 18:39:49.585129 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475975 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 08 18:39:49.585129 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475977 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 08 18:39:49.585129 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475980 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 08 18:39:49.585129 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475983 2575 flags.go:64] FLAG: --experimental-mounter-path="" Jul 08 18:39:49.585129 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475986 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 08 18:39:49.585129 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475989 2575 flags.go:64] FLAG: --fail-swap-on="true" Jul 08 18:39:49.585129 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475992 2575 flags.go:64] FLAG: --feature-gates="" Jul 08 18:39:49.585129 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475996 2575 flags.go:64] FLAG: --file-check-frequency="20s" Jul 08 18:39:49.585129 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.475999 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 08 18:39:49.585129 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476002 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 08 18:39:49.585129 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476005 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 08 18:39:49.585129 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476008 2575 flags.go:64] FLAG: --healthz-port="10248" Jul 08 18:39:49.585129 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476011 2575 flags.go:64] FLAG: --help="false" Jul 08 18:39:49.585129 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476013 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-131-102.ec2.internal" Jul 08 18:39:49.585129 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476016 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 08 18:39:49.585129 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476020 2575 flags.go:64] FLAG: --http-check-frequency="20s" Jul 08 18:39:49.585129 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476023 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 08 18:39:49.585129 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476027 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 08 18:39:49.585129 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476030 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 08 18:39:49.585129 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476033 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 08 18:39:49.585129 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476038 2575 flags.go:64] FLAG: --image-service-endpoint="" Jul 08 18:39:49.585129 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476041 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 08 18:39:49.585129 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476044 2575 flags.go:64] FLAG: --kube-api-burst="100" Jul 08 18:39:49.586216 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476047 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 08 18:39:49.586216 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476050 2575 flags.go:64] FLAG: --kube-api-qps="50" Jul 08 18:39:49.586216 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476053 2575 flags.go:64] FLAG: --kube-reserved="" Jul 08 18:39:49.586216 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476056 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 08 18:39:49.586216 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476058 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 08 18:39:49.586216 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476061 2575 flags.go:64] FLAG: --kubelet-cgroups="" Jul 08 18:39:49.586216 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476064 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 08 18:39:49.586216 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476067 2575 flags.go:64] FLAG: --lock-file="" Jul 08 18:39:49.586216 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476069 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 08 18:39:49.586216 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476072 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 08 18:39:49.586216 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476075 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 08 18:39:49.586216 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476081 2575 flags.go:64] FLAG: --log-json-split-stream="false" Jul 08 18:39:49.586216 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476083 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 08 18:39:49.586216 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476086 2575 flags.go:64] FLAG: --log-text-split-stream="false" Jul 08 18:39:49.586216 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476089 2575 flags.go:64] FLAG: --logging-format="text" Jul 08 18:39:49.586216 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476092 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 08 18:39:49.586216 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476095 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 08 18:39:49.586216 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476098 2575 flags.go:64] FLAG: --manifest-url="" Jul 08 18:39:49.586216 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476101 2575 flags.go:64] FLAG: --manifest-url-header="" Jul 08 18:39:49.586216 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476105 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 08 18:39:49.586216 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476108 2575 flags.go:64] FLAG: --max-open-files="1000000" Jul 08 18:39:49.586216 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476112 2575 flags.go:64] FLAG: --max-pods="110" Jul 08 18:39:49.586216 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476115 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 08 18:39:49.586216 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476118 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 08 18:39:49.586216 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476122 2575 flags.go:64] FLAG: --memory-manager-policy="None" Jul 08 18:39:49.588174 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476125 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 08 18:39:49.588174 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476128 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 08 18:39:49.588174 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476131 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 08 18:39:49.588174 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476133 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 08 18:39:49.588174 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476142 2575 flags.go:64] FLAG: --node-status-max-images="50" Jul 08 18:39:49.588174 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476145 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 08 18:39:49.588174 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476148 2575 flags.go:64] FLAG: --oom-score-adj="-999" Jul 08 18:39:49.588174 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476151 2575 flags.go:64] FLAG: --pod-cidr="" Jul 08 18:39:49.588174 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476154 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 08 18:39:49.588174 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476160 2575 flags.go:64] FLAG: --pod-manifest-path="" Jul 08 18:39:49.588174 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476163 2575 flags.go:64] FLAG: --pod-max-pids="-1" Jul 08 18:39:49.588174 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476166 2575 flags.go:64] FLAG: --pods-per-core="0" Jul 08 18:39:49.588174 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476169 2575 flags.go:64] FLAG: --port="10250" Jul 08 18:39:49.588174 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476172 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 08 18:39:49.588174 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476174 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0af7a6ffe9b46b68d" Jul 08 18:39:49.588174 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476178 2575 flags.go:64] FLAG: --qos-reserved="" Jul 08 18:39:49.588174 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476181 2575 flags.go:64] FLAG: --read-only-port="10255" Jul 08 18:39:49.588174 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476184 2575 flags.go:64] FLAG: --register-node="true" Jul 08 18:39:49.588174 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476187 2575 flags.go:64] FLAG: --register-with-taints="" Jul 08 18:39:49.588174 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476190 2575 flags.go:64] FLAG: --registry-burst="10" Jul 08 18:39:49.588174 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476193 2575 flags.go:64] FLAG: --registry-qps="5" Jul 08 18:39:49.588174 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476196 2575 flags.go:64] FLAG: --reserved-cpus="" Jul 08 18:39:49.588174 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476198 2575 flags.go:64] FLAG: --reserved-memory="" Jul 08 18:39:49.588174 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476202 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 08 18:39:49.588174 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476205 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 08 18:39:49.589262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476208 2575 flags.go:64] FLAG: --rotate-certificates="false" Jul 08 18:39:49.589262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476211 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 08 18:39:49.589262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476214 2575 flags.go:64] FLAG: --runonce="false" Jul 08 18:39:49.589262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476217 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 08 18:39:49.589262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476220 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 08 18:39:49.589262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476223 2575 flags.go:64] FLAG: --seccomp-default="false" Jul 08 18:39:49.589262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476228 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 08 18:39:49.589262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476230 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 08 18:39:49.589262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476234 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 08 18:39:49.589262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476237 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 08 18:39:49.589262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476240 2575 flags.go:64] FLAG: --storage-driver-password="root" Jul 08 18:39:49.589262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476242 2575 flags.go:64] FLAG: --storage-driver-secure="false" Jul 08 18:39:49.589262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476247 2575 flags.go:64] FLAG: --storage-driver-table="stats" Jul 08 18:39:49.589262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476249 2575 flags.go:64] FLAG: --storage-driver-user="root" Jul 08 18:39:49.589262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476252 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 08 18:39:49.589262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476255 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 08 18:39:49.589262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476259 2575 flags.go:64] FLAG: --system-cgroups="" Jul 08 18:39:49.589262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476262 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 08 18:39:49.589262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476267 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 08 18:39:49.589262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476271 2575 flags.go:64] FLAG: --tls-cert-file="" Jul 08 18:39:49.589262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476273 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 08 18:39:49.589262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476277 2575 flags.go:64] FLAG: --tls-min-version="" Jul 08 18:39:49.589262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476280 2575 flags.go:64] FLAG: --tls-private-key-file="" Jul 08 18:39:49.589262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476283 2575 flags.go:64] FLAG: --topology-manager-policy="none" Jul 08 18:39:49.589262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476286 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 08 18:39:49.694887 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476289 2575 flags.go:64] FLAG: --topology-manager-scope="container" Jul 08 18:39:49.694887 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476304 2575 flags.go:64] FLAG: --v="2" Jul 08 18:39:49.694887 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476309 2575 flags.go:64] FLAG: --version="false" Jul 08 18:39:49.694887 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476313 2575 flags.go:64] FLAG: --vmodule="" Jul 08 18:39:49.694887 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476317 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 08 18:39:49.694887 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.476320 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 08 18:39:49.694887 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476442 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 18:39:49.694887 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476445 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 18:39:49.694887 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476448 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 18:39:49.694887 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476451 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 18:39:49.694887 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476453 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 18:39:49.694887 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476456 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 18:39:49.694887 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476459 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 18:39:49.694887 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476463 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 18:39:49.694887 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476466 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 18:39:49.694887 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476468 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 18:39:49.694887 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476471 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 18:39:49.694887 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476473 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 18:39:49.694887 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476476 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 18:39:49.694887 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476480 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 18:39:49.694887 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476482 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 18:39:49.696439 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476486 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 18:39:49.696439 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476488 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 18:39:49.696439 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476491 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 18:39:49.696439 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476495 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 08 18:39:49.696439 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476498 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 18:39:49.696439 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476500 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 18:39:49.696439 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476503 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 18:39:49.696439 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476505 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 18:39:49.696439 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476508 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 18:39:49.696439 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476511 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 18:39:49.696439 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476513 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 18:39:49.696439 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476516 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 18:39:49.696439 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476518 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 18:39:49.696439 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476521 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 18:39:49.696439 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476525 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 18:39:49.696439 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476529 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 18:39:49.696439 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476532 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 18:39:49.696439 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476535 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 18:39:49.696439 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476537 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 18:39:49.696439 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476540 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 18:39:49.699459 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476543 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 18:39:49.699459 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476545 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 18:39:49.699459 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476547 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 18:39:49.699459 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476550 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 18:39:49.699459 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476554 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 18:39:49.699459 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476556 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 18:39:49.699459 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476559 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 18:39:49.699459 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476561 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 18:39:49.699459 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476565 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 18:39:49.699459 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476569 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 18:39:49.699459 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476573 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 18:39:49.699459 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476576 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 18:39:49.699459 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476578 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 18:39:49.699459 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476581 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 18:39:49.699459 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476583 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 18:39:49.699459 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476586 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 18:39:49.699459 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476590 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 18:39:49.699459 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476592 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 18:39:49.699459 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476595 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 18:39:49.700786 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476598 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 18:39:49.700786 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476601 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 18:39:49.700786 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476604 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 18:39:49.700786 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476607 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 18:39:49.700786 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476609 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 18:39:49.700786 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476611 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 18:39:49.700786 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476614 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 18:39:49.700786 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476617 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 18:39:49.700786 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476619 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 18:39:49.700786 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476622 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 18:39:49.700786 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476625 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 18:39:49.700786 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476627 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 18:39:49.700786 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476630 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 18:39:49.700786 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476632 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 18:39:49.700786 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476635 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 18:39:49.700786 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476637 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 18:39:49.700786 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476639 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 18:39:49.700786 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476643 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 18:39:49.700786 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476646 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 18:39:49.700786 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476648 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 18:39:49.851094 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476651 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 18:39:49.851094 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476653 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 18:39:49.851094 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476656 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 18:39:49.851094 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476659 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 18:39:49.851094 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476662 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 18:39:49.851094 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476664 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 18:39:49.851094 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476667 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 18:39:49.851094 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476670 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 18:39:49.851094 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476673 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 18:39:49.851094 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476675 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 18:39:49.851094 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476683 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 18:39:49.851094 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476685 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 18:39:49.851094 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476688 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 18:39:49.851094 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476690 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 18:39:49.851094 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476693 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 18:39:49.851094 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476695 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 18:39:49.851094 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476698 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 18:39:49.851094 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476700 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 18:39:49.851094 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476703 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 18:39:49.853126 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476705 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 18:39:49.853126 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476708 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 18:39:49.853126 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476710 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 18:39:49.853126 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476712 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 18:39:49.853126 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476715 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 18:39:49.853126 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.476717 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 18:39:49.853126 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.477246 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 18:39:49.853126 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.489830 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 08 18:39:49.853126 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.489845 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 08 18:39:49.853126 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.489862 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 08 18:39:49.853126 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.489870 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 08 18:39:49.853126 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.489938 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 18:39:49.853126 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.489943 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 18:39:49.853126 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.489946 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 18:39:49.854055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.489949 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 18:39:49.854055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.489952 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 18:39:49.854055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.489955 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 18:39:49.854055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.489961 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 18:39:49.854055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.489965 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 18:39:49.854055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.489969 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 18:39:49.854055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.489972 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 18:39:49.854055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.489975 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 18:39:49.854055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.489977 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 18:39:49.854055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.489989 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 18:39:49.854055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.489993 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 18:39:49.854055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.489997 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 18:39:49.854055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.489999 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 18:39:49.854055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490003 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 18:39:49.854055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490006 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 18:39:49.854055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490009 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 18:39:49.854055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490012 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 18:39:49.854055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490018 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 18:39:49.854055 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490021 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 18:39:49.855892 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490025 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 18:39:49.855892 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490028 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 18:39:49.855892 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490031 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 08 18:39:49.855892 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490034 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 18:39:49.855892 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490037 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 18:39:49.855892 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490040 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 18:39:49.855892 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490042 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 18:39:49.855892 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490045 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 18:39:49.855892 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490048 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 18:39:49.855892 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490052 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 18:39:49.855892 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490055 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 18:39:49.855892 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490057 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 18:39:49.855892 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490060 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 18:39:49.855892 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490063 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 18:39:49.855892 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490065 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 18:39:49.855892 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490068 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 18:39:49.855892 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490072 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 18:39:49.855892 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490075 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 18:39:49.855892 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490077 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 18:39:49.856972 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490080 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 18:39:49.856972 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490082 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 18:39:49.856972 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490085 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 18:39:49.856972 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490087 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 18:39:49.856972 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490097 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 18:39:49.856972 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490101 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 18:39:49.856972 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490103 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 18:39:49.856972 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490106 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 18:39:49.856972 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490108 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 18:39:49.856972 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490111 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 18:39:49.856972 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490113 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 18:39:49.856972 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490116 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 18:39:49.856972 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490119 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 18:39:49.856972 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490121 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 18:39:49.856972 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490123 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 18:39:49.856972 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490126 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 18:39:49.856972 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490129 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 18:39:49.856972 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490131 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 18:39:49.856972 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490134 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 18:39:49.856972 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490136 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 18:39:49.858352 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490138 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 18:39:49.858352 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490141 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 18:39:49.858352 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490145 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 18:39:49.858352 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490148 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 18:39:49.858352 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490150 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 18:39:49.858352 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490153 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 18:39:49.858352 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490155 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 18:39:49.858352 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490158 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 18:39:49.858352 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490160 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 18:39:49.858352 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490164 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 18:39:49.858352 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490167 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 18:39:49.858352 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490169 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 18:39:49.858352 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490172 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 18:39:49.858352 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490175 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 18:39:49.858352 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490178 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 18:39:49.858352 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490180 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 18:39:49.858352 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490183 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 18:39:49.858352 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490191 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 18:39:49.858352 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490194 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 18:39:49.858352 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490196 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 18:39:49.859746 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490198 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 18:39:49.859746 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490201 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 18:39:49.859746 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490204 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 18:39:49.859746 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490206 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 18:39:49.859746 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490209 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 18:39:49.859746 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490211 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 18:39:49.859746 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490214 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 18:39:49.859746 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490216 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 18:39:49.859746 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490219 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 18:39:49.859746 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490221 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 18:39:49.859746 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490224 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 18:39:49.859746 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490227 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 18:39:49.859746 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490229 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 18:39:49.859746 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490232 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 18:39:49.859746 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490236 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 18:39:49.859746 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490238 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 18:39:49.859746 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490240 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 18:39:49.859746 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490243 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 18:39:49.860913 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.490248 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 18:39:49.860913 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490413 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 18:39:49.860913 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490418 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 18:39:49.860913 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490421 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 18:39:49.860913 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490425 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 18:39:49.860913 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490428 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 18:39:49.860913 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490431 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 18:39:49.860913 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490434 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 18:39:49.860913 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490436 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 18:39:49.860913 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490439 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 18:39:49.860913 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490447 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 18:39:49.860913 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490451 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 18:39:49.860913 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490460 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 18:39:49.860913 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490462 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 18:39:49.860913 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490465 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 18:39:49.862761 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490468 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 18:39:49.862761 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490471 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 18:39:49.862761 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490473 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 18:39:49.862761 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490476 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 18:39:49.862761 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490480 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 18:39:49.862761 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490483 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 18:39:49.862761 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490486 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 18:39:49.862761 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490489 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 18:39:49.862761 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490491 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 18:39:49.862761 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490494 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 18:39:49.862761 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490496 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 18:39:49.862761 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490499 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 18:39:49.862761 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490502 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 18:39:49.862761 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490512 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 18:39:49.862761 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490515 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 18:39:49.862761 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490517 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 18:39:49.862761 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490520 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 18:39:49.862761 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490522 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 18:39:49.862761 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490525 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 18:39:49.862761 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490527 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 18:39:49.863666 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490530 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 18:39:49.863666 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490534 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 18:39:49.863666 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490536 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 18:39:49.863666 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490539 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 18:39:49.863666 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490541 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 18:39:49.863666 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490544 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 18:39:49.863666 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490547 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 18:39:49.863666 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490549 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 18:39:49.863666 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490552 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 18:39:49.863666 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490554 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 18:39:49.863666 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490563 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 18:39:49.863666 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490566 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 18:39:49.863666 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490569 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 18:39:49.863666 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490571 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 18:39:49.863666 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490574 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 18:39:49.863666 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490576 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 18:39:49.863666 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490579 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 18:39:49.863666 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490581 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 18:39:49.863666 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490584 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 18:39:49.864457 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490587 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 18:39:49.864457 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490589 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 18:39:49.864457 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490592 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 18:39:49.864457 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490594 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 18:39:49.864457 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490597 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 18:39:49.864457 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490599 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 18:39:49.864457 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490603 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 18:39:49.864457 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490605 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 18:39:49.864457 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490608 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 18:39:49.864457 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490610 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 18:39:49.864457 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490613 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 18:39:49.864457 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490615 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 18:39:49.864457 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490618 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 18:39:49.864457 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490620 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 18:39:49.864457 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490625 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 18:39:49.864457 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490627 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 18:39:49.864457 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490630 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 18:39:49.864457 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490632 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 18:39:49.864457 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490637 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 18:39:49.865249 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490640 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 18:39:49.865249 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490643 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 18:39:49.865249 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490646 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 18:39:49.865249 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490649 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 18:39:49.865249 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490651 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 18:39:49.865249 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490659 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 18:39:49.865249 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490662 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 18:39:49.865249 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490664 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 18:39:49.865249 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490667 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 08 18:39:49.865249 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490669 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 18:39:49.865249 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490672 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 18:39:49.865249 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490674 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 18:39:49.865249 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490677 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 18:39:49.865249 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490679 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 18:39:49.865249 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490682 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 18:39:49.865249 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490684 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 18:39:49.865249 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490687 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 18:39:49.865249 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490689 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 18:39:49.865249 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490692 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 18:39:49.865249 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490696 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 18:39:49.866088 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490699 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 18:39:49.866088 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490701 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 18:39:49.866088 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490704 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 18:39:49.866088 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490707 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 18:39:49.866088 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490709 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 18:39:49.866088 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490711 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 18:39:49.866088 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.490714 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 18:39:49.866088 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.490721 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 18:39:49.866088 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.490851 2575 server.go:962] "Client rotation is on, will bootstrap in background" Jul 08 18:39:49.866088 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.494422 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 08 18:39:49.866088 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.495991 2575 server.go:1019] "Starting client certificate rotation" Jul 08 18:39:49.866088 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.496099 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 18:39:49.866088 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.496138 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 18:39:49.866652 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.498611 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 18:39:49.866652 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.500312 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 18:39:49.866652 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.513406 2575 log.go:25] "Validated CRI v1 runtime API" Jul 08 18:39:49.866652 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.515665 2575 log.go:25] "Validated CRI v1 image API" Jul 08 18:39:49.866652 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.516286 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 08 18:39:49.866652 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.519361 2575 fs.go:135] Filesystem UUIDs: map[3bc9555f-3ff1-44ee-8bd4-a7780e286619:/dev/nvme0n1p3 7534be46-3d56-4037-85e3-995b1732c068:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 08 18:39:49.866652 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.519381 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 08 18:39:49.866957 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.525784 2575 manager.go:216] Machine: {Timestamp:2026-07-08 18:39:49.524262559 +0000 UTC m=+0.361590263 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101670 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25d31da8b42f13319f0d21893b7192 SystemUUID:ec25d31d-a8b4-2f13-319f-0d21893b7192 BootID:fab6fd16-0623-4939-9281-3a4437e3577e Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:90:58:e0:2d:db Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:90:58:e0:2d:db Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:bc:29:f9:c4:bd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 08 18:39:49.866957 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.525935 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 08 18:39:49.866957 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.526103 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 08 18:39:49.866957 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.528812 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 18:39:49.866957 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.529577 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 08 18:39:49.866957 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.529611 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-102.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 08 18:39:49.866957 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.530980 2575 topology_manager.go:138] "Creating topology manager with none policy" Jul 08 18:39:49.866957 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.530994 2575 container_manager_linux.go:309] "Creating device plugin manager" Jul 08 18:39:49.866957 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.531003 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 18:39:49.866957 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.532242 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 18:39:49.866957 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.533531 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 08 18:39:49.866957 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.536628 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 08 18:39:49.866957 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.536760 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 08 18:39:49.866957 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.536789 2575 kubelet.go:486] "Attempting to sync node with API server" Jul 08 18:39:49.866957 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.536804 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 08 18:39:49.866957 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.536817 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 08 18:39:49.866957 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.536827 2575 kubelet.go:398] "Adding apiserver pod source" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.536836 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.537246 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.537270 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.540147 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.541984 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.542012 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.543715 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.543734 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.543739 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.543744 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.543753 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.543759 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.543766 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.543806 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.543812 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.543817 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.543856 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.543861 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.545598 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.545613 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.548813 2575 server.go:1268] "Started kubelet" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.549205 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.550313 2575 server.go:310] "Adding debug handlers to kubelet server" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.551124 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.551162 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.552157 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.556875 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wz2ht" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.558005 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:49.613683 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-102.ec2.internal\" not found" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.613871 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.615730 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wz2ht" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.615981 2575 factory.go:55] Registering systemd factory Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.616022 2575 factory.go:223] Registration of the systemd container factory successfully Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.616397 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.616483 2575 factory.go:153] Registering CRI-O factory Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.616515 2575 factory.go:223] Registration of the crio container factory successfully Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.616569 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.616587 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:49.616491 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.616707 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.616762 2575 reconstruct.go:97] "Volume reconstruction finished" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.616821 2575 factory.go:103] Registering Raw factory Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.617507 2575 reconciler.go:29] "Reconciler: start to sync state" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.617523 2575 manager.go:1196] Started watching for new ooms in manager Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.617971 2575 manager.go:349] Starting recovery of all containers Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:49.620160 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-102.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 08 18:39:49.867807 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:49.620162 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.620172 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-102.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:49.620186 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.639191 2575 manager.go:354] Recovery completed Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:49.640121 2575 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-102.ec2.internal\" not found" node="ip-10-0-131-102.ec2.internal" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.651404 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.653854 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-102.ec2.internal" event="NodeHasSufficientMemory" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.653896 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-102.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.653912 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-102.ec2.internal" event="NodeHasSufficientPID" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.654492 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.654503 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.654523 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.654549 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.655359 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-102.ec2.internal" event="NodeHasSufficientMemory" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.655394 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-102.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.655406 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-102.ec2.internal" event="NodeHasSufficientPID" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.655851 2575 policy_none.go:49] "None policy: Start" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.655867 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.655878 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.661992 2575 policy_none.go:47] "Start" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:49.714522 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-102.ec2.internal\" not found" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.734677 2575 manager.go:337] "Starting Device Plugin manager" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:49.734730 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.734746 2575 server.go:87] "Starting device plugin registration server" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.735084 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.735119 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.735202 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.735349 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.735360 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:49.736419 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:49.736464 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-102.ec2.internal\" not found" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.772085 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.773455 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.773519 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.773553 2575 kubelet.go:2448] "Starting kubelet main sync loop" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:49.773614 2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.776265 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.835998 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.837160 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-102.ec2.internal" event="NodeHasSufficientMemory" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.837201 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-102.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.837217 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-102.ec2.internal" event="NodeHasSufficientPID" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.837249 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-102.ec2.internal" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.847209 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-102.ec2.internal" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:49.847230 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-102.ec2.internal\": node \"ip-10-0-131-102.ec2.internal\" not found" Jul 08 18:39:49.869759 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:49.868582 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-102.ec2.internal\" not found" Jul 08 18:39:49.875024 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.874998 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-102.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-102.ec2.internal"] Jul 08 18:39:49.875118 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.875086 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 18:39:49.876603 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.876587 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-102.ec2.internal" event="NodeHasSufficientMemory" Jul 08 18:39:49.876677 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.876614 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-102.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 18:39:49.876677 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.876624 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-102.ec2.internal" event="NodeHasSufficientPID" Jul 08 18:39:49.877826 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.877813 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 18:39:49.877945 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.877924 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-102.ec2.internal" Jul 08 18:39:49.877998 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.877968 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 18:39:49.878553 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.878534 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-102.ec2.internal" event="NodeHasSufficientMemory" Jul 08 18:39:49.878668 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.878563 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-102.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 18:39:49.878668 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.878574 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-102.ec2.internal" event="NodeHasSufficientPID" Jul 08 18:39:49.878668 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.878589 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-102.ec2.internal" event="NodeHasSufficientMemory" Jul 08 18:39:49.878668 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.878612 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-102.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 18:39:49.878668 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.878626 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-102.ec2.internal" event="NodeHasSufficientPID" Jul 08 18:39:49.879692 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.879676 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-102.ec2.internal" Jul 08 18:39:49.879795 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.879704 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 18:39:49.880352 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.880305 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-102.ec2.internal" event="NodeHasSufficientMemory" Jul 08 18:39:49.880464 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.880358 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-102.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 18:39:49.880464 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.880374 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-102.ec2.internal" event="NodeHasSufficientPID" Jul 08 18:39:49.887754 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:49.887732 2575 helpers.go:245] readString: Failed to read "/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7a407a9bbccda77188a4b8cd9fd490.slice/cpu.weight": read /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7a407a9bbccda77188a4b8cd9fd490.slice/cpu.weight: no such device Jul 08 18:39:49.912333 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:49.912296 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-102.ec2.internal\" not found" node="ip-10-0-131-102.ec2.internal" Jul 08 18:39:49.919131 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.919108 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/da7a407a9bbccda77188a4b8cd9fd490-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-102.ec2.internal\" (UID: \"da7a407a9bbccda77188a4b8cd9fd490\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-102.ec2.internal" Jul 08 18:39:49.919214 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.919140 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da7a407a9bbccda77188a4b8cd9fd490-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-102.ec2.internal\" (UID: \"da7a407a9bbccda77188a4b8cd9fd490\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-102.ec2.internal" Jul 08 18:39:49.919214 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:49.919150 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-102.ec2.internal\" not found" node="ip-10-0-131-102.ec2.internal" Jul 08 18:39:49.919214 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:49.919156 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a4bb240418508e65be90c2a28bf12044-config\") pod \"kube-apiserver-proxy-ip-10-0-131-102.ec2.internal\" (UID: \"a4bb240418508e65be90c2a28bf12044\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-102.ec2.internal" Jul 08 18:39:49.969108 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:49.969029 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-102.ec2.internal\" not found" Jul 08 18:39:50.019422 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:50.019390 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da7a407a9bbccda77188a4b8cd9fd490-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-102.ec2.internal\" (UID: \"da7a407a9bbccda77188a4b8cd9fd490\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-102.ec2.internal" Jul 08 18:39:50.019528 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:50.019430 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a4bb240418508e65be90c2a28bf12044-config\") pod \"kube-apiserver-proxy-ip-10-0-131-102.ec2.internal\" (UID: \"a4bb240418508e65be90c2a28bf12044\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-102.ec2.internal" Jul 08 18:39:50.019528 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:50.019450 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/da7a407a9bbccda77188a4b8cd9fd490-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-102.ec2.internal\" (UID: \"da7a407a9bbccda77188a4b8cd9fd490\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-102.ec2.internal" Jul 08 18:39:50.019528 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:50.019497 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/da7a407a9bbccda77188a4b8cd9fd490-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-102.ec2.internal\" (UID: \"da7a407a9bbccda77188a4b8cd9fd490\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-102.ec2.internal" Jul 08 18:39:50.019528 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:50.019507 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/da7a407a9bbccda77188a4b8cd9fd490-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-102.ec2.internal\" (UID: \"da7a407a9bbccda77188a4b8cd9fd490\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-102.ec2.internal" Jul 08 18:39:50.019666 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:50.019527 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a4bb240418508e65be90c2a28bf12044-config\") pod \"kube-apiserver-proxy-ip-10-0-131-102.ec2.internal\" (UID: \"a4bb240418508e65be90c2a28bf12044\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-102.ec2.internal" Jul 08 18:39:50.071564 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:50.071532 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-102.ec2.internal\" not found" Jul 08 18:39:50.175918 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:50.175879 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-102.ec2.internal\" not found" Jul 08 18:39:50.216100 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:50.216078 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-102.ec2.internal" Jul 08 18:39:50.223503 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:50.223487 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-102.ec2.internal" Jul 08 18:39:50.276356 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:50.276327 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-102.ec2.internal\" not found" Jul 08 18:39:50.380582 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:50.380531 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-102.ec2.internal\" not found" Jul 08 18:39:50.481348 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:50.481255 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-102.ec2.internal\" not found" Jul 08 18:39:50.495014 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:50.494987 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 08 18:39:50.495196 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:50.495151 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 08 18:39:50.578084 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:50.578055 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 18:39:50.578236 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:50.578161 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 18:39:50.606151 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:50.606126 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 18:39:50.614190 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:50.614169 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 08 18:39:50.614267 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:50.614191 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-102.ec2.internal" Jul 08 18:39:50.618737 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:50.618711 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 18:34:50 +0000 UTC" deadline="2028-04-01 19:49:23.387856558 +0000 UTC" Jul 08 18:39:50.618737 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:50.618736 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15193h9m32.76912308s" Jul 08 18:39:50.623963 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:50.623946 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 18:39:50.624986 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:50.624972 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-102.ec2.internal" Jul 08 18:39:50.632020 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:50.632003 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 18:39:50.640119 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:50.640101 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 18:39:50.654787 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:50.654766 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-scvhh" Jul 08 18:39:50.664526 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:50.664506 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-scvhh" Jul 08 18:39:50.768359 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:50.768322 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7a407a9bbccda77188a4b8cd9fd490.slice/crio-4d9793278c74fcd66f18c566fbaac1fac15f1d867f5187ea18f4fc62ac24dd79 WatchSource:0}: Error finding container 4d9793278c74fcd66f18c566fbaac1fac15f1d867f5187ea18f4fc62ac24dd79: Status 404 returned error can't find the container with id 4d9793278c74fcd66f18c566fbaac1fac15f1d867f5187ea18f4fc62ac24dd79 Jul 08 18:39:50.769022 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:50.768997 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4bb240418508e65be90c2a28bf12044.slice/crio-1ec342d77b48daf7e3289d526d5c7a2796c04d52c7c529b16bebdb230885fec1 WatchSource:0}: Error finding container 1ec342d77b48daf7e3289d526d5c7a2796c04d52c7c529b16bebdb230885fec1: Status 404 returned error can't find the container with id 1ec342d77b48daf7e3289d526d5c7a2796c04d52c7c529b16bebdb230885fec1 Jul 08 18:39:50.774826 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:50.774808 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 18:39:50.776843 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:50.776793 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-102.ec2.internal" event={"ID":"a4bb240418508e65be90c2a28bf12044","Type":"ContainerStarted","Data":"1ec342d77b48daf7e3289d526d5c7a2796c04d52c7c529b16bebdb230885fec1"} Jul 08 18:39:50.777845 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:50.777821 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-102.ec2.internal" event={"ID":"da7a407a9bbccda77188a4b8cd9fd490","Type":"ContainerStarted","Data":"4d9793278c74fcd66f18c566fbaac1fac15f1d867f5187ea18f4fc62ac24dd79"} Jul 08 18:39:51.537842 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.537799 2575 apiserver.go:52] "Watching apiserver" Jul 08 18:39:51.545980 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.545953 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 08 18:39:51.550105 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.549382 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-7gjd9","openshift-image-registry/node-ca-jj7hl","openshift-multus/network-metrics-daemon-252x5","openshift-ovn-kubernetes/ovnkube-node-jrzrj","kube-system/kube-apiserver-proxy-ip-10-0-131-102.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-102.ec2.internal","openshift-multus/multus-additional-cni-plugins-grt82","openshift-multus/multus-rsw44","openshift-network-diagnostics/network-check-target-pwffr","openshift-network-operator/iptables-alerter-p45pj","kube-system/global-pull-secret-syncer-szlwf","kube-system/konnectivity-agent-v7qvx","openshift-cluster-node-tuning-operator/tuned-mk6gn"] Jul 08 18:39:51.551219 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.551192 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:39:51.551387 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:51.551295 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwffr" podUID="ff43f31c-c779-406d-8352-4590967387a5" Jul 08 18:39:51.552528 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.552503 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p45pj" Jul 08 18:39:51.553763 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.553740 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:39:51.553869 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:51.553825 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-252x5" podUID="ee39c730-7ca1-4260-afea-a884940c00fc" Jul 08 18:39:51.555362 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.555341 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.555872 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.555819 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 08 18:39:51.555961 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.555913 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 08 18:39:51.556024 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.555959 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 08 18:39:51.556024 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.555977 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kd6mz\"" Jul 08 18:39:51.558336 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.557469 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.558476 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.558343 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vgr5m\"" Jul 08 18:39:51.558829 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.558806 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 08 18:39:51.559063 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.559028 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 08 18:39:51.559209 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.559167 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 08 18:39:51.559425 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.559406 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 08 18:39:51.559595 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.559574 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 08 18:39:51.559747 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.559527 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 08 18:39:51.560324 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.560302 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 08 18:39:51.560664 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.560640 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" Jul 08 18:39:51.560762 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.560703 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 08 18:39:51.560942 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.560640 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7f9jt\"" Jul 08 18:39:51.562395 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.562374 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.563649 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.563629 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 08 18:39:51.563759 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.563637 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 08 18:39:51.563844 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.563637 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mp6qk\"" Jul 08 18:39:51.563904 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.563892 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 08 18:39:51.564101 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.564082 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v7qvx" Jul 08 18:39:51.565133 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.565108 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 08 18:39:51.565550 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.565525 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-qp58f\"" Jul 08 18:39:51.565764 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.565743 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 08 18:39:51.566674 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.565966 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jj7hl" Jul 08 18:39:51.566945 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.566921 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7gjd9" Jul 08 18:39:51.567979 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.567698 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 08 18:39:51.567979 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.567797 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 08 18:39:51.567979 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.567823 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hxv7t\"" Jul 08 18:39:51.567979 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.567917 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 08 18:39:51.567979 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.567938 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 08 18:39:51.568604 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.568417 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 08 18:39:51.568837 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.568699 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b7m75\"" Jul 08 18:39:51.568915 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.568902 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 08 18:39:51.569426 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.568993 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-grt82" Jul 08 18:39:51.569426 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.569092 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 08 18:39:51.569743 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.569576 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 08 18:39:51.570143 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.569950 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rvg4v\"" Jul 08 18:39:51.570315 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.570290 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:39:51.570409 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:51.570382 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlwf" podUID="610063ae-dbb1-4a58-bacd-06bf90518168" Jul 08 18:39:51.571403 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.571144 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 08 18:39:51.572059 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.571938 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 08 18:39:51.572736 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.572443 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dhdwn\"" Jul 08 18:39:51.573454 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.573430 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 08 18:39:51.603168 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.602437 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 18:39:51.618235 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.618137 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 08 18:39:51.631285 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.631253 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-os-release\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.631417 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.631395 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-host-run-k8s-cni-cncf-io\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.631480 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.631430 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-host-var-lib-cni-bin\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.631553 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.631519 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-host-run-multus-certs\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.631613 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.631567 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/06234e92-3ccc-4e0a-9250-bfe6bf99befd-host\") pod \"node-ca-jj7hl\" (UID: \"06234e92-3ccc-4e0a-9250-bfe6bf99befd\") " pod="openshift-image-registry/node-ca-jj7hl" Jul 08 18:39:51.631673 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.631621 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-host-slash\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.631673 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.631647 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-run-systemd\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.631776 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.631732 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-run-openvswitch\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.631776 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.631758 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/06234e92-3ccc-4e0a-9250-bfe6bf99befd-serviceca\") pod \"node-ca-jj7hl\" (UID: \"06234e92-3ccc-4e0a-9250-bfe6bf99befd\") " pod="openshift-image-registry/node-ca-jj7hl" Jul 08 18:39:51.631872 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.631825 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e7898c0-39bf-42c9-a65e-30409d5a5ea1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-grt82\" (UID: \"1e7898c0-39bf-42c9-a65e-30409d5a5ea1\") " pod="openshift-multus/multus-additional-cni-plugins-grt82" Jul 08 18:39:51.631925 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.631855 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgbxp\" (UniqueName: \"kubernetes.io/projected/1e7898c0-39bf-42c9-a65e-30409d5a5ea1-kube-api-access-dgbxp\") pod \"multus-additional-cni-plugins-grt82\" (UID: \"1e7898c0-39bf-42c9-a65e-30409d5a5ea1\") " pod="openshift-multus/multus-additional-cni-plugins-grt82" Jul 08 18:39:51.631925 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.631900 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-etc-kubernetes\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.632027 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.631926 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e7898c0-39bf-42c9-a65e-30409d5a5ea1-cni-binary-copy\") pod \"multus-additional-cni-plugins-grt82\" (UID: \"1e7898c0-39bf-42c9-a65e-30409d5a5ea1\") " pod="openshift-multus/multus-additional-cni-plugins-grt82" Jul 08 18:39:51.632027 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.631971 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/08b493c4-bcc2-43fc-a7b6-2f51996d5aae-iptables-alerter-script\") pod \"iptables-alerter-p45pj\" (UID: \"08b493c4-bcc2-43fc-a7b6-2f51996d5aae\") " pod="openshift-network-operator/iptables-alerter-p45pj" Jul 08 18:39:51.632027 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632010 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46c9d0e3-09c2-407c-afd5-7c04d7f5f683-socket-dir\") pod \"aws-ebs-csi-driver-node-h4b2v\" (UID: \"46c9d0e3-09c2-407c-afd5-7c04d7f5f683\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" Jul 08 18:39:51.632181 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632062 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcc6d89d-de5e-44a5-9fe2-02785d57a8e0-tmp-dir\") pod \"node-resolver-7gjd9\" (UID: \"bcc6d89d-de5e-44a5-9fe2-02785d57a8e0\") " pod="openshift-dns/node-resolver-7gjd9" Jul 08 18:39:51.632181 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632091 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-etc-kubernetes\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.632181 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632149 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1e7898c0-39bf-42c9-a65e-30409d5a5ea1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-grt82\" (UID: \"1e7898c0-39bf-42c9-a65e-30409d5a5ea1\") " pod="openshift-multus/multus-additional-cni-plugins-grt82" Jul 08 18:39:51.632325 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632178 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.632325 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632231 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46c9d0e3-09c2-407c-afd5-7c04d7f5f683-etc-selinux\") pod \"aws-ebs-csi-driver-node-h4b2v\" (UID: \"46c9d0e3-09c2-407c-afd5-7c04d7f5f683\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" Jul 08 18:39:51.632325 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632258 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-multus-conf-dir\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.632325 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632284 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-host-run-netns\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.632562 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632360 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bwlf\" (UniqueName: \"kubernetes.io/projected/a6f52c90-705e-42cf-8676-580f6ec9544e-kube-api-access-6bwlf\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.632562 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632403 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-etc-openvswitch\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.632562 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632446 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-multus-socket-dir-parent\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.632562 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632487 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-etc-sysconfig\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.632562 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632534 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46c9d0e3-09c2-407c-afd5-7c04d7f5f683-device-dir\") pod \"aws-ebs-csi-driver-node-h4b2v\" (UID: \"46c9d0e3-09c2-407c-afd5-7c04d7f5f683\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" Jul 08 18:39:51.632816 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632563 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46c9d0e3-09c2-407c-afd5-7c04d7f5f683-sys-fs\") pod \"aws-ebs-csi-driver-node-h4b2v\" (UID: \"46c9d0e3-09c2-407c-afd5-7c04d7f5f683\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" Jul 08 18:39:51.632816 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632588 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-host-run-netns\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.632816 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632614 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-etc-sysctl-d\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.632816 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632639 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-sys\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.632816 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632662 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6c8c8ccc-4761-4a63-9361-0b4486ed8118-ovnkube-config\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.632816 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632685 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-system-cni-dir\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.632816 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632719 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-host-cni-bin\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.632816 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632749 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d61025be-679c-46f7-9a55-5049757d2d53-agent-certs\") pod \"konnectivity-agent-v7qvx\" (UID: \"d61025be-679c-46f7-9a55-5049757d2d53\") " pod="kube-system/konnectivity-agent-v7qvx" Jul 08 18:39:51.632816 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632790 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d61025be-679c-46f7-9a55-5049757d2d53-konnectivity-ca\") pod \"konnectivity-agent-v7qvx\" (UID: \"d61025be-679c-46f7-9a55-5049757d2d53\") " pod="kube-system/konnectivity-agent-v7qvx" Jul 08 18:39:51.632816 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632815 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-etc-modprobe-d\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.633307 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632839 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bcc6d89d-de5e-44a5-9fe2-02785d57a8e0-hosts-file\") pod \"node-resolver-7gjd9\" (UID: \"bcc6d89d-de5e-44a5-9fe2-02785d57a8e0\") " pod="openshift-dns/node-resolver-7gjd9" Jul 08 18:39:51.633307 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632862 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e7898c0-39bf-42c9-a65e-30409d5a5ea1-system-cni-dir\") pod \"multus-additional-cni-plugins-grt82\" (UID: \"1e7898c0-39bf-42c9-a65e-30409d5a5ea1\") " pod="openshift-multus/multus-additional-cni-plugins-grt82" Jul 08 18:39:51.633307 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632887 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e7898c0-39bf-42c9-a65e-30409d5a5ea1-cnibin\") pod \"multus-additional-cni-plugins-grt82\" (UID: \"1e7898c0-39bf-42c9-a65e-30409d5a5ea1\") " pod="openshift-multus/multus-additional-cni-plugins-grt82" Jul 08 18:39:51.633307 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632911 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/610063ae-dbb1-4a58-bacd-06bf90518168-dbus\") pod \"global-pull-secret-syncer-szlwf\" (UID: \"610063ae-dbb1-4a58-bacd-06bf90518168\") " pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:39:51.633307 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632933 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-run-ovn\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.633307 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632960 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46c9d0e3-09c2-407c-afd5-7c04d7f5f683-registration-dir\") pod \"aws-ebs-csi-driver-node-h4b2v\" (UID: \"46c9d0e3-09c2-407c-afd5-7c04d7f5f683\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" Jul 08 18:39:51.633307 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.632984 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-host-var-lib-cni-multus\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.633307 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633009 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-host\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.633307 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633031 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6f52c90-705e-42cf-8676-580f6ec9544e-tmp\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.633307 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633055 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-systemd-units\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.633307 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633079 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-log-socket\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.633307 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633103 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46c9d0e3-09c2-407c-afd5-7c04d7f5f683-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h4b2v\" (UID: \"46c9d0e3-09c2-407c-afd5-7c04d7f5f683\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" Jul 08 18:39:51.633307 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633128 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-multus-daemon-config\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.633307 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633153 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx7sb\" (UniqueName: \"kubernetes.io/projected/bcc6d89d-de5e-44a5-9fe2-02785d57a8e0-kube-api-access-bx7sb\") pod \"node-resolver-7gjd9\" (UID: \"bcc6d89d-de5e-44a5-9fe2-02785d57a8e0\") " pod="openshift-dns/node-resolver-7gjd9" Jul 08 18:39:51.633307 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633183 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm796\" (UniqueName: \"kubernetes.io/projected/06234e92-3ccc-4e0a-9250-bfe6bf99befd-kube-api-access-sm796\") pod \"node-ca-jj7hl\" (UID: \"06234e92-3ccc-4e0a-9250-bfe6bf99befd\") " pod="openshift-image-registry/node-ca-jj7hl" Jul 08 18:39:51.633307 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633231 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zwd9\" (UniqueName: \"kubernetes.io/projected/ff43f31c-c779-406d-8352-4590967387a5-kube-api-access-2zwd9\") pod \"network-check-target-pwffr\" (UID: \"ff43f31c-c779-406d-8352-4590967387a5\") " pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:39:51.634105 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633269 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-run\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.634105 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633292 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a6f52c90-705e-42cf-8676-580f6ec9544e-etc-tuned\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.634105 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633320 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1e7898c0-39bf-42c9-a65e-30409d5a5ea1-os-release\") pod \"multus-additional-cni-plugins-grt82\" (UID: \"1e7898c0-39bf-42c9-a65e-30409d5a5ea1\") " pod="openshift-multus/multus-additional-cni-plugins-grt82" Jul 08 18:39:51.634105 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633486 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-cni-binary-copy\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.634105 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633526 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-node-log\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.634105 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633558 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-etc-systemd\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.634105 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633583 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-lib-modules\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.634105 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633608 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e7898c0-39bf-42c9-a65e-30409d5a5ea1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-grt82\" (UID: \"1e7898c0-39bf-42c9-a65e-30409d5a5ea1\") " pod="openshift-multus/multus-additional-cni-plugins-grt82" Jul 08 18:39:51.634105 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633638 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08b493c4-bcc2-43fc-a7b6-2f51996d5aae-host-slash\") pod \"iptables-alerter-p45pj\" (UID: \"08b493c4-bcc2-43fc-a7b6-2f51996d5aae\") " pod="openshift-network-operator/iptables-alerter-p45pj" Jul 08 18:39:51.634105 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633669 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-var-lib-openvswitch\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.634105 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633702 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-multus-cni-dir\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.634105 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633731 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-host-var-lib-kubelet\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.634105 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633762 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-etc-sysctl-conf\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.634105 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633804 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-var-lib-kubelet\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.634105 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633829 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee39c730-7ca1-4260-afea-a884940c00fc-metrics-certs\") pod \"network-metrics-daemon-252x5\" (UID: \"ee39c730-7ca1-4260-afea-a884940c00fc\") " pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:39:51.634105 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633851 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-host-kubelet\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.634999 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633874 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-host-run-ovn-kubernetes\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.634999 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633917 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6c8c8ccc-4761-4a63-9361-0b4486ed8118-env-overrides\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.634999 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633948 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-cnibin\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.634999 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633971 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-hostroot\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.634999 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.633996 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/610063ae-dbb1-4a58-bacd-06bf90518168-kubelet-config\") pod \"global-pull-secret-syncer-szlwf\" (UID: \"610063ae-dbb1-4a58-bacd-06bf90518168\") " pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:39:51.634999 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.634022 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/610063ae-dbb1-4a58-bacd-06bf90518168-original-pull-secret\") pod \"global-pull-secret-syncer-szlwf\" (UID: \"610063ae-dbb1-4a58-bacd-06bf90518168\") " pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:39:51.634999 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.634054 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xpcb\" (UniqueName: \"kubernetes.io/projected/08b493c4-bcc2-43fc-a7b6-2f51996d5aae-kube-api-access-6xpcb\") pod \"iptables-alerter-p45pj\" (UID: \"08b493c4-bcc2-43fc-a7b6-2f51996d5aae\") " pod="openshift-network-operator/iptables-alerter-p45pj" Jul 08 18:39:51.634999 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.634090 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c8c8ccc-4761-4a63-9361-0b4486ed8118-ovn-node-metrics-cert\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.634999 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.634120 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6c8c8ccc-4761-4a63-9361-0b4486ed8118-ovnkube-script-lib\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.634999 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.634145 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjnw4\" (UniqueName: \"kubernetes.io/projected/46c9d0e3-09c2-407c-afd5-7c04d7f5f683-kube-api-access-fjnw4\") pod \"aws-ebs-csi-driver-node-h4b2v\" (UID: \"46c9d0e3-09c2-407c-afd5-7c04d7f5f683\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" Jul 08 18:39:51.634999 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.634170 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgz42\" (UniqueName: \"kubernetes.io/projected/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-kube-api-access-rgz42\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.634999 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.634195 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrmf7\" (UniqueName: \"kubernetes.io/projected/ee39c730-7ca1-4260-afea-a884940c00fc-kube-api-access-jrmf7\") pod \"network-metrics-daemon-252x5\" (UID: \"ee39c730-7ca1-4260-afea-a884940c00fc\") " pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:39:51.634999 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.634217 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-host-cni-netd\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.634999 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.634241 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2967x\" (UniqueName: \"kubernetes.io/projected/6c8c8ccc-4761-4a63-9361-0b4486ed8118-kube-api-access-2967x\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.667504 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.667441 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 18:34:51 +0000 UTC" deadline="2028-04-01 18:11:52.326603334 +0000 UTC" Jul 08 18:39:51.667504 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.667492 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15191h32m0.65911544s" Jul 08 18:39:51.734965 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.734931 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.734965 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.734966 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46c9d0e3-09c2-407c-afd5-7c04d7f5f683-etc-selinux\") pod \"aws-ebs-csi-driver-node-h4b2v\" (UID: \"46c9d0e3-09c2-407c-afd5-7c04d7f5f683\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" Jul 08 18:39:51.735203 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.734999 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-multus-conf-dir\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.735203 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735025 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-host-run-netns\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.735203 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735050 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bwlf\" (UniqueName: \"kubernetes.io/projected/a6f52c90-705e-42cf-8676-580f6ec9544e-kube-api-access-6bwlf\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.735203 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735074 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-etc-openvswitch\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.735203 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735110 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-multus-socket-dir-parent\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.735203 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735121 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-multus-conf-dir\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.735203 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735130 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-etc-sysconfig\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.735203 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735180 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46c9d0e3-09c2-407c-afd5-7c04d7f5f683-device-dir\") pod \"aws-ebs-csi-driver-node-h4b2v\" (UID: \"46c9d0e3-09c2-407c-afd5-7c04d7f5f683\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" Jul 08 18:39:51.735203 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735188 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-etc-sysconfig\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.735713 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735207 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46c9d0e3-09c2-407c-afd5-7c04d7f5f683-sys-fs\") pod \"aws-ebs-csi-driver-node-h4b2v\" (UID: \"46c9d0e3-09c2-407c-afd5-7c04d7f5f683\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" Jul 08 18:39:51.735713 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735221 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-host-run-netns\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.735713 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735234 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-host-run-netns\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.735713 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735279 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-etc-sysctl-d\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.735713 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735288 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-etc-openvswitch\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.735713 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735076 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.735713 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735304 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-sys\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.735713 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735330 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6c8c8ccc-4761-4a63-9361-0b4486ed8118-ovnkube-config\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.735713 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735364 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-multus-socket-dir-parent\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.735713 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735372 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-system-cni-dir\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.735713 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735408 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-host-cni-bin\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.735713 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735434 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d61025be-679c-46f7-9a55-5049757d2d53-agent-certs\") pod \"konnectivity-agent-v7qvx\" (UID: \"d61025be-679c-46f7-9a55-5049757d2d53\") " pod="kube-system/konnectivity-agent-v7qvx" Jul 08 18:39:51.735713 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735460 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d61025be-679c-46f7-9a55-5049757d2d53-konnectivity-ca\") pod \"konnectivity-agent-v7qvx\" (UID: \"d61025be-679c-46f7-9a55-5049757d2d53\") " pod="kube-system/konnectivity-agent-v7qvx" Jul 08 18:39:51.735713 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735485 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-etc-modprobe-d\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.735713 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735510 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bcc6d89d-de5e-44a5-9fe2-02785d57a8e0-hosts-file\") pod \"node-resolver-7gjd9\" (UID: \"bcc6d89d-de5e-44a5-9fe2-02785d57a8e0\") " pod="openshift-dns/node-resolver-7gjd9" Jul 08 18:39:51.735713 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735539 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e7898c0-39bf-42c9-a65e-30409d5a5ea1-system-cni-dir\") pod \"multus-additional-cni-plugins-grt82\" (UID: \"1e7898c0-39bf-42c9-a65e-30409d5a5ea1\") " pod="openshift-multus/multus-additional-cni-plugins-grt82" Jul 08 18:39:51.735713 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735550 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46c9d0e3-09c2-407c-afd5-7c04d7f5f683-device-dir\") pod \"aws-ebs-csi-driver-node-h4b2v\" (UID: \"46c9d0e3-09c2-407c-afd5-7c04d7f5f683\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" Jul 08 18:39:51.736655 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735564 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e7898c0-39bf-42c9-a65e-30409d5a5ea1-cnibin\") pod \"multus-additional-cni-plugins-grt82\" (UID: \"1e7898c0-39bf-42c9-a65e-30409d5a5ea1\") " pod="openshift-multus/multus-additional-cni-plugins-grt82" Jul 08 18:39:51.736655 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735588 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/610063ae-dbb1-4a58-bacd-06bf90518168-dbus\") pod \"global-pull-secret-syncer-szlwf\" (UID: \"610063ae-dbb1-4a58-bacd-06bf90518168\") " pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:39:51.736655 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735600 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46c9d0e3-09c2-407c-afd5-7c04d7f5f683-sys-fs\") pod \"aws-ebs-csi-driver-node-h4b2v\" (UID: \"46c9d0e3-09c2-407c-afd5-7c04d7f5f683\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" Jul 08 18:39:51.736655 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735614 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-run-ovn\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.736655 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735640 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46c9d0e3-09c2-407c-afd5-7c04d7f5f683-registration-dir\") pod \"aws-ebs-csi-driver-node-h4b2v\" (UID: \"46c9d0e3-09c2-407c-afd5-7c04d7f5f683\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" Jul 08 18:39:51.736655 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735667 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-host-var-lib-cni-multus\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.736655 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735691 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-host\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.736655 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735715 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6f52c90-705e-42cf-8676-580f6ec9544e-tmp\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.736655 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735740 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-systemd-units\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.736655 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735769 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-log-socket\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.736655 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735806 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46c9d0e3-09c2-407c-afd5-7c04d7f5f683-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h4b2v\" (UID: \"46c9d0e3-09c2-407c-afd5-7c04d7f5f683\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" Jul 08 18:39:51.736655 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735832 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-multus-daemon-config\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.736655 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735857 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx7sb\" (UniqueName: \"kubernetes.io/projected/bcc6d89d-de5e-44a5-9fe2-02785d57a8e0-kube-api-access-bx7sb\") pod \"node-resolver-7gjd9\" (UID: \"bcc6d89d-de5e-44a5-9fe2-02785d57a8e0\") " pod="openshift-dns/node-resolver-7gjd9" Jul 08 18:39:51.736655 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735885 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm796\" (UniqueName: \"kubernetes.io/projected/06234e92-3ccc-4e0a-9250-bfe6bf99befd-kube-api-access-sm796\") pod \"node-ca-jj7hl\" (UID: \"06234e92-3ccc-4e0a-9250-bfe6bf99befd\") " pod="openshift-image-registry/node-ca-jj7hl" Jul 08 18:39:51.736655 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735912 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zwd9\" (UniqueName: \"kubernetes.io/projected/ff43f31c-c779-406d-8352-4590967387a5-kube-api-access-2zwd9\") pod \"network-check-target-pwffr\" (UID: \"ff43f31c-c779-406d-8352-4590967387a5\") " pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:39:51.736655 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735940 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-run\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.736655 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735966 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a6f52c90-705e-42cf-8676-580f6ec9544e-etc-tuned\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.736655 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735968 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 08 18:39:51.737569 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736018 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1e7898c0-39bf-42c9-a65e-30409d5a5ea1-os-release\") pod \"multus-additional-cni-plugins-grt82\" (UID: \"1e7898c0-39bf-42c9-a65e-30409d5a5ea1\") " pod="openshift-multus/multus-additional-cni-plugins-grt82" Jul 08 18:39:51.737569 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736044 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-cni-binary-copy\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.737569 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736069 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-node-log\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.737569 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736096 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-etc-systemd\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.737569 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736127 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-lib-modules\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.737569 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736170 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e7898c0-39bf-42c9-a65e-30409d5a5ea1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-grt82\" (UID: \"1e7898c0-39bf-42c9-a65e-30409d5a5ea1\") " pod="openshift-multus/multus-additional-cni-plugins-grt82" Jul 08 18:39:51.737569 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736196 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08b493c4-bcc2-43fc-a7b6-2f51996d5aae-host-slash\") pod \"iptables-alerter-p45pj\" (UID: \"08b493c4-bcc2-43fc-a7b6-2f51996d5aae\") " pod="openshift-network-operator/iptables-alerter-p45pj" Jul 08 18:39:51.737569 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736223 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-var-lib-openvswitch\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.737569 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736248 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-multus-cni-dir\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.737569 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736274 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-host-var-lib-kubelet\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.737569 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736301 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-etc-sysctl-conf\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.737569 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736327 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-var-lib-kubelet\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.737569 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736354 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee39c730-7ca1-4260-afea-a884940c00fc-metrics-certs\") pod \"network-metrics-daemon-252x5\" (UID: \"ee39c730-7ca1-4260-afea-a884940c00fc\") " pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:39:51.737569 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736403 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-host-kubelet\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.737569 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736443 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-host-run-ovn-kubernetes\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.737569 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736469 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6c8c8ccc-4761-4a63-9361-0b4486ed8118-env-overrides\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.737569 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736495 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-cnibin\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.738485 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736521 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-hostroot\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.738485 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736549 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/610063ae-dbb1-4a58-bacd-06bf90518168-kubelet-config\") pod \"global-pull-secret-syncer-szlwf\" (UID: \"610063ae-dbb1-4a58-bacd-06bf90518168\") " pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:39:51.738485 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736575 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/610063ae-dbb1-4a58-bacd-06bf90518168-original-pull-secret\") pod \"global-pull-secret-syncer-szlwf\" (UID: \"610063ae-dbb1-4a58-bacd-06bf90518168\") " pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:39:51.738485 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736605 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xpcb\" (UniqueName: \"kubernetes.io/projected/08b493c4-bcc2-43fc-a7b6-2f51996d5aae-kube-api-access-6xpcb\") pod \"iptables-alerter-p45pj\" (UID: \"08b493c4-bcc2-43fc-a7b6-2f51996d5aae\") " pod="openshift-network-operator/iptables-alerter-p45pj" Jul 08 18:39:51.738485 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736632 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c8c8ccc-4761-4a63-9361-0b4486ed8118-ovn-node-metrics-cert\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.738485 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736676 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6c8c8ccc-4761-4a63-9361-0b4486ed8118-ovnkube-script-lib\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.738485 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736707 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjnw4\" (UniqueName: \"kubernetes.io/projected/46c9d0e3-09c2-407c-afd5-7c04d7f5f683-kube-api-access-fjnw4\") pod \"aws-ebs-csi-driver-node-h4b2v\" (UID: \"46c9d0e3-09c2-407c-afd5-7c04d7f5f683\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" Jul 08 18:39:51.738485 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736735 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgz42\" (UniqueName: \"kubernetes.io/projected/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-kube-api-access-rgz42\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.738485 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736764 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrmf7\" (UniqueName: \"kubernetes.io/projected/ee39c730-7ca1-4260-afea-a884940c00fc-kube-api-access-jrmf7\") pod \"network-metrics-daemon-252x5\" (UID: \"ee39c730-7ca1-4260-afea-a884940c00fc\") " pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:39:51.738485 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736803 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-host-cni-netd\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.738485 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736831 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2967x\" (UniqueName: \"kubernetes.io/projected/6c8c8ccc-4761-4a63-9361-0b4486ed8118-kube-api-access-2967x\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.738485 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736868 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-os-release\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.738485 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736911 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-host-run-k8s-cni-cncf-io\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.738485 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736943 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-host-var-lib-cni-bin\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.738485 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736971 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-host-run-multus-certs\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.738485 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.736998 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/06234e92-3ccc-4e0a-9250-bfe6bf99befd-host\") pod \"node-ca-jj7hl\" (UID: \"06234e92-3ccc-4e0a-9250-bfe6bf99befd\") " pod="openshift-image-registry/node-ca-jj7hl" Jul 08 18:39:51.738485 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.737024 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-host-slash\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.739327 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.737052 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-run-systemd\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.739327 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.737080 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-run-openvswitch\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.739327 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.737107 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/06234e92-3ccc-4e0a-9250-bfe6bf99befd-serviceca\") pod \"node-ca-jj7hl\" (UID: \"06234e92-3ccc-4e0a-9250-bfe6bf99befd\") " pod="openshift-image-registry/node-ca-jj7hl" Jul 08 18:39:51.739327 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.737136 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e7898c0-39bf-42c9-a65e-30409d5a5ea1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-grt82\" (UID: \"1e7898c0-39bf-42c9-a65e-30409d5a5ea1\") " pod="openshift-multus/multus-additional-cni-plugins-grt82" Jul 08 18:39:51.739327 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.737168 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgbxp\" (UniqueName: \"kubernetes.io/projected/1e7898c0-39bf-42c9-a65e-30409d5a5ea1-kube-api-access-dgbxp\") pod \"multus-additional-cni-plugins-grt82\" (UID: \"1e7898c0-39bf-42c9-a65e-30409d5a5ea1\") " pod="openshift-multus/multus-additional-cni-plugins-grt82" Jul 08 18:39:51.739327 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.737204 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-etc-kubernetes\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.739327 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.737233 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e7898c0-39bf-42c9-a65e-30409d5a5ea1-cni-binary-copy\") pod \"multus-additional-cni-plugins-grt82\" (UID: \"1e7898c0-39bf-42c9-a65e-30409d5a5ea1\") " pod="openshift-multus/multus-additional-cni-plugins-grt82" Jul 08 18:39:51.739327 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.737262 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/08b493c4-bcc2-43fc-a7b6-2f51996d5aae-iptables-alerter-script\") pod \"iptables-alerter-p45pj\" (UID: \"08b493c4-bcc2-43fc-a7b6-2f51996d5aae\") " pod="openshift-network-operator/iptables-alerter-p45pj" Jul 08 18:39:51.739327 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.737293 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46c9d0e3-09c2-407c-afd5-7c04d7f5f683-socket-dir\") pod \"aws-ebs-csi-driver-node-h4b2v\" (UID: \"46c9d0e3-09c2-407c-afd5-7c04d7f5f683\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" Jul 08 18:39:51.739327 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.737324 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcc6d89d-de5e-44a5-9fe2-02785d57a8e0-tmp-dir\") pod \"node-resolver-7gjd9\" (UID: \"bcc6d89d-de5e-44a5-9fe2-02785d57a8e0\") " pod="openshift-dns/node-resolver-7gjd9" Jul 08 18:39:51.739327 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.737353 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-etc-kubernetes\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.739327 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.737402 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1e7898c0-39bf-42c9-a65e-30409d5a5ea1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-grt82\" (UID: \"1e7898c0-39bf-42c9-a65e-30409d5a5ea1\") " pod="openshift-multus/multus-additional-cni-plugins-grt82" Jul 08 18:39:51.739327 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.737591 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-var-lib-kubelet\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.739327 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.738088 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1e7898c0-39bf-42c9-a65e-30409d5a5ea1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-grt82\" (UID: \"1e7898c0-39bf-42c9-a65e-30409d5a5ea1\") " pod="openshift-multus/multus-additional-cni-plugins-grt82" Jul 08 18:39:51.739327 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:51.738238 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:39:51.739327 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.738291 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d61025be-679c-46f7-9a55-5049757d2d53-konnectivity-ca\") pod \"konnectivity-agent-v7qvx\" (UID: \"d61025be-679c-46f7-9a55-5049757d2d53\") " pod="kube-system/konnectivity-agent-v7qvx" Jul 08 18:39:51.743523 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:51.738303 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee39c730-7ca1-4260-afea-a884940c00fc-metrics-certs podName:ee39c730-7ca1-4260-afea-a884940c00fc nodeName:}" failed. No retries permitted until 2026-07-08 18:39:52.238280911 +0000 UTC m=+3.075608626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee39c730-7ca1-4260-afea-a884940c00fc-metrics-certs") pod "network-metrics-daemon-252x5" (UID: "ee39c730-7ca1-4260-afea-a884940c00fc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:39:51.743523 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.738358 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-host-kubelet\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.743523 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.738397 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bcc6d89d-de5e-44a5-9fe2-02785d57a8e0-hosts-file\") pod \"node-resolver-7gjd9\" (UID: \"bcc6d89d-de5e-44a5-9fe2-02785d57a8e0\") " pod="openshift-dns/node-resolver-7gjd9" Jul 08 18:39:51.743523 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735510 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46c9d0e3-09c2-407c-afd5-7c04d7f5f683-etc-selinux\") pod \"aws-ebs-csi-driver-node-h4b2v\" (UID: \"46c9d0e3-09c2-407c-afd5-7c04d7f5f683\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" Jul 08 18:39:51.743523 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.738463 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e7898c0-39bf-42c9-a65e-30409d5a5ea1-system-cni-dir\") pod \"multus-additional-cni-plugins-grt82\" (UID: \"1e7898c0-39bf-42c9-a65e-30409d5a5ea1\") " pod="openshift-multus/multus-additional-cni-plugins-grt82" Jul 08 18:39:51.743523 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.738473 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-etc-modprobe-d\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.743523 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.738508 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e7898c0-39bf-42c9-a65e-30409d5a5ea1-cnibin\") pod \"multus-additional-cni-plugins-grt82\" (UID: \"1e7898c0-39bf-42c9-a65e-30409d5a5ea1\") " pod="openshift-multus/multus-additional-cni-plugins-grt82" Jul 08 18:39:51.743523 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.738522 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-host-run-ovn-kubernetes\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.743523 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.738625 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/610063ae-dbb1-4a58-bacd-06bf90518168-dbus\") pod \"global-pull-secret-syncer-szlwf\" (UID: \"610063ae-dbb1-4a58-bacd-06bf90518168\") " pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:39:51.743523 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.738682 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-run-ovn\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.743523 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.738738 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46c9d0e3-09c2-407c-afd5-7c04d7f5f683-registration-dir\") pod \"aws-ebs-csi-driver-node-h4b2v\" (UID: \"46c9d0e3-09c2-407c-afd5-7c04d7f5f683\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" Jul 08 18:39:51.743523 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.738780 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-host-var-lib-cni-multus\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.743523 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.738823 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-host\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.743523 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.739032 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6c8c8ccc-4761-4a63-9361-0b4486ed8118-env-overrides\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.743523 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.739108 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-cnibin\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.743523 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.739157 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-hostroot\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.743523 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.739213 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/610063ae-dbb1-4a58-bacd-06bf90518168-kubelet-config\") pod \"global-pull-secret-syncer-szlwf\" (UID: \"610063ae-dbb1-4a58-bacd-06bf90518168\") " pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:39:51.744438 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:51.739293 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 18:39:51.744438 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:51.739342 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/610063ae-dbb1-4a58-bacd-06bf90518168-original-pull-secret podName:610063ae-dbb1-4a58-bacd-06bf90518168 nodeName:}" failed. No retries permitted until 2026-07-08 18:39:52.239325206 +0000 UTC m=+3.076652910 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/610063ae-dbb1-4a58-bacd-06bf90518168-original-pull-secret") pod "global-pull-secret-syncer-szlwf" (UID: "610063ae-dbb1-4a58-bacd-06bf90518168") : object "kube-system"/"original-pull-secret" not registered Jul 08 18:39:51.744438 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.739609 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-systemd-units\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.744438 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.739658 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-log-socket\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.744438 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.739702 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46c9d0e3-09c2-407c-afd5-7c04d7f5f683-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h4b2v\" (UID: \"46c9d0e3-09c2-407c-afd5-7c04d7f5f683\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" Jul 08 18:39:51.744438 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.739844 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-run-systemd\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.744438 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.739893 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-run-openvswitch\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.744438 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.740218 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-multus-daemon-config\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.744438 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.740323 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/06234e92-3ccc-4e0a-9250-bfe6bf99befd-serviceca\") pod \"node-ca-jj7hl\" (UID: \"06234e92-3ccc-4e0a-9250-bfe6bf99befd\") " pod="openshift-image-registry/node-ca-jj7hl" Jul 08 18:39:51.744438 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.740511 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-etc-systemd\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.744438 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.740784 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-lib-modules\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.744438 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.740857 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1e7898c0-39bf-42c9-a65e-30409d5a5ea1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-grt82\" (UID: \"1e7898c0-39bf-42c9-a65e-30409d5a5ea1\") " pod="openshift-multus/multus-additional-cni-plugins-grt82" Jul 08 18:39:51.744438 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.740934 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e7898c0-39bf-42c9-a65e-30409d5a5ea1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-grt82\" (UID: \"1e7898c0-39bf-42c9-a65e-30409d5a5ea1\") " pod="openshift-multus/multus-additional-cni-plugins-grt82" Jul 08 18:39:51.744438 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.740958 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6c8c8ccc-4761-4a63-9361-0b4486ed8118-ovnkube-config\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.744438 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.740989 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-run\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.744438 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.741044 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08b493c4-bcc2-43fc-a7b6-2f51996d5aae-host-slash\") pod \"iptables-alerter-p45pj\" (UID: \"08b493c4-bcc2-43fc-a7b6-2f51996d5aae\") " pod="openshift-network-operator/iptables-alerter-p45pj" Jul 08 18:39:51.744438 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.741087 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-var-lib-openvswitch\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.745322 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.741142 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-multus-cni-dir\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.745322 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.741185 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-host-var-lib-kubelet\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.745322 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.741169 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-etc-sysctl-d\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.745322 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.741207 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-etc-kubernetes\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.745322 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.741299 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-etc-sysctl-conf\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.745322 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.741305 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-host-cni-bin\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.745322 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.741347 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-sys\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.745322 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.741396 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-system-cni-dir\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.745322 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.741443 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-host-var-lib-cni-bin\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.745322 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.741479 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-host-run-multus-certs\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.745322 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.741518 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-os-release\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.745322 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.741561 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-host-run-k8s-cni-cncf-io\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.745322 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.741607 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-node-log\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.745322 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.741650 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/06234e92-3ccc-4e0a-9250-bfe6bf99befd-host\") pod \"node-ca-jj7hl\" (UID: \"06234e92-3ccc-4e0a-9250-bfe6bf99befd\") " pod="openshift-image-registry/node-ca-jj7hl" Jul 08 18:39:51.745322 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.741704 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1e7898c0-39bf-42c9-a65e-30409d5a5ea1-os-release\") pod \"multus-additional-cni-plugins-grt82\" (UID: \"1e7898c0-39bf-42c9-a65e-30409d5a5ea1\") " pod="openshift-multus/multus-additional-cni-plugins-grt82" Jul 08 18:39:51.745322 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.735409 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-host-run-netns\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.745322 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.741769 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e7898c0-39bf-42c9-a65e-30409d5a5ea1-cni-binary-copy\") pod \"multus-additional-cni-plugins-grt82\" (UID: \"1e7898c0-39bf-42c9-a65e-30409d5a5ea1\") " pod="openshift-multus/multus-additional-cni-plugins-grt82" Jul 08 18:39:51.745322 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.741848 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6f52c90-705e-42cf-8676-580f6ec9544e-etc-kubernetes\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.746303 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.742005 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bcc6d89d-de5e-44a5-9fe2-02785d57a8e0-tmp-dir\") pod \"node-resolver-7gjd9\" (UID: \"bcc6d89d-de5e-44a5-9fe2-02785d57a8e0\") " pod="openshift-dns/node-resolver-7gjd9" Jul 08 18:39:51.746303 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.742564 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-host-cni-netd\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.746303 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.742619 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c8c8ccc-4761-4a63-9361-0b4486ed8118-host-slash\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.746303 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.742635 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/08b493c4-bcc2-43fc-a7b6-2f51996d5aae-iptables-alerter-script\") pod \"iptables-alerter-p45pj\" (UID: \"08b493c4-bcc2-43fc-a7b6-2f51996d5aae\") " pod="openshift-network-operator/iptables-alerter-p45pj" Jul 08 18:39:51.746303 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.742734 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46c9d0e3-09c2-407c-afd5-7c04d7f5f683-socket-dir\") pod \"aws-ebs-csi-driver-node-h4b2v\" (UID: \"46c9d0e3-09c2-407c-afd5-7c04d7f5f683\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" Jul 08 18:39:51.746303 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.742887 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a6f52c90-705e-42cf-8676-580f6ec9544e-tmp\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.746303 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.743180 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d61025be-679c-46f7-9a55-5049757d2d53-agent-certs\") pod \"konnectivity-agent-v7qvx\" (UID: \"d61025be-679c-46f7-9a55-5049757d2d53\") " pod="kube-system/konnectivity-agent-v7qvx" Jul 08 18:39:51.746303 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.743691 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6c8c8ccc-4761-4a63-9361-0b4486ed8118-ovnkube-script-lib\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.746303 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.744166 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a6f52c90-705e-42cf-8676-580f6ec9544e-etc-tuned\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.746303 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.744194 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-cni-binary-copy\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.746303 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.744166 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c8c8ccc-4761-4a63-9361-0b4486ed8118-ovn-node-metrics-cert\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.750652 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.750632 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx7sb\" (UniqueName: \"kubernetes.io/projected/bcc6d89d-de5e-44a5-9fe2-02785d57a8e0-kube-api-access-bx7sb\") pod \"node-resolver-7gjd9\" (UID: \"bcc6d89d-de5e-44a5-9fe2-02785d57a8e0\") " pod="openshift-dns/node-resolver-7gjd9" Jul 08 18:39:51.753222 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:51.752461 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 18:39:51.753222 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:51.752487 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 18:39:51.753222 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:51.752502 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2zwd9 for pod openshift-network-diagnostics/network-check-target-pwffr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:39:51.753222 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:51.752559 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff43f31c-c779-406d-8352-4590967387a5-kube-api-access-2zwd9 podName:ff43f31c-c779-406d-8352-4590967387a5 nodeName:}" failed. No retries permitted until 2026-07-08 18:39:52.252540988 +0000 UTC m=+3.089868694 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2zwd9" (UniqueName: "kubernetes.io/projected/ff43f31c-c779-406d-8352-4590967387a5-kube-api-access-2zwd9") pod "network-check-target-pwffr" (UID: "ff43f31c-c779-406d-8352-4590967387a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:39:51.756342 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.756289 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bwlf\" (UniqueName: \"kubernetes.io/projected/a6f52c90-705e-42cf-8676-580f6ec9544e-kube-api-access-6bwlf\") pod \"tuned-mk6gn\" (UID: \"a6f52c90-705e-42cf-8676-580f6ec9544e\") " pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.757153 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.757084 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgbxp\" (UniqueName: \"kubernetes.io/projected/1e7898c0-39bf-42c9-a65e-30409d5a5ea1-kube-api-access-dgbxp\") pod \"multus-additional-cni-plugins-grt82\" (UID: \"1e7898c0-39bf-42c9-a65e-30409d5a5ea1\") " pod="openshift-multus/multus-additional-cni-plugins-grt82" Jul 08 18:39:51.757634 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.757572 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjnw4\" (UniqueName: \"kubernetes.io/projected/46c9d0e3-09c2-407c-afd5-7c04d7f5f683-kube-api-access-fjnw4\") pod \"aws-ebs-csi-driver-node-h4b2v\" (UID: \"46c9d0e3-09c2-407c-afd5-7c04d7f5f683\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" Jul 08 18:39:51.757735 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.757665 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2967x\" (UniqueName: \"kubernetes.io/projected/6c8c8ccc-4761-4a63-9361-0b4486ed8118-kube-api-access-2967x\") pod \"ovnkube-node-jrzrj\" (UID: \"6c8c8ccc-4761-4a63-9361-0b4486ed8118\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.758102 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.758067 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgz42\" (UniqueName: \"kubernetes.io/projected/713392b6-78a2-4ca4-8cf8-43f8ab24b0cb-kube-api-access-rgz42\") pod \"multus-rsw44\" (UID: \"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb\") " pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.758952 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.758907 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm796\" (UniqueName: \"kubernetes.io/projected/06234e92-3ccc-4e0a-9250-bfe6bf99befd-kube-api-access-sm796\") pod \"node-ca-jj7hl\" (UID: \"06234e92-3ccc-4e0a-9250-bfe6bf99befd\") " pod="openshift-image-registry/node-ca-jj7hl" Jul 08 18:39:51.760809 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.760635 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrmf7\" (UniqueName: \"kubernetes.io/projected/ee39c730-7ca1-4260-afea-a884940c00fc-kube-api-access-jrmf7\") pod \"network-metrics-daemon-252x5\" (UID: \"ee39c730-7ca1-4260-afea-a884940c00fc\") " pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:39:51.761488 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.761460 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xpcb\" (UniqueName: \"kubernetes.io/projected/08b493c4-bcc2-43fc-a7b6-2f51996d5aae-kube-api-access-6xpcb\") pod \"iptables-alerter-p45pj\" (UID: \"08b493c4-bcc2-43fc-a7b6-2f51996d5aae\") " pod="openshift-network-operator/iptables-alerter-p45pj" Jul 08 18:39:51.869534 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.869494 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p45pj" Jul 08 18:39:51.878487 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:51.878451 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b493c4_bcc2_43fc_a7b6_2f51996d5aae.slice/crio-9e0cc0dce69e49d34321c4f7f2000ce022068d49c534eb2068f5a5b828f53848 WatchSource:0}: Error finding container 9e0cc0dce69e49d34321c4f7f2000ce022068d49c534eb2068f5a5b828f53848: Status 404 returned error can't find the container with id 9e0cc0dce69e49d34321c4f7f2000ce022068d49c534eb2068f5a5b828f53848 Jul 08 18:39:51.882340 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.882315 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" Jul 08 18:39:51.890627 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:51.890601 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f52c90_705e_42cf_8676_580f6ec9544e.slice/crio-83c4495cbc6f615ef512ee6cff4d9aad8c273fab8254e32236509e0961da59d5 WatchSource:0}: Error finding container 83c4495cbc6f615ef512ee6cff4d9aad8c273fab8254e32236509e0961da59d5: Status 404 returned error can't find the container with id 83c4495cbc6f615ef512ee6cff4d9aad8c273fab8254e32236509e0961da59d5 Jul 08 18:39:51.892915 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.892887 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:39:51.899507 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.899483 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" Jul 08 18:39:51.901771 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:51.901745 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8c8ccc_4761_4a63_9361_0b4486ed8118.slice/crio-a3aea78914cfc9219542099b315705c305b7d644ba014124cdeca7ce88562d2e WatchSource:0}: Error finding container a3aea78914cfc9219542099b315705c305b7d644ba014124cdeca7ce88562d2e: Status 404 returned error can't find the container with id a3aea78914cfc9219542099b315705c305b7d644ba014124cdeca7ce88562d2e Jul 08 18:39:51.908354 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:51.908327 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c9d0e3_09c2_407c_afd5_7c04d7f5f683.slice/crio-c9699bf0c2a6d0590624027aff2645fbc86bc3fbb35809c1e3027df6e9e13a82 WatchSource:0}: Error finding container c9699bf0c2a6d0590624027aff2645fbc86bc3fbb35809c1e3027df6e9e13a82: Status 404 returned error can't find the container with id c9699bf0c2a6d0590624027aff2645fbc86bc3fbb35809c1e3027df6e9e13a82 Jul 08 18:39:51.909122 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.909106 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rsw44" Jul 08 18:39:51.917934 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:51.917910 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod713392b6_78a2_4ca4_8cf8_43f8ab24b0cb.slice/crio-bd3e6e49fc002566ab9a9d2dd6ee1bb87d7ae80181e231b235a76af67c117c94 WatchSource:0}: Error finding container bd3e6e49fc002566ab9a9d2dd6ee1bb87d7ae80181e231b235a76af67c117c94: Status 404 returned error can't find the container with id bd3e6e49fc002566ab9a9d2dd6ee1bb87d7ae80181e231b235a76af67c117c94 Jul 08 18:39:51.918026 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.917936 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v7qvx" Jul 08 18:39:51.927442 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.927414 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jj7hl" Jul 08 18:39:51.935488 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:51.935463 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06234e92_3ccc_4e0a_9250_bfe6bf99befd.slice/crio-b331089c73eb7823e1867ae0b1737788d07f063447ed9afacc1e7e4bfccc1b18 WatchSource:0}: Error finding container b331089c73eb7823e1867ae0b1737788d07f063447ed9afacc1e7e4bfccc1b18: Status 404 returned error can't find the container with id b331089c73eb7823e1867ae0b1737788d07f063447ed9afacc1e7e4bfccc1b18 Jul 08 18:39:51.939429 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.939403 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7gjd9" Jul 08 18:39:51.946588 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:51.946557 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-grt82" Jul 08 18:39:51.948044 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:51.948018 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc6d89d_de5e_44a5_9fe2_02785d57a8e0.slice/crio-bf1e93fcd138eb72335162e00f02332ffa5b461a5dc3751a5f02e3a55093fdc5 WatchSource:0}: Error finding container bf1e93fcd138eb72335162e00f02332ffa5b461a5dc3751a5f02e3a55093fdc5: Status 404 returned error can't find the container with id bf1e93fcd138eb72335162e00f02332ffa5b461a5dc3751a5f02e3a55093fdc5 Jul 08 18:39:51.955503 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:39:51.955476 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e7898c0_39bf_42c9_a65e_30409d5a5ea1.slice/crio-4f714ba08b22bc1d57b6b619912e849f17ddf5faf2f22a0573bda1e8019f6b6d WatchSource:0}: Error finding container 4f714ba08b22bc1d57b6b619912e849f17ddf5faf2f22a0573bda1e8019f6b6d: Status 404 returned error can't find the container with id 4f714ba08b22bc1d57b6b619912e849f17ddf5faf2f22a0573bda1e8019f6b6d Jul 08 18:39:52.242808 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:52.242719 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee39c730-7ca1-4260-afea-a884940c00fc-metrics-certs\") pod \"network-metrics-daemon-252x5\" (UID: \"ee39c730-7ca1-4260-afea-a884940c00fc\") " pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:39:52.242808 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:52.242766 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/610063ae-dbb1-4a58-bacd-06bf90518168-original-pull-secret\") pod \"global-pull-secret-syncer-szlwf\" (UID: \"610063ae-dbb1-4a58-bacd-06bf90518168\") " pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:39:52.243022 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:52.242872 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 18:39:52.243022 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:52.242891 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:39:52.243022 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:52.242935 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/610063ae-dbb1-4a58-bacd-06bf90518168-original-pull-secret podName:610063ae-dbb1-4a58-bacd-06bf90518168 nodeName:}" failed. No retries permitted until 2026-07-08 18:39:53.242918014 +0000 UTC m=+4.080245704 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/610063ae-dbb1-4a58-bacd-06bf90518168-original-pull-secret") pod "global-pull-secret-syncer-szlwf" (UID: "610063ae-dbb1-4a58-bacd-06bf90518168") : object "kube-system"/"original-pull-secret" not registered Jul 08 18:39:52.243022 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:52.242952 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee39c730-7ca1-4260-afea-a884940c00fc-metrics-certs podName:ee39c730-7ca1-4260-afea-a884940c00fc nodeName:}" failed. No retries permitted until 2026-07-08 18:39:53.242945196 +0000 UTC m=+4.080272884 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee39c730-7ca1-4260-afea-a884940c00fc-metrics-certs") pod "network-metrics-daemon-252x5" (UID: "ee39c730-7ca1-4260-afea-a884940c00fc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:39:52.343646 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:52.343604 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zwd9\" (UniqueName: \"kubernetes.io/projected/ff43f31c-c779-406d-8352-4590967387a5-kube-api-access-2zwd9\") pod \"network-check-target-pwffr\" (UID: \"ff43f31c-c779-406d-8352-4590967387a5\") " pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:39:52.343838 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:52.343760 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 18:39:52.343838 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:52.343778 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 18:39:52.343838 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:52.343791 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2zwd9 for pod openshift-network-diagnostics/network-check-target-pwffr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:39:52.343966 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:52.343859 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff43f31c-c779-406d-8352-4590967387a5-kube-api-access-2zwd9 podName:ff43f31c-c779-406d-8352-4590967387a5 nodeName:}" failed. No retries permitted until 2026-07-08 18:39:53.343838343 +0000 UTC m=+4.181166045 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2zwd9" (UniqueName: "kubernetes.io/projected/ff43f31c-c779-406d-8352-4590967387a5-kube-api-access-2zwd9") pod "network-check-target-pwffr" (UID: "ff43f31c-c779-406d-8352-4590967387a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:39:52.668389 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:52.668293 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 18:34:51 +0000 UTC" deadline="2028-03-20 09:47:59.23847948 +0000 UTC" Jul 08 18:39:52.668389 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:52.668384 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14895h8m6.570100179s" Jul 08 18:39:52.775049 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:52.775007 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:39:52.775210 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:52.775165 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlwf" podUID="610063ae-dbb1-4a58-bacd-06bf90518168" Jul 08 18:39:52.782463 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:52.782425 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" event={"ID":"6c8c8ccc-4761-4a63-9361-0b4486ed8118","Type":"ContainerStarted","Data":"a3aea78914cfc9219542099b315705c305b7d644ba014124cdeca7ce88562d2e"} Jul 08 18:39:52.783940 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:52.783900 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p45pj" event={"ID":"08b493c4-bcc2-43fc-a7b6-2f51996d5aae","Type":"ContainerStarted","Data":"9e0cc0dce69e49d34321c4f7f2000ce022068d49c534eb2068f5a5b828f53848"} Jul 08 18:39:52.785683 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:52.785655 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7gjd9" event={"ID":"bcc6d89d-de5e-44a5-9fe2-02785d57a8e0","Type":"ContainerStarted","Data":"bf1e93fcd138eb72335162e00f02332ffa5b461a5dc3751a5f02e3a55093fdc5"} Jul 08 18:39:52.787050 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:52.787021 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jj7hl" event={"ID":"06234e92-3ccc-4e0a-9250-bfe6bf99befd","Type":"ContainerStarted","Data":"b331089c73eb7823e1867ae0b1737788d07f063447ed9afacc1e7e4bfccc1b18"} Jul 08 18:39:52.788592 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:52.788551 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v7qvx" event={"ID":"d61025be-679c-46f7-9a55-5049757d2d53","Type":"ContainerStarted","Data":"8e36058bfcb486ccac081ed7fb6d17926535abc0861307b774999c4922d8abe0"} Jul 08 18:39:52.789948 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:52.789918 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" event={"ID":"46c9d0e3-09c2-407c-afd5-7c04d7f5f683","Type":"ContainerStarted","Data":"c9699bf0c2a6d0590624027aff2645fbc86bc3fbb35809c1e3027df6e9e13a82"} Jul 08 18:39:52.791260 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:52.791235 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" event={"ID":"a6f52c90-705e-42cf-8676-580f6ec9544e","Type":"ContainerStarted","Data":"83c4495cbc6f615ef512ee6cff4d9aad8c273fab8254e32236509e0961da59d5"} Jul 08 18:39:52.792582 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:52.792549 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grt82" event={"ID":"1e7898c0-39bf-42c9-a65e-30409d5a5ea1","Type":"ContainerStarted","Data":"4f714ba08b22bc1d57b6b619912e849f17ddf5faf2f22a0573bda1e8019f6b6d"} Jul 08 18:39:52.793827 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:52.793801 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rsw44" event={"ID":"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb","Type":"ContainerStarted","Data":"bd3e6e49fc002566ab9a9d2dd6ee1bb87d7ae80181e231b235a76af67c117c94"} Jul 08 18:39:53.252734 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:53.252701 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee39c730-7ca1-4260-afea-a884940c00fc-metrics-certs\") pod \"network-metrics-daemon-252x5\" (UID: \"ee39c730-7ca1-4260-afea-a884940c00fc\") " pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:39:53.252921 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:53.252752 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/610063ae-dbb1-4a58-bacd-06bf90518168-original-pull-secret\") pod \"global-pull-secret-syncer-szlwf\" (UID: \"610063ae-dbb1-4a58-bacd-06bf90518168\") " pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:39:53.252985 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:53.252943 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 18:39:53.253053 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:53.253006 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/610063ae-dbb1-4a58-bacd-06bf90518168-original-pull-secret podName:610063ae-dbb1-4a58-bacd-06bf90518168 nodeName:}" failed. No retries permitted until 2026-07-08 18:39:55.252986721 +0000 UTC m=+6.090314413 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/610063ae-dbb1-4a58-bacd-06bf90518168-original-pull-secret") pod "global-pull-secret-syncer-szlwf" (UID: "610063ae-dbb1-4a58-bacd-06bf90518168") : object "kube-system"/"original-pull-secret" not registered Jul 08 18:39:53.253455 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:53.253436 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:39:53.253547 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:53.253497 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee39c730-7ca1-4260-afea-a884940c00fc-metrics-certs podName:ee39c730-7ca1-4260-afea-a884940c00fc nodeName:}" failed. No retries permitted until 2026-07-08 18:39:55.253478579 +0000 UTC m=+6.090806270 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee39c730-7ca1-4260-afea-a884940c00fc-metrics-certs") pod "network-metrics-daemon-252x5" (UID: "ee39c730-7ca1-4260-afea-a884940c00fc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:39:53.353247 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:53.353211 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zwd9\" (UniqueName: \"kubernetes.io/projected/ff43f31c-c779-406d-8352-4590967387a5-kube-api-access-2zwd9\") pod \"network-check-target-pwffr\" (UID: \"ff43f31c-c779-406d-8352-4590967387a5\") " pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:39:53.353458 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:53.353448 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 18:39:53.353532 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:53.353468 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 18:39:53.353532 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:53.353480 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2zwd9 for pod openshift-network-diagnostics/network-check-target-pwffr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:39:53.353647 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:53.353543 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff43f31c-c779-406d-8352-4590967387a5-kube-api-access-2zwd9 podName:ff43f31c-c779-406d-8352-4590967387a5 nodeName:}" failed. No retries permitted until 2026-07-08 18:39:55.353523468 +0000 UTC m=+6.190851177 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2zwd9" (UniqueName: "kubernetes.io/projected/ff43f31c-c779-406d-8352-4590967387a5-kube-api-access-2zwd9") pod "network-check-target-pwffr" (UID: "ff43f31c-c779-406d-8352-4590967387a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:39:53.776767 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:53.776514 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:39:53.777237 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:53.776890 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-252x5" podUID="ee39c730-7ca1-4260-afea-a884940c00fc" Jul 08 18:39:53.777340 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:53.777318 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:39:53.777511 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:53.777450 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwffr" podUID="ff43f31c-c779-406d-8352-4590967387a5" Jul 08 18:39:53.801069 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:53.799924 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-102.ec2.internal" event={"ID":"a4bb240418508e65be90c2a28bf12044","Type":"ContainerStarted","Data":"23e1e8bfefd671e0187ad77f579456e8c94f3d2ee2ed1eeddf968c023d6353c7"} Jul 08 18:39:53.821825 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:53.821761 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-102.ec2.internal" podStartSLOduration=2.821741817 podStartE2EDuration="2.821741817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:39:53.820807274 +0000 UTC m=+4.658134988" watchObservedRunningTime="2026-07-08 18:39:53.821741817 +0000 UTC m=+4.659069531" Jul 08 18:39:54.775537 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:54.774136 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:39:54.775537 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:54.774276 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlwf" podUID="610063ae-dbb1-4a58-bacd-06bf90518168" Jul 08 18:39:54.808099 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:54.807919 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-102.ec2.internal" event={"ID":"da7a407a9bbccda77188a4b8cd9fd490","Type":"ContainerStarted","Data":"6dbe04c1e6e0460413bb47fdf8110061a062ddcc1ea2bd0ad693b018e015d92d"} Jul 08 18:39:55.271466 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:55.271428 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee39c730-7ca1-4260-afea-a884940c00fc-metrics-certs\") pod \"network-metrics-daemon-252x5\" (UID: \"ee39c730-7ca1-4260-afea-a884940c00fc\") " pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:39:55.271668 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:55.271483 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/610063ae-dbb1-4a58-bacd-06bf90518168-original-pull-secret\") pod \"global-pull-secret-syncer-szlwf\" (UID: \"610063ae-dbb1-4a58-bacd-06bf90518168\") " pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:39:55.271668 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:55.271617 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 18:39:55.271791 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:55.271684 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/610063ae-dbb1-4a58-bacd-06bf90518168-original-pull-secret podName:610063ae-dbb1-4a58-bacd-06bf90518168 nodeName:}" failed. No retries permitted until 2026-07-08 18:39:59.271663702 +0000 UTC m=+10.108991407 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/610063ae-dbb1-4a58-bacd-06bf90518168-original-pull-secret") pod "global-pull-secret-syncer-szlwf" (UID: "610063ae-dbb1-4a58-bacd-06bf90518168") : object "kube-system"/"original-pull-secret" not registered Jul 08 18:39:55.272123 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:55.272098 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:39:55.272220 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:55.272160 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee39c730-7ca1-4260-afea-a884940c00fc-metrics-certs podName:ee39c730-7ca1-4260-afea-a884940c00fc nodeName:}" failed. No retries permitted until 2026-07-08 18:39:59.272143052 +0000 UTC m=+10.109470759 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee39c730-7ca1-4260-afea-a884940c00fc-metrics-certs") pod "network-metrics-daemon-252x5" (UID: "ee39c730-7ca1-4260-afea-a884940c00fc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:39:55.373444 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:55.372652 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zwd9\" (UniqueName: \"kubernetes.io/projected/ff43f31c-c779-406d-8352-4590967387a5-kube-api-access-2zwd9\") pod \"network-check-target-pwffr\" (UID: \"ff43f31c-c779-406d-8352-4590967387a5\") " pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:39:55.373444 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:55.372912 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 18:39:55.373444 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:55.372940 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 18:39:55.373444 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:55.372955 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2zwd9 for pod openshift-network-diagnostics/network-check-target-pwffr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:39:55.373444 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:55.373058 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff43f31c-c779-406d-8352-4590967387a5-kube-api-access-2zwd9 podName:ff43f31c-c779-406d-8352-4590967387a5 nodeName:}" failed. No retries permitted until 2026-07-08 18:39:59.373036506 +0000 UTC m=+10.210364195 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2zwd9" (UniqueName: "kubernetes.io/projected/ff43f31c-c779-406d-8352-4590967387a5-kube-api-access-2zwd9") pod "network-check-target-pwffr" (UID: "ff43f31c-c779-406d-8352-4590967387a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:39:55.774991 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:55.774429 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:39:55.774991 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:55.774583 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-252x5" podUID="ee39c730-7ca1-4260-afea-a884940c00fc" Jul 08 18:39:55.775550 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:55.775363 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:39:55.775550 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:55.775478 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwffr" podUID="ff43f31c-c779-406d-8352-4590967387a5" Jul 08 18:39:56.774633 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:56.774557 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:39:56.775076 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:56.774690 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlwf" podUID="610063ae-dbb1-4a58-bacd-06bf90518168" Jul 08 18:39:56.814540 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:56.814503 2575 generic.go:356] "Generic (PLEG): container finished" podID="da7a407a9bbccda77188a4b8cd9fd490" containerID="6dbe04c1e6e0460413bb47fdf8110061a062ddcc1ea2bd0ad693b018e015d92d" exitCode=0 Jul 08 18:39:56.814691 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:56.814563 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-102.ec2.internal" event={"ID":"da7a407a9bbccda77188a4b8cd9fd490","Type":"ContainerDied","Data":"6dbe04c1e6e0460413bb47fdf8110061a062ddcc1ea2bd0ad693b018e015d92d"} Jul 08 18:39:57.775044 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:57.775011 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:39:57.775532 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:57.775140 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwffr" podUID="ff43f31c-c779-406d-8352-4590967387a5" Jul 08 18:39:57.775663 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:57.775646 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:39:57.775776 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:57.775755 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-252x5" podUID="ee39c730-7ca1-4260-afea-a884940c00fc" Jul 08 18:39:58.774734 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:58.774699 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:39:58.774925 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:58.774821 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlwf" podUID="610063ae-dbb1-4a58-bacd-06bf90518168" Jul 08 18:39:59.307881 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:59.307788 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee39c730-7ca1-4260-afea-a884940c00fc-metrics-certs\") pod \"network-metrics-daemon-252x5\" (UID: \"ee39c730-7ca1-4260-afea-a884940c00fc\") " pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:39:59.307881 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:59.307856 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/610063ae-dbb1-4a58-bacd-06bf90518168-original-pull-secret\") pod \"global-pull-secret-syncer-szlwf\" (UID: \"610063ae-dbb1-4a58-bacd-06bf90518168\") " pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:39:59.308420 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:59.307932 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:39:59.308420 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:59.307979 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 18:39:59.308420 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:59.308006 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee39c730-7ca1-4260-afea-a884940c00fc-metrics-certs podName:ee39c730-7ca1-4260-afea-a884940c00fc nodeName:}" failed. No retries permitted until 2026-07-08 18:40:07.307983981 +0000 UTC m=+18.145311683 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee39c730-7ca1-4260-afea-a884940c00fc-metrics-certs") pod "network-metrics-daemon-252x5" (UID: "ee39c730-7ca1-4260-afea-a884940c00fc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:39:59.308420 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:59.308035 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/610063ae-dbb1-4a58-bacd-06bf90518168-original-pull-secret podName:610063ae-dbb1-4a58-bacd-06bf90518168 nodeName:}" failed. No retries permitted until 2026-07-08 18:40:07.308018919 +0000 UTC m=+18.145346606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/610063ae-dbb1-4a58-bacd-06bf90518168-original-pull-secret") pod "global-pull-secret-syncer-szlwf" (UID: "610063ae-dbb1-4a58-bacd-06bf90518168") : object "kube-system"/"original-pull-secret" not registered Jul 08 18:39:59.409000 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:59.408958 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zwd9\" (UniqueName: \"kubernetes.io/projected/ff43f31c-c779-406d-8352-4590967387a5-kube-api-access-2zwd9\") pod \"network-check-target-pwffr\" (UID: \"ff43f31c-c779-406d-8352-4590967387a5\") " pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:39:59.409207 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:59.409122 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 18:39:59.409207 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:59.409143 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 18:39:59.409207 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:59.409155 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2zwd9 for pod openshift-network-diagnostics/network-check-target-pwffr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:39:59.409396 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:59.409218 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff43f31c-c779-406d-8352-4590967387a5-kube-api-access-2zwd9 podName:ff43f31c-c779-406d-8352-4590967387a5 nodeName:}" failed. No retries permitted until 2026-07-08 18:40:07.409200515 +0000 UTC m=+18.246528224 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2zwd9" (UniqueName: "kubernetes.io/projected/ff43f31c-c779-406d-8352-4590967387a5-kube-api-access-2zwd9") pod "network-check-target-pwffr" (UID: "ff43f31c-c779-406d-8352-4590967387a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:39:59.775455 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:59.775419 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:39:59.775643 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:39:59.775419 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:39:59.775643 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:59.775536 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwffr" podUID="ff43f31c-c779-406d-8352-4590967387a5" Jul 08 18:39:59.775749 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:39:59.775654 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-252x5" podUID="ee39c730-7ca1-4260-afea-a884940c00fc" Jul 08 18:40:00.774653 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:00.774613 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:40:00.775052 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:00.774732 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlwf" podUID="610063ae-dbb1-4a58-bacd-06bf90518168" Jul 08 18:40:01.775687 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:01.775651 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:40:01.776152 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:01.775793 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-252x5" podUID="ee39c730-7ca1-4260-afea-a884940c00fc" Jul 08 18:40:01.776152 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:01.775938 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:40:01.776152 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:01.776044 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwffr" podUID="ff43f31c-c779-406d-8352-4590967387a5" Jul 08 18:40:02.774636 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:02.774369 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:40:02.774840 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:02.774744 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlwf" podUID="610063ae-dbb1-4a58-bacd-06bf90518168" Jul 08 18:40:02.827647 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:02.827604 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jj7hl" event={"ID":"06234e92-3ccc-4e0a-9250-bfe6bf99befd","Type":"ContainerStarted","Data":"7fb357bd1bc318ce1f4475c4638ca8059a41c5b2a67e1fd38f863cfc36dae809"} Jul 08 18:40:02.829678 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:02.829443 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v7qvx" event={"ID":"d61025be-679c-46f7-9a55-5049757d2d53","Type":"ContainerStarted","Data":"b173429db3577edf1cc39ba0c76fbcb690e46ecb3807e49c0d3dae0d0b7a37f1"} Jul 08 18:40:02.833404 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:02.833378 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" event={"ID":"46c9d0e3-09c2-407c-afd5-7c04d7f5f683","Type":"ContainerStarted","Data":"4b5028df262ee97d4d6eba8acc230dc75b91fb2268f3772d72a61f1e1f1437f0"} Jul 08 18:40:02.834697 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:02.834664 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" event={"ID":"a6f52c90-705e-42cf-8676-580f6ec9544e","Type":"ContainerStarted","Data":"c09025d277158d2ef9d3a16ddd9c419f440145f689501762eac6bf9a419d7047"} Jul 08 18:40:02.836018 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:02.835998 2575 generic.go:356] "Generic (PLEG): container finished" podID="1e7898c0-39bf-42c9-a65e-30409d5a5ea1" containerID="bbf00318f9bfdda49288a9f41a1972667ede7a6a28b34686d66d4452e8792680" exitCode=0 Jul 08 18:40:02.836111 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:02.836060 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grt82" event={"ID":"1e7898c0-39bf-42c9-a65e-30409d5a5ea1","Type":"ContainerDied","Data":"bbf00318f9bfdda49288a9f41a1972667ede7a6a28b34686d66d4452e8792680"} Jul 08 18:40:02.837931 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:02.837912 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-102.ec2.internal" event={"ID":"da7a407a9bbccda77188a4b8cd9fd490","Type":"ContainerStarted","Data":"7b177d2d523e3ea21d2c176702bd9143f993af1948c9a261d605d56ef3eea0c7"} Jul 08 18:40:02.839274 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:02.839252 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7gjd9" event={"ID":"bcc6d89d-de5e-44a5-9fe2-02785d57a8e0","Type":"ContainerStarted","Data":"e3419ce1c2420045cb8adc97b8f3b2daccd1c8930fc3e00432f54c603c8d4c58"} Jul 08 18:40:02.852378 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:02.852336 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jj7hl" podStartSLOduration=2.686658756 podStartE2EDuration="12.852323894s" totalImagesPullingTime="10.165665138s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:40:02.852010213 +0000 UTC m=+13.689337917" watchObservedRunningTime="2026-07-08 18:40:02.852323894 +0000 UTC m=+13.689651602" Jul 08 18:40:02.875548 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:02.875483 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mk6gn" podStartSLOduration=2.641608249 podStartE2EDuration="12.875464046s" totalImagesPullingTime="10.233855797s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:40:02.874764332 +0000 UTC m=+13.712092043" watchObservedRunningTime="2026-07-08 18:40:02.875464046 +0000 UTC m=+13.712791753" Jul 08 18:40:02.892699 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:02.892643 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v7qvx" podStartSLOduration=2.71958871 podStartE2EDuration="12.892626141s" totalImagesPullingTime="10.173037431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:40:02.891954547 +0000 UTC m=+13.729282256" watchObservedRunningTime="2026-07-08 18:40:02.892626141 +0000 UTC m=+13.729953849" Jul 08 18:40:02.939834 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:02.939760 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7gjd9" podStartSLOduration=2.783080301 podStartE2EDuration="12.939740727s" totalImagesPullingTime="10.156660426s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:40:02.912637293 +0000 UTC m=+13.749965003" watchObservedRunningTime="2026-07-08 18:40:02.939740727 +0000 UTC m=+13.777068436" Jul 08 18:40:02.965479 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:02.965420 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-102.ec2.internal" podStartSLOduration=11.965404217 podStartE2EDuration="11.965404217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:40:02.965089803 +0000 UTC m=+13.802417503" watchObservedRunningTime="2026-07-08 18:40:02.965404217 +0000 UTC m=+13.802731924" Jul 08 18:40:03.774454 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:03.774420 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:40:03.774647 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:03.774421 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:40:03.774647 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:03.774569 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwffr" podUID="ff43f31c-c779-406d-8352-4590967387a5" Jul 08 18:40:03.774770 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:03.774643 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-252x5" podUID="ee39c730-7ca1-4260-afea-a884940c00fc" Jul 08 18:40:03.843942 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:03.843885 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p45pj" event={"ID":"08b493c4-bcc2-43fc-a7b6-2f51996d5aae","Type":"ContainerStarted","Data":"d32ef789c7cc32a4a61b47e54382c88cf4da2d5e5197e33add5910ad344ba2ea"} Jul 08 18:40:04.432509 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:04.432474 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 08 18:40:04.748280 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:04.748163 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T18:40:04.432502762Z","UUID":"98531243-f667-4da7-adc9-51921fcd8e19","Handler":null,"Name":"","Endpoint":""} Jul 08 18:40:04.750353 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:04.750315 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 08 18:40:04.750504 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:04.750370 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 08 18:40:04.774438 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:04.774408 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:40:04.774642 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:04.774542 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlwf" podUID="610063ae-dbb1-4a58-bacd-06bf90518168" Jul 08 18:40:04.855819 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:04.855780 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" event={"ID":"46c9d0e3-09c2-407c-afd5-7c04d7f5f683","Type":"ContainerStarted","Data":"df0c946ce7296bea9e420b686fdbe57c8723e4902fe0af638c1463c019963392"} Jul 08 18:40:05.522117 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:05.522074 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v7qvx" Jul 08 18:40:05.522973 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:05.522950 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v7qvx" Jul 08 18:40:05.541837 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:05.541776 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-p45pj" podStartSLOduration=5.318490498 podStartE2EDuration="15.541755535s" totalImagesPullingTime="10.223265037s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:40:03.861018205 +0000 UTC m=+14.698345914" watchObservedRunningTime="2026-07-08 18:40:05.541755535 +0000 UTC m=+16.379083244" Jul 08 18:40:05.774838 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:05.774735 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:40:05.774838 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:05.774780 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:40:05.775053 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:05.774871 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-252x5" podUID="ee39c730-7ca1-4260-afea-a884940c00fc" Jul 08 18:40:05.775053 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:05.774980 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwffr" podUID="ff43f31c-c779-406d-8352-4590967387a5" Jul 08 18:40:06.774028 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:06.773991 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:40:06.774827 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:06.774132 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlwf" podUID="610063ae-dbb1-4a58-bacd-06bf90518168" Jul 08 18:40:07.366540 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:07.366509 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee39c730-7ca1-4260-afea-a884940c00fc-metrics-certs\") pod \"network-metrics-daemon-252x5\" (UID: \"ee39c730-7ca1-4260-afea-a884940c00fc\") " pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:40:07.366715 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:07.366558 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/610063ae-dbb1-4a58-bacd-06bf90518168-original-pull-secret\") pod \"global-pull-secret-syncer-szlwf\" (UID: \"610063ae-dbb1-4a58-bacd-06bf90518168\") " pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:40:07.366715 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:07.366666 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:40:07.366715 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:07.366682 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 18:40:07.366879 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:07.366735 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee39c730-7ca1-4260-afea-a884940c00fc-metrics-certs podName:ee39c730-7ca1-4260-afea-a884940c00fc nodeName:}" failed. No retries permitted until 2026-07-08 18:40:23.366715484 +0000 UTC m=+34.204043178 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee39c730-7ca1-4260-afea-a884940c00fc-metrics-certs") pod "network-metrics-daemon-252x5" (UID: "ee39c730-7ca1-4260-afea-a884940c00fc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:40:07.366879 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:07.366755 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/610063ae-dbb1-4a58-bacd-06bf90518168-original-pull-secret podName:610063ae-dbb1-4a58-bacd-06bf90518168 nodeName:}" failed. No retries permitted until 2026-07-08 18:40:23.366745947 +0000 UTC m=+34.204073637 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/610063ae-dbb1-4a58-bacd-06bf90518168-original-pull-secret") pod "global-pull-secret-syncer-szlwf" (UID: "610063ae-dbb1-4a58-bacd-06bf90518168") : object "kube-system"/"original-pull-secret" not registered Jul 08 18:40:07.467682 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:07.467649 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zwd9\" (UniqueName: \"kubernetes.io/projected/ff43f31c-c779-406d-8352-4590967387a5-kube-api-access-2zwd9\") pod \"network-check-target-pwffr\" (UID: \"ff43f31c-c779-406d-8352-4590967387a5\") " pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:40:07.467899 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:07.467876 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 18:40:07.467971 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:07.467907 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 18:40:07.467971 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:07.467921 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2zwd9 for pod openshift-network-diagnostics/network-check-target-pwffr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:40:07.468073 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:07.468060 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff43f31c-c779-406d-8352-4590967387a5-kube-api-access-2zwd9 podName:ff43f31c-c779-406d-8352-4590967387a5 nodeName:}" failed. No retries permitted until 2026-07-08 18:40:23.468037356 +0000 UTC m=+34.305365055 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2zwd9" (UniqueName: "kubernetes.io/projected/ff43f31c-c779-406d-8352-4590967387a5-kube-api-access-2zwd9") pod "network-check-target-pwffr" (UID: "ff43f31c-c779-406d-8352-4590967387a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:40:07.774363 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:07.774105 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:40:07.774786 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:07.774117 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:40:07.774786 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:07.774461 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-252x5" podUID="ee39c730-7ca1-4260-afea-a884940c00fc" Jul 08 18:40:07.774786 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:07.774598 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwffr" podUID="ff43f31c-c779-406d-8352-4590967387a5" Jul 08 18:40:08.774196 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:08.774160 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:40:08.774387 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:08.774288 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlwf" podUID="610063ae-dbb1-4a58-bacd-06bf90518168" Jul 08 18:40:09.775026 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:09.774990 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:40:09.775479 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:09.775097 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-252x5" podUID="ee39c730-7ca1-4260-afea-a884940c00fc" Jul 08 18:40:09.775479 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:09.775154 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:40:09.775479 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:09.775267 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwffr" podUID="ff43f31c-c779-406d-8352-4590967387a5" Jul 08 18:40:10.773908 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:10.773877 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:40:10.774100 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:10.774000 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlwf" podUID="610063ae-dbb1-4a58-bacd-06bf90518168" Jul 08 18:40:11.773866 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:11.773816 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:40:11.774458 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:11.773969 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-252x5" podUID="ee39c730-7ca1-4260-afea-a884940c00fc" Jul 08 18:40:11.774458 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:11.774043 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:40:11.774458 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:11.774163 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwffr" podUID="ff43f31c-c779-406d-8352-4590967387a5" Jul 08 18:40:12.774210 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:12.774176 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:40:12.774774 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:12.774357 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlwf" podUID="610063ae-dbb1-4a58-bacd-06bf90518168" Jul 08 18:40:13.324006 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:13.323963 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v7qvx" Jul 08 18:40:13.324212 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:13.324137 2575 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 08 18:40:13.324852 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:13.324822 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v7qvx" Jul 08 18:40:13.774717 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:13.774687 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:40:13.775171 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:13.774689 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:40:13.775171 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:13.774792 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwffr" podUID="ff43f31c-c779-406d-8352-4590967387a5" Jul 08 18:40:13.775171 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:13.774914 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-252x5" podUID="ee39c730-7ca1-4260-afea-a884940c00fc" Jul 08 18:40:14.774641 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:14.774608 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:40:14.774838 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:14.774728 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlwf" podUID="610063ae-dbb1-4a58-bacd-06bf90518168" Jul 08 18:40:15.775029 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:15.774991 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:40:15.775422 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:15.775129 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwffr" podUID="ff43f31c-c779-406d-8352-4590967387a5" Jul 08 18:40:15.775422 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:15.775186 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:40:15.775698 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:15.775672 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-252x5" podUID="ee39c730-7ca1-4260-afea-a884940c00fc" Jul 08 18:40:15.881326 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:15.880910 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" event={"ID":"46c9d0e3-09c2-407c-afd5-7c04d7f5f683","Type":"ContainerStarted","Data":"58d01c3478e518da8f84cf07770dfcb31c212892a719920e09390f3ba616fc1c"} Jul 08 18:40:15.883407 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:15.883379 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grt82" event={"ID":"1e7898c0-39bf-42c9-a65e-30409d5a5ea1","Type":"ContainerStarted","Data":"0dc58d57dd9836c837ac1bf3e3109d70edd186b7c0f992f10eafcde2c2991c03"} Jul 08 18:40:15.886650 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:15.886465 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rsw44" event={"ID":"713392b6-78a2-4ca4-8cf8-43f8ab24b0cb","Type":"ContainerStarted","Data":"b60db64343df57acc9f3506554df4bf423b7d7263f87254b5f33e05626c29730"} Jul 08 18:40:15.890430 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:15.890406 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" event={"ID":"6c8c8ccc-4761-4a63-9361-0b4486ed8118","Type":"ContainerStarted","Data":"53a11c5ef0e31a9a4076c9c3b6ee5e2c713488a4782e34b63b16cee97c9f4e76"} Jul 08 18:40:15.900511 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:15.900467 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4b2v" podStartSLOduration=2.5455651230000003 podStartE2EDuration="25.900447776s" totalImagesPullingTime="23.354882653s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:39:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:40:15.899996307 +0000 UTC m=+26.737324017" watchObservedRunningTime="2026-07-08 18:40:15.900447776 +0000 UTC m=+26.737775484" Jul 08 18:40:15.920241 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:15.920185 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rsw44" podStartSLOduration=2.133715932 podStartE2EDuration="25.920169958s" totalImagesPullingTime="23.786454026s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:40:15.91959233 +0000 UTC m=+26.756920039" watchObservedRunningTime="2026-07-08 18:40:15.920169958 +0000 UTC m=+26.757497665" Jul 08 18:40:16.774450 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:16.774421 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:40:16.774579 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:16.774531 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlwf" podUID="610063ae-dbb1-4a58-bacd-06bf90518168" Jul 08 18:40:16.897283 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:16.897244 2575 generic.go:356] "Generic (PLEG): container finished" podID="1e7898c0-39bf-42c9-a65e-30409d5a5ea1" containerID="0dc58d57dd9836c837ac1bf3e3109d70edd186b7c0f992f10eafcde2c2991c03" exitCode=0 Jul 08 18:40:16.897854 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:16.897326 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grt82" event={"ID":"1e7898c0-39bf-42c9-a65e-30409d5a5ea1","Type":"ContainerDied","Data":"0dc58d57dd9836c837ac1bf3e3109d70edd186b7c0f992f10eafcde2c2991c03"} Jul 08 18:40:16.900290 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:16.900267 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" event={"ID":"6c8c8ccc-4761-4a63-9361-0b4486ed8118","Type":"ContainerStarted","Data":"879d275b2b7df485f46f774379717aada481c136a5470999fed4b3768476a9dd"} Jul 08 18:40:16.900383 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:16.900341 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" event={"ID":"6c8c8ccc-4761-4a63-9361-0b4486ed8118","Type":"ContainerStarted","Data":"0d14e35edf726baebf5edcd63a905fb20b723712d9e5c248dc7b39cc6bb1bac5"} Jul 08 18:40:16.900425 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:16.900385 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" event={"ID":"6c8c8ccc-4761-4a63-9361-0b4486ed8118","Type":"ContainerStarted","Data":"c8f7cb3721eb9f61c74bbaaa95c12e8cfbd90d3313fa9c750cbb7e27af62b210"} Jul 08 18:40:16.900425 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:16.900415 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" event={"ID":"6c8c8ccc-4761-4a63-9361-0b4486ed8118","Type":"ContainerStarted","Data":"6858e9c124b5d7fbd338c58e0b0454096e0861a07c7593d721ed5e8898e61591"} Jul 08 18:40:16.900487 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:16.900449 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" event={"ID":"6c8c8ccc-4761-4a63-9361-0b4486ed8118","Type":"ContainerStarted","Data":"ce23e047de0edbce1ad2808603f6f9728a755d7e9125688c16d99a9379169bb8"} Jul 08 18:40:17.774472 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:17.774434 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:40:17.774685 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:17.774595 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-252x5" podUID="ee39c730-7ca1-4260-afea-a884940c00fc" Jul 08 18:40:17.775418 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:17.775396 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:40:17.775514 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:17.775497 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwffr" podUID="ff43f31c-c779-406d-8352-4590967387a5" Jul 08 18:40:18.774769 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:18.774683 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:40:18.775258 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:18.774787 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlwf" podUID="610063ae-dbb1-4a58-bacd-06bf90518168" Jul 08 18:40:18.907643 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:18.907609 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" event={"ID":"6c8c8ccc-4761-4a63-9361-0b4486ed8118","Type":"ContainerStarted","Data":"e6c3ce3bcb9b11b1ccf7c8e39d567ea9ae56ee791806f3d29f97dd45ed56d8ef"} Jul 08 18:40:18.909568 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:18.909544 2575 generic.go:356] "Generic (PLEG): container finished" podID="1e7898c0-39bf-42c9-a65e-30409d5a5ea1" containerID="665113354d5d2756a6b182223be96467cf925993b6b07f9b9adc5debeebc47ce" exitCode=0 Jul 08 18:40:18.909680 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:18.909578 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grt82" event={"ID":"1e7898c0-39bf-42c9-a65e-30409d5a5ea1","Type":"ContainerDied","Data":"665113354d5d2756a6b182223be96467cf925993b6b07f9b9adc5debeebc47ce"} Jul 08 18:40:19.774877 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:19.774844 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:40:19.775358 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:19.774971 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-252x5" podUID="ee39c730-7ca1-4260-afea-a884940c00fc" Jul 08 18:40:19.775358 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:19.775013 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:40:19.775358 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:19.775129 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwffr" podUID="ff43f31c-c779-406d-8352-4590967387a5" Jul 08 18:40:20.774609 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:20.774368 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:40:20.774774 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:20.774631 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlwf" podUID="610063ae-dbb1-4a58-bacd-06bf90518168" Jul 08 18:40:20.916746 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:20.916709 2575 generic.go:356] "Generic (PLEG): container finished" podID="1e7898c0-39bf-42c9-a65e-30409d5a5ea1" containerID="0b3d5e73f737361954a43d4010475d6776ac8a3953163099133072247ac1845d" exitCode=0 Jul 08 18:40:20.917165 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:20.916786 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grt82" event={"ID":"1e7898c0-39bf-42c9-a65e-30409d5a5ea1","Type":"ContainerDied","Data":"0b3d5e73f737361954a43d4010475d6776ac8a3953163099133072247ac1845d"} Jul 08 18:40:20.920271 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:20.920246 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" event={"ID":"6c8c8ccc-4761-4a63-9361-0b4486ed8118","Type":"ContainerStarted","Data":"0f3585e83e1920f1fcc3a469c08bf41d1bde6403640e06478a4bb919bc97b7ae"} Jul 08 18:40:20.920531 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:20.920511 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:40:20.920629 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:20.920576 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:40:20.920629 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:20.920625 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:40:20.935896 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:20.935868 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:40:20.936012 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:20.935965 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:40:20.970617 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:20.970568 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" podStartSLOduration=7.140013704 podStartE2EDuration="30.970552968s" totalImagesPullingTime="23.830539264s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:40:20.970093606 +0000 UTC m=+31.807421314" watchObservedRunningTime="2026-07-08 18:40:20.970552968 +0000 UTC m=+31.807880676" Jul 08 18:40:21.776815 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:21.776780 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:40:21.777013 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:21.776924 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-252x5" podUID="ee39c730-7ca1-4260-afea-a884940c00fc" Jul 08 18:40:21.777384 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:21.777363 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:40:21.777484 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:21.777465 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwffr" podUID="ff43f31c-c779-406d-8352-4590967387a5" Jul 08 18:40:22.757454 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:22.757419 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pwffr"] Jul 08 18:40:22.757819 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:22.757541 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:40:22.757819 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:22.757644 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwffr" podUID="ff43f31c-c779-406d-8352-4590967387a5" Jul 08 18:40:22.759691 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:22.759664 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-szlwf"] Jul 08 18:40:22.759823 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:22.759774 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:40:22.759891 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:22.759871 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlwf" podUID="610063ae-dbb1-4a58-bacd-06bf90518168" Jul 08 18:40:22.773973 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:22.773894 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-252x5"] Jul 08 18:40:22.774099 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:22.774014 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:40:22.774143 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:22.774119 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-252x5" podUID="ee39c730-7ca1-4260-afea-a884940c00fc" Jul 08 18:40:23.381273 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:23.381239 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee39c730-7ca1-4260-afea-a884940c00fc-metrics-certs\") pod \"network-metrics-daemon-252x5\" (UID: \"ee39c730-7ca1-4260-afea-a884940c00fc\") " pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:40:23.381273 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:23.381286 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/610063ae-dbb1-4a58-bacd-06bf90518168-original-pull-secret\") pod \"global-pull-secret-syncer-szlwf\" (UID: \"610063ae-dbb1-4a58-bacd-06bf90518168\") " pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:40:23.381499 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:23.381400 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 18:40:23.381499 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:23.381450 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/610063ae-dbb1-4a58-bacd-06bf90518168-original-pull-secret podName:610063ae-dbb1-4a58-bacd-06bf90518168 nodeName:}" failed. No retries permitted until 2026-07-08 18:40:55.381437327 +0000 UTC m=+66.218765013 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/610063ae-dbb1-4a58-bacd-06bf90518168-original-pull-secret") pod "global-pull-secret-syncer-szlwf" (UID: "610063ae-dbb1-4a58-bacd-06bf90518168") : object "kube-system"/"original-pull-secret" not registered Jul 08 18:40:23.381499 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:23.381400 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:40:23.381499 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:23.381488 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee39c730-7ca1-4260-afea-a884940c00fc-metrics-certs podName:ee39c730-7ca1-4260-afea-a884940c00fc nodeName:}" failed. No retries permitted until 2026-07-08 18:40:55.381477797 +0000 UTC m=+66.218805488 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee39c730-7ca1-4260-afea-a884940c00fc-metrics-certs") pod "network-metrics-daemon-252x5" (UID: "ee39c730-7ca1-4260-afea-a884940c00fc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 18:40:23.482239 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:23.482197 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zwd9\" (UniqueName: \"kubernetes.io/projected/ff43f31c-c779-406d-8352-4590967387a5-kube-api-access-2zwd9\") pod \"network-check-target-pwffr\" (UID: \"ff43f31c-c779-406d-8352-4590967387a5\") " pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:40:23.482425 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:23.482378 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 18:40:23.482425 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:23.482403 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 18:40:23.482425 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:23.482417 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2zwd9 for pod openshift-network-diagnostics/network-check-target-pwffr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:40:23.482606 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:23.482481 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff43f31c-c779-406d-8352-4590967387a5-kube-api-access-2zwd9 podName:ff43f31c-c779-406d-8352-4590967387a5 nodeName:}" failed. No retries permitted until 2026-07-08 18:40:55.482459241 +0000 UTC m=+66.319786932 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2zwd9" (UniqueName: "kubernetes.io/projected/ff43f31c-c779-406d-8352-4590967387a5-kube-api-access-2zwd9") pod "network-check-target-pwffr" (UID: "ff43f31c-c779-406d-8352-4590967387a5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 18:40:24.774225 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:24.774186 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:40:24.774225 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:24.774211 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:40:24.774225 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:24.774223 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:40:24.774830 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:24.774342 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlwf" podUID="610063ae-dbb1-4a58-bacd-06bf90518168" Jul 08 18:40:24.774830 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:24.774463 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-252x5" podUID="ee39c730-7ca1-4260-afea-a884940c00fc" Jul 08 18:40:24.774830 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:24.774548 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwffr" podUID="ff43f31c-c779-406d-8352-4590967387a5" Jul 08 18:40:26.774080 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:26.774051 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:40:26.774080 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:26.774085 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:40:26.774744 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:26.774085 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:40:26.774744 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:26.774151 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-252x5" podUID="ee39c730-7ca1-4260-afea-a884940c00fc" Jul 08 18:40:26.774744 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:26.774257 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pwffr" podUID="ff43f31c-c779-406d-8352-4590967387a5" Jul 08 18:40:26.774744 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:26.774325 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szlwf" podUID="610063ae-dbb1-4a58-bacd-06bf90518168" Jul 08 18:40:26.937920 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:26.937845 2575 generic.go:356] "Generic (PLEG): container finished" podID="1e7898c0-39bf-42c9-a65e-30409d5a5ea1" containerID="22e8365ced17a40a04a704d7828498bec28256f595ec6b05d7e7518f64dad32e" exitCode=0 Jul 08 18:40:26.937920 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:26.937893 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grt82" event={"ID":"1e7898c0-39bf-42c9-a65e-30409d5a5ea1","Type":"ContainerDied","Data":"22e8365ced17a40a04a704d7828498bec28256f595ec6b05d7e7518f64dad32e"} Jul 08 18:40:27.942683 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:27.942651 2575 generic.go:356] "Generic (PLEG): container finished" podID="1e7898c0-39bf-42c9-a65e-30409d5a5ea1" containerID="c9918302ff2cdf091c05d2b751ac6731627b8a8a102fccc3892a3f6819eb87eb" exitCode=0 Jul 08 18:40:27.943077 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:27.942711 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grt82" event={"ID":"1e7898c0-39bf-42c9-a65e-30409d5a5ea1","Type":"ContainerDied","Data":"c9918302ff2cdf091c05d2b751ac6731627b8a8a102fccc3892a3f6819eb87eb"} Jul 08 18:40:28.486745 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.486717 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-102.ec2.internal" event="NodeReady" Jul 08 18:40:28.486915 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.486835 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 08 18:40:28.535788 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.535757 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ljtbq"] Jul 08 18:40:28.547165 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.547140 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-l582k"] Jul 08 18:40:28.547331 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.547279 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ljtbq" Jul 08 18:40:28.550102 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.550085 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 08 18:40:28.550223 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.550101 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 08 18:40:28.550288 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.550260 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rmskx\"" Jul 08 18:40:28.557345 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.557328 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ljtbq"] Jul 08 18:40:28.557433 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.557358 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l582k"] Jul 08 18:40:28.557478 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.557433 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l582k" Jul 08 18:40:28.560199 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.560180 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 08 18:40:28.560321 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.560221 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 08 18:40:28.560321 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.560236 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 08 18:40:28.560419 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.560344 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nbwh9\"" Jul 08 18:40:28.720634 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.720593 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b2b7480-7b33-4fd9-9993-71441bbb6c78-metrics-tls\") pod \"dns-default-ljtbq\" (UID: \"0b2b7480-7b33-4fd9-9993-71441bbb6c78\") " pod="openshift-dns/dns-default-ljtbq" Jul 08 18:40:28.720841 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.720644 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jm4j\" (UniqueName: \"kubernetes.io/projected/a2d88ce3-6a8c-4b89-8bcd-e5a8ead15f44-kube-api-access-9jm4j\") pod \"ingress-canary-l582k\" (UID: \"a2d88ce3-6a8c-4b89-8bcd-e5a8ead15f44\") " pod="openshift-ingress-canary/ingress-canary-l582k" Jul 08 18:40:28.720841 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.720667 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0b2b7480-7b33-4fd9-9993-71441bbb6c78-config-volume\") pod \"dns-default-ljtbq\" (UID: \"0b2b7480-7b33-4fd9-9993-71441bbb6c78\") " pod="openshift-dns/dns-default-ljtbq" Jul 08 18:40:28.720841 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.720693 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b2b7480-7b33-4fd9-9993-71441bbb6c78-tmp-dir\") pod \"dns-default-ljtbq\" (UID: \"0b2b7480-7b33-4fd9-9993-71441bbb6c78\") " pod="openshift-dns/dns-default-ljtbq" Jul 08 18:40:28.720841 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.720771 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2d88ce3-6a8c-4b89-8bcd-e5a8ead15f44-cert\") pod \"ingress-canary-l582k\" (UID: \"a2d88ce3-6a8c-4b89-8bcd-e5a8ead15f44\") " pod="openshift-ingress-canary/ingress-canary-l582k" Jul 08 18:40:28.720841 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.720802 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv65l\" (UniqueName: \"kubernetes.io/projected/0b2b7480-7b33-4fd9-9993-71441bbb6c78-kube-api-access-lv65l\") pod \"dns-default-ljtbq\" (UID: \"0b2b7480-7b33-4fd9-9993-71441bbb6c78\") " pod="openshift-dns/dns-default-ljtbq" Jul 08 18:40:28.774007 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.773938 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:40:28.774136 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.773944 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:40:28.774136 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.773944 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:40:28.777360 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.777337 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 18:40:28.777597 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.777414 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 18:40:28.777597 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.777513 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 18:40:28.777597 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.777520 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-msx7w\"" Jul 08 18:40:28.777815 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.777798 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sc5vc\"" Jul 08 18:40:28.777864 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.777814 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 18:40:28.822000 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.821976 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv65l\" (UniqueName: \"kubernetes.io/projected/0b2b7480-7b33-4fd9-9993-71441bbb6c78-kube-api-access-lv65l\") pod \"dns-default-ljtbq\" (UID: \"0b2b7480-7b33-4fd9-9993-71441bbb6c78\") " pod="openshift-dns/dns-default-ljtbq" Jul 08 18:40:28.822136 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.822006 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b2b7480-7b33-4fd9-9993-71441bbb6c78-metrics-tls\") pod \"dns-default-ljtbq\" (UID: \"0b2b7480-7b33-4fd9-9993-71441bbb6c78\") " pod="openshift-dns/dns-default-ljtbq" Jul 08 18:40:28.822136 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.822026 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jm4j\" (UniqueName: \"kubernetes.io/projected/a2d88ce3-6a8c-4b89-8bcd-e5a8ead15f44-kube-api-access-9jm4j\") pod \"ingress-canary-l582k\" (UID: \"a2d88ce3-6a8c-4b89-8bcd-e5a8ead15f44\") " pod="openshift-ingress-canary/ingress-canary-l582k" Jul 08 18:40:28.822136 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.822046 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0b2b7480-7b33-4fd9-9993-71441bbb6c78-config-volume\") pod \"dns-default-ljtbq\" (UID: \"0b2b7480-7b33-4fd9-9993-71441bbb6c78\") " pod="openshift-dns/dns-default-ljtbq" Jul 08 18:40:28.822136 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.822073 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b2b7480-7b33-4fd9-9993-71441bbb6c78-tmp-dir\") pod \"dns-default-ljtbq\" (UID: \"0b2b7480-7b33-4fd9-9993-71441bbb6c78\") " pod="openshift-dns/dns-default-ljtbq" Jul 08 18:40:28.822136 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:28.822133 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 18:40:28.822360 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:28.822194 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2b7480-7b33-4fd9-9993-71441bbb6c78-metrics-tls podName:0b2b7480-7b33-4fd9-9993-71441bbb6c78 nodeName:}" failed. No retries permitted until 2026-07-08 18:40:29.32217371 +0000 UTC m=+40.159501413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b2b7480-7b33-4fd9-9993-71441bbb6c78-metrics-tls") pod "dns-default-ljtbq" (UID: "0b2b7480-7b33-4fd9-9993-71441bbb6c78") : secret "dns-default-metrics-tls" not found Jul 08 18:40:28.822360 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.822270 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2d88ce3-6a8c-4b89-8bcd-e5a8ead15f44-cert\") pod \"ingress-canary-l582k\" (UID: \"a2d88ce3-6a8c-4b89-8bcd-e5a8ead15f44\") " pod="openshift-ingress-canary/ingress-canary-l582k" Jul 08 18:40:28.822440 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.822403 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b2b7480-7b33-4fd9-9993-71441bbb6c78-tmp-dir\") pod \"dns-default-ljtbq\" (UID: \"0b2b7480-7b33-4fd9-9993-71441bbb6c78\") " pod="openshift-dns/dns-default-ljtbq" Jul 08 18:40:28.822691 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.822674 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0b2b7480-7b33-4fd9-9993-71441bbb6c78-config-volume\") pod \"dns-default-ljtbq\" (UID: \"0b2b7480-7b33-4fd9-9993-71441bbb6c78\") " pod="openshift-dns/dns-default-ljtbq" Jul 08 18:40:28.826359 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.826340 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2d88ce3-6a8c-4b89-8bcd-e5a8ead15f44-cert\") pod \"ingress-canary-l582k\" (UID: \"a2d88ce3-6a8c-4b89-8bcd-e5a8ead15f44\") " pod="openshift-ingress-canary/ingress-canary-l582k" Jul 08 18:40:28.831210 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.831191 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv65l\" (UniqueName: \"kubernetes.io/projected/0b2b7480-7b33-4fd9-9993-71441bbb6c78-kube-api-access-lv65l\") pod \"dns-default-ljtbq\" (UID: \"0b2b7480-7b33-4fd9-9993-71441bbb6c78\") " pod="openshift-dns/dns-default-ljtbq" Jul 08 18:40:28.831316 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.831258 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jm4j\" (UniqueName: \"kubernetes.io/projected/a2d88ce3-6a8c-4b89-8bcd-e5a8ead15f44-kube-api-access-9jm4j\") pod \"ingress-canary-l582k\" (UID: \"a2d88ce3-6a8c-4b89-8bcd-e5a8ead15f44\") " pod="openshift-ingress-canary/ingress-canary-l582k" Jul 08 18:40:28.865355 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.865326 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-l582k" Jul 08 18:40:28.948545 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.948516 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-grt82" event={"ID":"1e7898c0-39bf-42c9-a65e-30409d5a5ea1","Type":"ContainerStarted","Data":"48255fcb628d9112a4072ad067e9a314e50c6283f14c9330b02a15f839e6d022"} Jul 08 18:40:28.973943 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:28.973886 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-grt82" podStartSLOduration=8.194150199 podStartE2EDuration="38.973869591s" totalImagesPullingTime="30.779719392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:39:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:40:28.972050701 +0000 UTC m=+39.809378409" watchObservedRunningTime="2026-07-08 18:40:28.973869591 +0000 UTC m=+39.811197301" Jul 08 18:40:29.059830 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:29.059570 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-l582k"] Jul 08 18:40:29.063267 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:40:29.063233 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d88ce3_6a8c_4b89_8bcd_e5a8ead15f44.slice/crio-71f290db8b26a76a79e55298ac0f00637cbaca81a38f74ebfe9c69836d3eca60 WatchSource:0}: Error finding container 71f290db8b26a76a79e55298ac0f00637cbaca81a38f74ebfe9c69836d3eca60: Status 404 returned error can't find the container with id 71f290db8b26a76a79e55298ac0f00637cbaca81a38f74ebfe9c69836d3eca60 Jul 08 18:40:29.327120 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:29.327034 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b2b7480-7b33-4fd9-9993-71441bbb6c78-metrics-tls\") pod \"dns-default-ljtbq\" (UID: \"0b2b7480-7b33-4fd9-9993-71441bbb6c78\") " pod="openshift-dns/dns-default-ljtbq" Jul 08 18:40:29.327268 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:29.327138 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 18:40:29.327268 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:29.327191 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2b7480-7b33-4fd9-9993-71441bbb6c78-metrics-tls podName:0b2b7480-7b33-4fd9-9993-71441bbb6c78 nodeName:}" failed. No retries permitted until 2026-07-08 18:40:30.32717729 +0000 UTC m=+41.164504976 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b2b7480-7b33-4fd9-9993-71441bbb6c78-metrics-tls") pod "dns-default-ljtbq" (UID: "0b2b7480-7b33-4fd9-9993-71441bbb6c78") : secret "dns-default-metrics-tls" not found Jul 08 18:40:29.952259 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:29.952221 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l582k" event={"ID":"a2d88ce3-6a8c-4b89-8bcd-e5a8ead15f44","Type":"ContainerStarted","Data":"71f290db8b26a76a79e55298ac0f00637cbaca81a38f74ebfe9c69836d3eca60"} Jul 08 18:40:30.334353 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:30.334263 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b2b7480-7b33-4fd9-9993-71441bbb6c78-metrics-tls\") pod \"dns-default-ljtbq\" (UID: \"0b2b7480-7b33-4fd9-9993-71441bbb6c78\") " pod="openshift-dns/dns-default-ljtbq" Jul 08 18:40:30.334509 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:30.334418 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 18:40:30.334509 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:30.334494 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2b7480-7b33-4fd9-9993-71441bbb6c78-metrics-tls podName:0b2b7480-7b33-4fd9-9993-71441bbb6c78 nodeName:}" failed. No retries permitted until 2026-07-08 18:40:32.334478356 +0000 UTC m=+43.171806042 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b2b7480-7b33-4fd9-9993-71441bbb6c78-metrics-tls") pod "dns-default-ljtbq" (UID: "0b2b7480-7b33-4fd9-9993-71441bbb6c78") : secret "dns-default-metrics-tls" not found Jul 08 18:40:32.348230 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:32.348206 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b2b7480-7b33-4fd9-9993-71441bbb6c78-metrics-tls\") pod \"dns-default-ljtbq\" (UID: \"0b2b7480-7b33-4fd9-9993-71441bbb6c78\") " pod="openshift-dns/dns-default-ljtbq" Jul 08 18:40:32.348516 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:32.348345 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 18:40:32.348516 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:32.348394 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2b7480-7b33-4fd9-9993-71441bbb6c78-metrics-tls podName:0b2b7480-7b33-4fd9-9993-71441bbb6c78 nodeName:}" failed. No retries permitted until 2026-07-08 18:40:36.348382015 +0000 UTC m=+47.185709701 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b2b7480-7b33-4fd9-9993-71441bbb6c78-metrics-tls") pod "dns-default-ljtbq" (UID: "0b2b7480-7b33-4fd9-9993-71441bbb6c78") : secret "dns-default-metrics-tls" not found Jul 08 18:40:32.960014 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:32.959982 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-l582k" event={"ID":"a2d88ce3-6a8c-4b89-8bcd-e5a8ead15f44","Type":"ContainerStarted","Data":"d0972775e9bfa5d0917ccbde5aa32ffda3ea4abe2c17ab3502be029b0e06a6a8"} Jul 08 18:40:32.989602 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:32.989553 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-l582k" podStartSLOduration=1.756880567 podStartE2EDuration="4.989539841s" totalImagesPullingTime="3.232659274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:40:32.98913439 +0000 UTC m=+43.826462098" watchObservedRunningTime="2026-07-08 18:40:32.989539841 +0000 UTC m=+43.826867527" Jul 08 18:40:36.375938 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:36.375752 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b2b7480-7b33-4fd9-9993-71441bbb6c78-metrics-tls\") pod \"dns-default-ljtbq\" (UID: \"0b2b7480-7b33-4fd9-9993-71441bbb6c78\") " pod="openshift-dns/dns-default-ljtbq" Jul 08 18:40:36.375938 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:36.375913 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 18:40:36.376486 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:36.376005 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2b7480-7b33-4fd9-9993-71441bbb6c78-metrics-tls podName:0b2b7480-7b33-4fd9-9993-71441bbb6c78 nodeName:}" failed. No retries permitted until 2026-07-08 18:40:44.375984964 +0000 UTC m=+55.213312653 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b2b7480-7b33-4fd9-9993-71441bbb6c78-metrics-tls") pod "dns-default-ljtbq" (UID: "0b2b7480-7b33-4fd9-9993-71441bbb6c78") : secret "dns-default-metrics-tls" not found Jul 08 18:40:39.600406 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.600363 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2wffx"] Jul 08 18:40:39.611123 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.611097 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7drlq"] Jul 08 18:40:39.611250 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.611218 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2wffx" Jul 08 18:40:39.615373 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.615323 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 08 18:40:39.615509 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.615492 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 08 18:40:39.615584 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.615549 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-sbtk8\"" Jul 08 18:40:39.615731 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.615718 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 08 18:40:39.615787 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.615754 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 08 18:40:39.619813 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.619763 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-zpzc6"] Jul 08 18:40:39.619919 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.619873 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7drlq" Jul 08 18:40:39.622857 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.622838 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 08 18:40:39.622953 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.622839 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 08 18:40:39.624085 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.624033 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cg9vg\"" Jul 08 18:40:39.624188 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.624086 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 08 18:40:39.624188 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.624102 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 08 18:40:39.636514 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.636494 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-p4gn9"] Jul 08 18:40:39.636629 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.636540 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-zpzc6" Jul 08 18:40:39.639559 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.639542 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 08 18:40:39.639672 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.639564 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-hznk5\"" Jul 08 18:40:39.639819 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.639804 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 08 18:40:39.652985 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.652962 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2wffx"] Jul 08 18:40:39.653101 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.652991 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-zpzc6"] Jul 08 18:40:39.653101 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.653004 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7drlq"] Jul 08 18:40:39.653101 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.653016 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-p4gn9"] Jul 08 18:40:39.653101 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.653072 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-p4gn9" Jul 08 18:40:39.655603 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.655582 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 08 18:40:39.655713 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.655582 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-sm6wg\"" Jul 08 18:40:39.655851 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.655829 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 08 18:40:39.655920 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.655849 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 08 18:40:39.655988 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.655924 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 08 18:40:39.705906 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.705877 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-qcqpm"] Jul 08 18:40:39.720733 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.720706 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-88bdr"] Jul 08 18:40:39.720875 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.720812 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-qcqpm" Jul 08 18:40:39.726937 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.726912 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 08 18:40:39.727363 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.727338 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-qzbn7\"" Jul 08 18:40:39.727958 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.727938 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 08 18:40:39.729780 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.729762 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-qcqpm"] Jul 08 18:40:39.729873 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.729847 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-88bdr" Jul 08 18:40:39.741996 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.741978 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 08 18:40:39.743105 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.743090 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 08 18:40:39.744461 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.744443 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 08 18:40:39.744832 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.744816 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-z2lrl\"" Jul 08 18:40:39.747325 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.747306 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-88bdr"] Jul 08 18:40:39.798837 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.798804 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aedf6d82-9258-424b-a9f0-af1f0154058b-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-2wffx\" (UID: \"aedf6d82-9258-424b-a9f0-af1f0154058b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2wffx" Jul 08 18:40:39.798837 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.798836 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tmgn\" (UniqueName: \"kubernetes.io/projected/4baf0565-cc14-47a2-823c-c02c1d82e981-kube-api-access-5tmgn\") pod \"volume-data-source-validator-5d456fdbcd-zpzc6\" (UID: \"4baf0565-cc14-47a2-823c-c02c1d82e981\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-zpzc6" Jul 08 18:40:39.799025 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.798882 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0e3be5af-3630-43f1-80b1-daf23fb9b78b-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-p4gn9\" (UID: \"0e3be5af-3630-43f1-80b1-daf23fb9b78b\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-p4gn9" Jul 08 18:40:39.799025 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.798980 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-7drlq\" (UID: \"3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7drlq" Jul 08 18:40:39.799025 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.799010 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aedf6d82-9258-424b-a9f0-af1f0154058b-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-2wffx\" (UID: \"aedf6d82-9258-424b-a9f0-af1f0154058b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2wffx" Jul 08 18:40:39.799153 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.799033 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rms5j\" (UniqueName: \"kubernetes.io/projected/3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7-kube-api-access-rms5j\") pod \"cluster-monitoring-operator-64786d9cbb-7drlq\" (UID: \"3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7drlq" Jul 08 18:40:39.799153 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.799066 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0e3be5af-3630-43f1-80b1-daf23fb9b78b-config\") pod \"service-ca-operator-5d46d9ffd4-p4gn9\" (UID: \"0e3be5af-3630-43f1-80b1-daf23fb9b78b\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-p4gn9" Jul 08 18:40:39.799153 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.799084 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t789j\" (UniqueName: \"kubernetes.io/projected/0e3be5af-3630-43f1-80b1-daf23fb9b78b-kube-api-access-t789j\") pod \"service-ca-operator-5d46d9ffd4-p4gn9\" (UID: \"0e3be5af-3630-43f1-80b1-daf23fb9b78b\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-p4gn9" Jul 08 18:40:39.799153 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.799103 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pkq9\" (UniqueName: \"kubernetes.io/projected/aedf6d82-9258-424b-a9f0-af1f0154058b-kube-api-access-6pkq9\") pod \"kube-storage-version-migrator-operator-846bf668f7-2wffx\" (UID: \"aedf6d82-9258-424b-a9f0-af1f0154058b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2wffx" Jul 08 18:40:39.799153 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.799136 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-7drlq\" (UID: \"3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7drlq" Jul 08 18:40:39.900173 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.900142 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aedf6d82-9258-424b-a9f0-af1f0154058b-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-2wffx\" (UID: \"aedf6d82-9258-424b-a9f0-af1f0154058b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2wffx" Jul 08 18:40:39.900362 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.900186 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tmgn\" (UniqueName: \"kubernetes.io/projected/4baf0565-cc14-47a2-823c-c02c1d82e981-kube-api-access-5tmgn\") pod \"volume-data-source-validator-5d456fdbcd-zpzc6\" (UID: \"4baf0565-cc14-47a2-823c-c02c1d82e981\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-zpzc6" Jul 08 18:40:39.900362 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.900218 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff5zd\" (UniqueName: \"kubernetes.io/projected/6cb0e900-30c7-416f-a732-2f590e174b8f-kube-api-access-ff5zd\") pod \"cluster-samples-operator-56fcf86bf-88bdr\" (UID: \"6cb0e900-30c7-416f-a732-2f590e174b8f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-88bdr" Jul 08 18:40:39.900362 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.900248 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0e3be5af-3630-43f1-80b1-daf23fb9b78b-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-p4gn9\" (UID: \"0e3be5af-3630-43f1-80b1-daf23fb9b78b\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-p4gn9" Jul 08 18:40:39.900362 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.900315 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-7drlq\" (UID: \"3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7drlq" Jul 08 18:40:39.900362 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.900338 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aedf6d82-9258-424b-a9f0-af1f0154058b-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-2wffx\" (UID: \"aedf6d82-9258-424b-a9f0-af1f0154058b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2wffx" Jul 08 18:40:39.900579 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.900367 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rms5j\" (UniqueName: \"kubernetes.io/projected/3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7-kube-api-access-rms5j\") pod \"cluster-monitoring-operator-64786d9cbb-7drlq\" (UID: \"3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7drlq" Jul 08 18:40:39.900579 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.900392 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f18a01b1-7095-480e-ac45-e3a3cce3cfe5-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-qcqpm\" (UID: \"f18a01b1-7095-480e-ac45-e3a3cce3cfe5\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-qcqpm" Jul 08 18:40:39.900579 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.900429 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0e3be5af-3630-43f1-80b1-daf23fb9b78b-config\") pod \"service-ca-operator-5d46d9ffd4-p4gn9\" (UID: \"0e3be5af-3630-43f1-80b1-daf23fb9b78b\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-p4gn9" Jul 08 18:40:39.900579 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.900453 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t789j\" (UniqueName: \"kubernetes.io/projected/0e3be5af-3630-43f1-80b1-daf23fb9b78b-kube-api-access-t789j\") pod \"service-ca-operator-5d46d9ffd4-p4gn9\" (UID: \"0e3be5af-3630-43f1-80b1-daf23fb9b78b\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-p4gn9" Jul 08 18:40:39.900579 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.900482 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pkq9\" (UniqueName: \"kubernetes.io/projected/aedf6d82-9258-424b-a9f0-af1f0154058b-kube-api-access-6pkq9\") pod \"kube-storage-version-migrator-operator-846bf668f7-2wffx\" (UID: \"aedf6d82-9258-424b-a9f0-af1f0154058b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2wffx" Jul 08 18:40:39.900579 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.900505 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f18a01b1-7095-480e-ac45-e3a3cce3cfe5-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-qcqpm\" (UID: \"f18a01b1-7095-480e-ac45-e3a3cce3cfe5\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-qcqpm" Jul 08 18:40:39.900579 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.900539 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-7drlq\" (UID: \"3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7drlq" Jul 08 18:40:39.900579 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.900563 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6cb0e900-30c7-416f-a732-2f590e174b8f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-88bdr\" (UID: \"6cb0e900-30c7-416f-a732-2f590e174b8f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-88bdr" Jul 08 18:40:39.900955 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:39.900889 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 18:40:39.900955 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.900921 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aedf6d82-9258-424b-a9f0-af1f0154058b-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-2wffx\" (UID: \"aedf6d82-9258-424b-a9f0-af1f0154058b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2wffx" Jul 08 18:40:39.900955 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:39.900947 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7-cluster-monitoring-operator-tls podName:3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7 nodeName:}" failed. No retries permitted until 2026-07-08 18:40:40.400930357 +0000 UTC m=+51.238258060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-7drlq" (UID: "3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7") : secret "cluster-monitoring-operator-tls" not found Jul 08 18:40:39.900955 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.900946 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0e3be5af-3630-43f1-80b1-daf23fb9b78b-config\") pod \"service-ca-operator-5d46d9ffd4-p4gn9\" (UID: \"0e3be5af-3630-43f1-80b1-daf23fb9b78b\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-p4gn9" Jul 08 18:40:39.901148 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.901070 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-7drlq\" (UID: \"3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7drlq" Jul 08 18:40:39.904377 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.904359 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0e3be5af-3630-43f1-80b1-daf23fb9b78b-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-p4gn9\" (UID: \"0e3be5af-3630-43f1-80b1-daf23fb9b78b\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-p4gn9" Jul 08 18:40:39.911537 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.911505 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pkq9\" (UniqueName: \"kubernetes.io/projected/aedf6d82-9258-424b-a9f0-af1f0154058b-kube-api-access-6pkq9\") pod \"kube-storage-version-migrator-operator-846bf668f7-2wffx\" (UID: \"aedf6d82-9258-424b-a9f0-af1f0154058b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2wffx" Jul 08 18:40:39.911647 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.911597 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tmgn\" (UniqueName: \"kubernetes.io/projected/4baf0565-cc14-47a2-823c-c02c1d82e981-kube-api-access-5tmgn\") pod \"volume-data-source-validator-5d456fdbcd-zpzc6\" (UID: \"4baf0565-cc14-47a2-823c-c02c1d82e981\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-zpzc6" Jul 08 18:40:39.911720 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.911700 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t789j\" (UniqueName: \"kubernetes.io/projected/0e3be5af-3630-43f1-80b1-daf23fb9b78b-kube-api-access-t789j\") pod \"service-ca-operator-5d46d9ffd4-p4gn9\" (UID: \"0e3be5af-3630-43f1-80b1-daf23fb9b78b\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-p4gn9" Jul 08 18:40:39.911934 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.911915 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rms5j\" (UniqueName: \"kubernetes.io/projected/3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7-kube-api-access-rms5j\") pod \"cluster-monitoring-operator-64786d9cbb-7drlq\" (UID: \"3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7drlq" Jul 08 18:40:39.912580 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.912562 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aedf6d82-9258-424b-a9f0-af1f0154058b-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-2wffx\" (UID: \"aedf6d82-9258-424b-a9f0-af1f0154058b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2wffx" Jul 08 18:40:39.921655 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.921632 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2wffx" Jul 08 18:40:39.944581 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.944560 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-zpzc6" Jul 08 18:40:39.960893 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:39.960865 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-p4gn9" Jul 08 18:40:40.007639 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:40.005767 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f18a01b1-7095-480e-ac45-e3a3cce3cfe5-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-qcqpm\" (UID: \"f18a01b1-7095-480e-ac45-e3a3cce3cfe5\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-qcqpm" Jul 08 18:40:40.007639 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:40.005844 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f18a01b1-7095-480e-ac45-e3a3cce3cfe5-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-qcqpm\" (UID: \"f18a01b1-7095-480e-ac45-e3a3cce3cfe5\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-qcqpm" Jul 08 18:40:40.007639 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:40.005894 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6cb0e900-30c7-416f-a732-2f590e174b8f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-88bdr\" (UID: \"6cb0e900-30c7-416f-a732-2f590e174b8f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-88bdr" Jul 08 18:40:40.007639 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:40.005949 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff5zd\" (UniqueName: \"kubernetes.io/projected/6cb0e900-30c7-416f-a732-2f590e174b8f-kube-api-access-ff5zd\") pod \"cluster-samples-operator-56fcf86bf-88bdr\" (UID: \"6cb0e900-30c7-416f-a732-2f590e174b8f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-88bdr" Jul 08 18:40:40.007639 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:40.007103 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f18a01b1-7095-480e-ac45-e3a3cce3cfe5-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-qcqpm\" (UID: \"f18a01b1-7095-480e-ac45-e3a3cce3cfe5\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-qcqpm" Jul 08 18:40:40.007639 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:40.007216 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 18:40:40.007639 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:40.007280 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f18a01b1-7095-480e-ac45-e3a3cce3cfe5-networking-console-plugin-cert podName:f18a01b1-7095-480e-ac45-e3a3cce3cfe5 nodeName:}" failed. No retries permitted until 2026-07-08 18:40:40.507260566 +0000 UTC m=+51.344588257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f18a01b1-7095-480e-ac45-e3a3cce3cfe5-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-qcqpm" (UID: "f18a01b1-7095-480e-ac45-e3a3cce3cfe5") : secret "networking-console-plugin-cert" not found Jul 08 18:40:40.007639 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:40.007553 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 18:40:40.007639 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:40.007595 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb0e900-30c7-416f-a732-2f590e174b8f-samples-operator-tls podName:6cb0e900-30c7-416f-a732-2f590e174b8f nodeName:}" failed. No retries permitted until 2026-07-08 18:40:40.507581501 +0000 UTC m=+51.344909196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6cb0e900-30c7-416f-a732-2f590e174b8f-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-88bdr" (UID: "6cb0e900-30c7-416f-a732-2f590e174b8f") : secret "samples-operator-tls" not found Jul 08 18:40:40.031832 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:40.031795 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff5zd\" (UniqueName: \"kubernetes.io/projected/6cb0e900-30c7-416f-a732-2f590e174b8f-kube-api-access-ff5zd\") pod \"cluster-samples-operator-56fcf86bf-88bdr\" (UID: \"6cb0e900-30c7-416f-a732-2f590e174b8f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-88bdr" Jul 08 18:40:40.099772 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:40.099736 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2wffx"] Jul 08 18:40:40.103851 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:40:40.103812 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaedf6d82_9258_424b_a9f0_af1f0154058b.slice/crio-d8fa0d853031dc32429212e61f47b1497cba7048b90e801c47713157ec71d001 WatchSource:0}: Error finding container d8fa0d853031dc32429212e61f47b1497cba7048b90e801c47713157ec71d001: Status 404 returned error can't find the container with id d8fa0d853031dc32429212e61f47b1497cba7048b90e801c47713157ec71d001 Jul 08 18:40:40.121866 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:40.121838 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-zpzc6"] Jul 08 18:40:40.124431 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:40:40.124409 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4baf0565_cc14_47a2_823c_c02c1d82e981.slice/crio-c33ca75157f2736cfc5326a8e8807e7c3759cd822f99f603a00ee6aa44542f25 WatchSource:0}: Error finding container c33ca75157f2736cfc5326a8e8807e7c3759cd822f99f603a00ee6aa44542f25: Status 404 returned error can't find the container with id c33ca75157f2736cfc5326a8e8807e7c3759cd822f99f603a00ee6aa44542f25 Jul 08 18:40:40.141814 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:40.141791 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-p4gn9"] Jul 08 18:40:40.144881 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:40:40.144858 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3be5af_3630_43f1_80b1_daf23fb9b78b.slice/crio-e96cf27f040f4779d979f871615590671f6473835a4474440c187823af27c188 WatchSource:0}: Error finding container e96cf27f040f4779d979f871615590671f6473835a4474440c187823af27c188: Status 404 returned error can't find the container with id e96cf27f040f4779d979f871615590671f6473835a4474440c187823af27c188 Jul 08 18:40:40.408431 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:40.408384 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-7drlq\" (UID: \"3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7drlq" Jul 08 18:40:40.408580 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:40.408527 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 18:40:40.408634 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:40.408594 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7-cluster-monitoring-operator-tls podName:3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7 nodeName:}" failed. No retries permitted until 2026-07-08 18:40:41.40857871 +0000 UTC m=+52.245906428 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-7drlq" (UID: "3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7") : secret "cluster-monitoring-operator-tls" not found Jul 08 18:40:40.509844 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:40.509378 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f18a01b1-7095-480e-ac45-e3a3cce3cfe5-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-qcqpm\" (UID: \"f18a01b1-7095-480e-ac45-e3a3cce3cfe5\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-qcqpm" Jul 08 18:40:40.509844 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:40.509460 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6cb0e900-30c7-416f-a732-2f590e174b8f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-88bdr\" (UID: \"6cb0e900-30c7-416f-a732-2f590e174b8f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-88bdr" Jul 08 18:40:40.509844 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:40.509735 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 18:40:40.509844 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:40.509779 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 18:40:40.509844 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:40.509827 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb0e900-30c7-416f-a732-2f590e174b8f-samples-operator-tls podName:6cb0e900-30c7-416f-a732-2f590e174b8f nodeName:}" failed. No retries permitted until 2026-07-08 18:40:41.509795937 +0000 UTC m=+52.347123638 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6cb0e900-30c7-416f-a732-2f590e174b8f-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-88bdr" (UID: "6cb0e900-30c7-416f-a732-2f590e174b8f") : secret "samples-operator-tls" not found Jul 08 18:40:40.509844 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:40.509846 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f18a01b1-7095-480e-ac45-e3a3cce3cfe5-networking-console-plugin-cert podName:f18a01b1-7095-480e-ac45-e3a3cce3cfe5 nodeName:}" failed. No retries permitted until 2026-07-08 18:40:41.5098369 +0000 UTC m=+52.347164591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f18a01b1-7095-480e-ac45-e3a3cce3cfe5-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-qcqpm" (UID: "f18a01b1-7095-480e-ac45-e3a3cce3cfe5") : secret "networking-console-plugin-cert" not found Jul 08 18:40:40.980781 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:40.980747 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-p4gn9" event={"ID":"0e3be5af-3630-43f1-80b1-daf23fb9b78b","Type":"ContainerStarted","Data":"e96cf27f040f4779d979f871615590671f6473835a4474440c187823af27c188"} Jul 08 18:40:40.981890 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:40.981859 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-zpzc6" event={"ID":"4baf0565-cc14-47a2-823c-c02c1d82e981","Type":"ContainerStarted","Data":"c33ca75157f2736cfc5326a8e8807e7c3759cd822f99f603a00ee6aa44542f25"} Jul 08 18:40:40.982917 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:40.982892 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2wffx" event={"ID":"aedf6d82-9258-424b-a9f0-af1f0154058b","Type":"ContainerStarted","Data":"d8fa0d853031dc32429212e61f47b1497cba7048b90e801c47713157ec71d001"} Jul 08 18:40:41.416318 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:41.416263 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-7drlq\" (UID: \"3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7drlq" Jul 08 18:40:41.416541 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:41.416517 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 18:40:41.416620 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:41.416592 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7-cluster-monitoring-operator-tls podName:3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7 nodeName:}" failed. No retries permitted until 2026-07-08 18:40:43.41657187 +0000 UTC m=+54.253899573 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-7drlq" (UID: "3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7") : secret "cluster-monitoring-operator-tls" not found Jul 08 18:40:41.516954 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:41.516906 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f18a01b1-7095-480e-ac45-e3a3cce3cfe5-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-qcqpm\" (UID: \"f18a01b1-7095-480e-ac45-e3a3cce3cfe5\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-qcqpm" Jul 08 18:40:41.517144 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:41.516970 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6cb0e900-30c7-416f-a732-2f590e174b8f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-88bdr\" (UID: \"6cb0e900-30c7-416f-a732-2f590e174b8f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-88bdr" Jul 08 18:40:41.517144 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:41.517056 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 18:40:41.517144 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:41.517117 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 18:40:41.517144 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:41.517142 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f18a01b1-7095-480e-ac45-e3a3cce3cfe5-networking-console-plugin-cert podName:f18a01b1-7095-480e-ac45-e3a3cce3cfe5 nodeName:}" failed. No retries permitted until 2026-07-08 18:40:43.517118121 +0000 UTC m=+54.354445830 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f18a01b1-7095-480e-ac45-e3a3cce3cfe5-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-qcqpm" (UID: "f18a01b1-7095-480e-ac45-e3a3cce3cfe5") : secret "networking-console-plugin-cert" not found Jul 08 18:40:41.517388 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:41.517177 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb0e900-30c7-416f-a732-2f590e174b8f-samples-operator-tls podName:6cb0e900-30c7-416f-a732-2f590e174b8f nodeName:}" failed. No retries permitted until 2026-07-08 18:40:43.517162639 +0000 UTC m=+54.354490326 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6cb0e900-30c7-416f-a732-2f590e174b8f-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-88bdr" (UID: "6cb0e900-30c7-416f-a732-2f590e174b8f") : secret "samples-operator-tls" not found Jul 08 18:40:42.989095 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:42.989053 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-zpzc6" event={"ID":"4baf0565-cc14-47a2-823c-c02c1d82e981","Type":"ContainerStarted","Data":"79e9125379b038f222c7dab16ba88cf2728f2d35f5b65305921b5b93505615c4"} Jul 08 18:40:43.009569 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:43.009516 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-zpzc6" podStartSLOduration=2.127786795 podStartE2EDuration="4.009500389s" totalImagesPullingTime="1.881713594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:40:43.009017845 +0000 UTC m=+53.846345554" watchObservedRunningTime="2026-07-08 18:40:43.009500389 +0000 UTC m=+53.846828096" Jul 08 18:40:43.429582 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:43.429533 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-7drlq\" (UID: \"3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7drlq" Jul 08 18:40:43.429767 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:43.429696 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 18:40:43.429832 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:43.429772 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7-cluster-monitoring-operator-tls podName:3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7 nodeName:}" failed. No retries permitted until 2026-07-08 18:40:47.429754299 +0000 UTC m=+58.267082002 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-7drlq" (UID: "3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7") : secret "cluster-monitoring-operator-tls" not found Jul 08 18:40:43.530809 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:43.530766 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6cb0e900-30c7-416f-a732-2f590e174b8f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-88bdr\" (UID: \"6cb0e900-30c7-416f-a732-2f590e174b8f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-88bdr" Jul 08 18:40:43.530973 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:43.530914 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f18a01b1-7095-480e-ac45-e3a3cce3cfe5-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-qcqpm\" (UID: \"f18a01b1-7095-480e-ac45-e3a3cce3cfe5\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-qcqpm" Jul 08 18:40:43.530973 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:43.530927 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 18:40:43.531089 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:43.530993 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb0e900-30c7-416f-a732-2f590e174b8f-samples-operator-tls podName:6cb0e900-30c7-416f-a732-2f590e174b8f nodeName:}" failed. No retries permitted until 2026-07-08 18:40:47.530973933 +0000 UTC m=+58.368301637 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6cb0e900-30c7-416f-a732-2f590e174b8f-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-88bdr" (UID: "6cb0e900-30c7-416f-a732-2f590e174b8f") : secret "samples-operator-tls" not found Jul 08 18:40:43.531089 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:43.531012 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 18:40:43.531089 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:43.531050 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f18a01b1-7095-480e-ac45-e3a3cce3cfe5-networking-console-plugin-cert podName:f18a01b1-7095-480e-ac45-e3a3cce3cfe5 nodeName:}" failed. No retries permitted until 2026-07-08 18:40:47.53103806 +0000 UTC m=+58.368365758 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f18a01b1-7095-480e-ac45-e3a3cce3cfe5-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-qcqpm" (UID: "f18a01b1-7095-480e-ac45-e3a3cce3cfe5") : secret "networking-console-plugin-cert" not found Jul 08 18:40:43.992813 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:43.992770 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2wffx" event={"ID":"aedf6d82-9258-424b-a9f0-af1f0154058b","Type":"ContainerStarted","Data":"f6d9faae29caca4b72e3ba0e7503a56ccc4fbfed817dc4c2e1c3d498aaa9d6f9"} Jul 08 18:40:44.016615 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:44.016534 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2wffx" podStartSLOduration=1.362623026 podStartE2EDuration="5.016518738s" totalImagesPullingTime="3.653895712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:40:44.015836477 +0000 UTC m=+54.853164187" watchObservedRunningTime="2026-07-08 18:40:44.016518738 +0000 UTC m=+54.853846487" Jul 08 18:40:44.436124 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:44.436089 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b2b7480-7b33-4fd9-9993-71441bbb6c78-metrics-tls\") pod \"dns-default-ljtbq\" (UID: \"0b2b7480-7b33-4fd9-9993-71441bbb6c78\") " pod="openshift-dns/dns-default-ljtbq" Jul 08 18:40:44.436335 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:44.436270 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 18:40:44.436406 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:44.436369 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2b7480-7b33-4fd9-9993-71441bbb6c78-metrics-tls podName:0b2b7480-7b33-4fd9-9993-71441bbb6c78 nodeName:}" failed. No retries permitted until 2026-07-08 18:41:00.436346703 +0000 UTC m=+71.273674407 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0b2b7480-7b33-4fd9-9993-71441bbb6c78-metrics-tls") pod "dns-default-ljtbq" (UID: "0b2b7480-7b33-4fd9-9993-71441bbb6c78") : secret "dns-default-metrics-tls" not found Jul 08 18:40:44.520441 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:44.520401 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lz2g7"] Jul 08 18:40:44.533602 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:44.533578 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lz2g7" Jul 08 18:40:44.536765 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:44.536740 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23d40abf-d3f0-4faa-9768-57e6486a5a6c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lz2g7\" (UID: \"23d40abf-d3f0-4faa-9768-57e6486a5a6c\") " pod="openshift-insights/insights-runtime-extractor-lz2g7" Jul 08 18:40:44.536902 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:44.536773 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23d40abf-d3f0-4faa-9768-57e6486a5a6c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lz2g7\" (UID: \"23d40abf-d3f0-4faa-9768-57e6486a5a6c\") " pod="openshift-insights/insights-runtime-extractor-lz2g7" Jul 08 18:40:44.536902 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:44.536853 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/23d40abf-d3f0-4faa-9768-57e6486a5a6c-crio-socket\") pod \"insights-runtime-extractor-lz2g7\" (UID: \"23d40abf-d3f0-4faa-9768-57e6486a5a6c\") " pod="openshift-insights/insights-runtime-extractor-lz2g7" Jul 08 18:40:44.536902 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:44.536879 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk7k7\" (UniqueName: \"kubernetes.io/projected/23d40abf-d3f0-4faa-9768-57e6486a5a6c-kube-api-access-nk7k7\") pod \"insights-runtime-extractor-lz2g7\" (UID: \"23d40abf-d3f0-4faa-9768-57e6486a5a6c\") " pod="openshift-insights/insights-runtime-extractor-lz2g7" Jul 08 18:40:44.537043 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:44.537022 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/23d40abf-d3f0-4faa-9768-57e6486a5a6c-data-volume\") pod \"insights-runtime-extractor-lz2g7\" (UID: \"23d40abf-d3f0-4faa-9768-57e6486a5a6c\") " pod="openshift-insights/insights-runtime-extractor-lz2g7" Jul 08 18:40:44.543909 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:44.543892 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 08 18:40:44.544172 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:44.544152 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 08 18:40:44.544232 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:44.544195 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 08 18:40:44.544318 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:44.544287 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 08 18:40:44.544549 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:44.544537 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jtszc\"" Jul 08 18:40:44.557832 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:44.557804 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lz2g7"] Jul 08 18:40:44.637646 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:44.637608 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/23d40abf-d3f0-4faa-9768-57e6486a5a6c-data-volume\") pod \"insights-runtime-extractor-lz2g7\" (UID: \"23d40abf-d3f0-4faa-9768-57e6486a5a6c\") " pod="openshift-insights/insights-runtime-extractor-lz2g7" Jul 08 18:40:44.637841 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:44.637660 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23d40abf-d3f0-4faa-9768-57e6486a5a6c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lz2g7\" (UID: \"23d40abf-d3f0-4faa-9768-57e6486a5a6c\") " pod="openshift-insights/insights-runtime-extractor-lz2g7" Jul 08 18:40:44.637841 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:44.637747 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 18:40:44.637841 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:44.637779 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23d40abf-d3f0-4faa-9768-57e6486a5a6c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lz2g7\" (UID: \"23d40abf-d3f0-4faa-9768-57e6486a5a6c\") " pod="openshift-insights/insights-runtime-extractor-lz2g7" Jul 08 18:40:44.637841 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:44.637827 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23d40abf-d3f0-4faa-9768-57e6486a5a6c-kube-rbac-proxy-cm podName:23d40abf-d3f0-4faa-9768-57e6486a5a6c nodeName:}" failed. No retries permitted until 2026-07-08 18:40:45.137804399 +0000 UTC m=+55.975132099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/23d40abf-d3f0-4faa-9768-57e6486a5a6c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lz2g7" (UID: "23d40abf-d3f0-4faa-9768-57e6486a5a6c") : configmap "kube-rbac-proxy" not found Jul 08 18:40:44.638066 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:44.637889 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 18:40:44.638066 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:44.637934 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/23d40abf-d3f0-4faa-9768-57e6486a5a6c-crio-socket\") pod \"insights-runtime-extractor-lz2g7\" (UID: \"23d40abf-d3f0-4faa-9768-57e6486a5a6c\") " pod="openshift-insights/insights-runtime-extractor-lz2g7" Jul 08 18:40:44.638066 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:44.637945 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23d40abf-d3f0-4faa-9768-57e6486a5a6c-insights-runtime-extractor-tls podName:23d40abf-d3f0-4faa-9768-57e6486a5a6c nodeName:}" failed. No retries permitted until 2026-07-08 18:40:45.137926934 +0000 UTC m=+55.975254642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23d40abf-d3f0-4faa-9768-57e6486a5a6c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lz2g7" (UID: "23d40abf-d3f0-4faa-9768-57e6486a5a6c") : secret "insights-runtime-extractor-tls" not found Jul 08 18:40:44.638066 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:44.637981 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk7k7\" (UniqueName: \"kubernetes.io/projected/23d40abf-d3f0-4faa-9768-57e6486a5a6c-kube-api-access-nk7k7\") pod \"insights-runtime-extractor-lz2g7\" (UID: \"23d40abf-d3f0-4faa-9768-57e6486a5a6c\") " pod="openshift-insights/insights-runtime-extractor-lz2g7" Jul 08 18:40:44.638066 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:44.638001 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/23d40abf-d3f0-4faa-9768-57e6486a5a6c-data-volume\") pod \"insights-runtime-extractor-lz2g7\" (UID: \"23d40abf-d3f0-4faa-9768-57e6486a5a6c\") " pod="openshift-insights/insights-runtime-extractor-lz2g7" Jul 08 18:40:44.638271 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:44.638120 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/23d40abf-d3f0-4faa-9768-57e6486a5a6c-crio-socket\") pod \"insights-runtime-extractor-lz2g7\" (UID: \"23d40abf-d3f0-4faa-9768-57e6486a5a6c\") " pod="openshift-insights/insights-runtime-extractor-lz2g7" Jul 08 18:40:44.648999 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:44.648967 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk7k7\" (UniqueName: \"kubernetes.io/projected/23d40abf-d3f0-4faa-9768-57e6486a5a6c-kube-api-access-nk7k7\") pod \"insights-runtime-extractor-lz2g7\" (UID: \"23d40abf-d3f0-4faa-9768-57e6486a5a6c\") " pod="openshift-insights/insights-runtime-extractor-lz2g7" Jul 08 18:40:45.001462 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:45.001420 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-p4gn9" event={"ID":"0e3be5af-3630-43f1-80b1-daf23fb9b78b","Type":"ContainerStarted","Data":"98f4737e5eb0409b81179a069c1ff34c55f621d18e84067f9d2bfb7bb3b04498"} Jul 08 18:40:45.142139 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:45.142106 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23d40abf-d3f0-4faa-9768-57e6486a5a6c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lz2g7\" (UID: \"23d40abf-d3f0-4faa-9768-57e6486a5a6c\") " pod="openshift-insights/insights-runtime-extractor-lz2g7" Jul 08 18:40:45.142139 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:45.142140 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23d40abf-d3f0-4faa-9768-57e6486a5a6c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lz2g7\" (UID: \"23d40abf-d3f0-4faa-9768-57e6486a5a6c\") " pod="openshift-insights/insights-runtime-extractor-lz2g7" Jul 08 18:40:45.142350 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:45.142215 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 18:40:45.142350 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:45.142239 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 18:40:45.142350 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:45.142279 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23d40abf-d3f0-4faa-9768-57e6486a5a6c-kube-rbac-proxy-cm podName:23d40abf-d3f0-4faa-9768-57e6486a5a6c nodeName:}" failed. No retries permitted until 2026-07-08 18:40:46.142263399 +0000 UTC m=+56.979591097 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/23d40abf-d3f0-4faa-9768-57e6486a5a6c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lz2g7" (UID: "23d40abf-d3f0-4faa-9768-57e6486a5a6c") : configmap "kube-rbac-proxy" not found Jul 08 18:40:45.142350 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:45.142308 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23d40abf-d3f0-4faa-9768-57e6486a5a6c-insights-runtime-extractor-tls podName:23d40abf-d3f0-4faa-9768-57e6486a5a6c nodeName:}" failed. No retries permitted until 2026-07-08 18:40:46.142288172 +0000 UTC m=+56.979615858 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23d40abf-d3f0-4faa-9768-57e6486a5a6c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lz2g7" (UID: "23d40abf-d3f0-4faa-9768-57e6486a5a6c") : secret "insights-runtime-extractor-tls" not found Jul 08 18:40:46.151158 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:46.151123 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23d40abf-d3f0-4faa-9768-57e6486a5a6c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lz2g7\" (UID: \"23d40abf-d3f0-4faa-9768-57e6486a5a6c\") " pod="openshift-insights/insights-runtime-extractor-lz2g7" Jul 08 18:40:46.151158 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:46.151158 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23d40abf-d3f0-4faa-9768-57e6486a5a6c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lz2g7\" (UID: \"23d40abf-d3f0-4faa-9768-57e6486a5a6c\") " pod="openshift-insights/insights-runtime-extractor-lz2g7" Jul 08 18:40:46.151594 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:46.151244 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 18:40:46.151594 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:46.151289 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 18:40:46.151594 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:46.151331 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23d40abf-d3f0-4faa-9768-57e6486a5a6c-kube-rbac-proxy-cm podName:23d40abf-d3f0-4faa-9768-57e6486a5a6c nodeName:}" failed. No retries permitted until 2026-07-08 18:40:48.151310842 +0000 UTC m=+58.988638542 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/23d40abf-d3f0-4faa-9768-57e6486a5a6c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lz2g7" (UID: "23d40abf-d3f0-4faa-9768-57e6486a5a6c") : configmap "kube-rbac-proxy" not found Jul 08 18:40:46.151594 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:46.151351 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23d40abf-d3f0-4faa-9768-57e6486a5a6c-insights-runtime-extractor-tls podName:23d40abf-d3f0-4faa-9768-57e6486a5a6c nodeName:}" failed. No retries permitted until 2026-07-08 18:40:48.151339676 +0000 UTC m=+58.988667361 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23d40abf-d3f0-4faa-9768-57e6486a5a6c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lz2g7" (UID: "23d40abf-d3f0-4faa-9768-57e6486a5a6c") : secret "insights-runtime-extractor-tls" not found Jul 08 18:40:46.293543 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:46.293519 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7gjd9_bcc6d89d-de5e-44a5-9fe2-02785d57a8e0/dns-node-resolver/0.log" Jul 08 18:40:47.462108 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:47.462059 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-7drlq\" (UID: \"3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7drlq" Jul 08 18:40:47.462538 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:47.462214 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 18:40:47.462538 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:47.462283 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7-cluster-monitoring-operator-tls podName:3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7 nodeName:}" failed. No retries permitted until 2026-07-08 18:40:55.462266668 +0000 UTC m=+66.299594386 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-7drlq" (UID: "3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7") : secret "cluster-monitoring-operator-tls" not found Jul 08 18:40:47.494378 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:47.494352 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jj7hl_06234e92-3ccc-4e0a-9250-bfe6bf99befd/node-ca/0.log" Jul 08 18:40:47.563404 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:47.563374 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f18a01b1-7095-480e-ac45-e3a3cce3cfe5-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-qcqpm\" (UID: \"f18a01b1-7095-480e-ac45-e3a3cce3cfe5\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-qcqpm" Jul 08 18:40:47.563559 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:47.563419 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6cb0e900-30c7-416f-a732-2f590e174b8f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-88bdr\" (UID: \"6cb0e900-30c7-416f-a732-2f590e174b8f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-88bdr" Jul 08 18:40:47.563559 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:47.563526 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 18:40:47.563559 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:47.563553 2575 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 18:40:47.563663 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:47.563590 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f18a01b1-7095-480e-ac45-e3a3cce3cfe5-networking-console-plugin-cert podName:f18a01b1-7095-480e-ac45-e3a3cce3cfe5 nodeName:}" failed. No retries permitted until 2026-07-08 18:40:55.563573762 +0000 UTC m=+66.400901451 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f18a01b1-7095-480e-ac45-e3a3cce3cfe5-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-qcqpm" (UID: "f18a01b1-7095-480e-ac45-e3a3cce3cfe5") : secret "networking-console-plugin-cert" not found Jul 08 18:40:47.563663 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:47.563606 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb0e900-30c7-416f-a732-2f590e174b8f-samples-operator-tls podName:6cb0e900-30c7-416f-a732-2f590e174b8f nodeName:}" failed. No retries permitted until 2026-07-08 18:40:55.563599143 +0000 UTC m=+66.400926829 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/6cb0e900-30c7-416f-a732-2f590e174b8f-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-88bdr" (UID: "6cb0e900-30c7-416f-a732-2f590e174b8f") : secret "samples-operator-tls" not found Jul 08 18:40:47.610602 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:47.610550 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-p4gn9" podStartSLOduration=4.709994705 podStartE2EDuration="8.610530179s" totalImagesPullingTime="3.900535474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:40:45.02875491 +0000 UTC m=+55.866082619" watchObservedRunningTime="2026-07-08 18:40:47.610530179 +0000 UTC m=+58.447857881" Jul 08 18:40:47.611489 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:47.611474 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-mp7f9"] Jul 08 18:40:47.613760 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:47.613746 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-mp7f9" Jul 08 18:40:47.616695 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:47.616671 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 08 18:40:47.616860 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:47.616743 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 08 18:40:47.616860 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:47.616766 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 08 18:40:47.618267 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:47.618251 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-lfgft\"" Jul 08 18:40:47.618387 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:47.618269 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 08 18:40:47.625835 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:47.625816 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-mp7f9"] Jul 08 18:40:47.664779 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:47.664751 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9435fc52-9618-4a7e-9b4e-e09a7bb498c9-signing-key\") pod \"service-ca-758f599c5-mp7f9\" (UID: \"9435fc52-9618-4a7e-9b4e-e09a7bb498c9\") " pod="openshift-service-ca/service-ca-758f599c5-mp7f9" Jul 08 18:40:47.664903 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:47.664809 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9435fc52-9618-4a7e-9b4e-e09a7bb498c9-signing-cabundle\") pod \"service-ca-758f599c5-mp7f9\" (UID: \"9435fc52-9618-4a7e-9b4e-e09a7bb498c9\") " pod="openshift-service-ca/service-ca-758f599c5-mp7f9" Jul 08 18:40:47.664903 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:47.664849 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zrhf\" (UniqueName: \"kubernetes.io/projected/9435fc52-9618-4a7e-9b4e-e09a7bb498c9-kube-api-access-2zrhf\") pod \"service-ca-758f599c5-mp7f9\" (UID: \"9435fc52-9618-4a7e-9b4e-e09a7bb498c9\") " pod="openshift-service-ca/service-ca-758f599c5-mp7f9" Jul 08 18:40:47.765878 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:47.765803 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9435fc52-9618-4a7e-9b4e-e09a7bb498c9-signing-key\") pod \"service-ca-758f599c5-mp7f9\" (UID: \"9435fc52-9618-4a7e-9b4e-e09a7bb498c9\") " pod="openshift-service-ca/service-ca-758f599c5-mp7f9" Jul 08 18:40:47.765878 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:47.765867 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9435fc52-9618-4a7e-9b4e-e09a7bb498c9-signing-cabundle\") pod \"service-ca-758f599c5-mp7f9\" (UID: \"9435fc52-9618-4a7e-9b4e-e09a7bb498c9\") " pod="openshift-service-ca/service-ca-758f599c5-mp7f9" Jul 08 18:40:47.766051 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:47.765906 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zrhf\" (UniqueName: \"kubernetes.io/projected/9435fc52-9618-4a7e-9b4e-e09a7bb498c9-kube-api-access-2zrhf\") pod \"service-ca-758f599c5-mp7f9\" (UID: \"9435fc52-9618-4a7e-9b4e-e09a7bb498c9\") " pod="openshift-service-ca/service-ca-758f599c5-mp7f9" Jul 08 18:40:47.766532 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:47.766514 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9435fc52-9618-4a7e-9b4e-e09a7bb498c9-signing-cabundle\") pod \"service-ca-758f599c5-mp7f9\" (UID: \"9435fc52-9618-4a7e-9b4e-e09a7bb498c9\") " pod="openshift-service-ca/service-ca-758f599c5-mp7f9" Jul 08 18:40:47.768119 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:47.768100 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9435fc52-9618-4a7e-9b4e-e09a7bb498c9-signing-key\") pod \"service-ca-758f599c5-mp7f9\" (UID: \"9435fc52-9618-4a7e-9b4e-e09a7bb498c9\") " pod="openshift-service-ca/service-ca-758f599c5-mp7f9" Jul 08 18:40:47.774442 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:47.774421 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zrhf\" (UniqueName: \"kubernetes.io/projected/9435fc52-9618-4a7e-9b4e-e09a7bb498c9-kube-api-access-2zrhf\") pod \"service-ca-758f599c5-mp7f9\" (UID: \"9435fc52-9618-4a7e-9b4e-e09a7bb498c9\") " pod="openshift-service-ca/service-ca-758f599c5-mp7f9" Jul 08 18:40:47.923113 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:47.923070 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-mp7f9" Jul 08 18:40:48.046651 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:48.046578 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-mp7f9"] Jul 08 18:40:48.049853 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:40:48.049824 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9435fc52_9618_4a7e_9b4e_e09a7bb498c9.slice/crio-591e556e36b5ca96ac08860f349b188ccb85ec7f99e75365bb1894e4ccf3c601 WatchSource:0}: Error finding container 591e556e36b5ca96ac08860f349b188ccb85ec7f99e75365bb1894e4ccf3c601: Status 404 returned error can't find the container with id 591e556e36b5ca96ac08860f349b188ccb85ec7f99e75365bb1894e4ccf3c601 Jul 08 18:40:48.169535 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:48.169498 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23d40abf-d3f0-4faa-9768-57e6486a5a6c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lz2g7\" (UID: \"23d40abf-d3f0-4faa-9768-57e6486a5a6c\") " pod="openshift-insights/insights-runtime-extractor-lz2g7" Jul 08 18:40:48.169871 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:48.169564 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23d40abf-d3f0-4faa-9768-57e6486a5a6c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lz2g7\" (UID: \"23d40abf-d3f0-4faa-9768-57e6486a5a6c\") " pod="openshift-insights/insights-runtime-extractor-lz2g7" Jul 08 18:40:48.169871 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:48.169632 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 18:40:48.169871 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:48.169641 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 18:40:48.169871 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:48.169683 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23d40abf-d3f0-4faa-9768-57e6486a5a6c-kube-rbac-proxy-cm podName:23d40abf-d3f0-4faa-9768-57e6486a5a6c nodeName:}" failed. No retries permitted until 2026-07-08 18:40:52.169669922 +0000 UTC m=+63.006997608 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/23d40abf-d3f0-4faa-9768-57e6486a5a6c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lz2g7" (UID: "23d40abf-d3f0-4faa-9768-57e6486a5a6c") : configmap "kube-rbac-proxy" not found Jul 08 18:40:48.169871 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:48.169697 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23d40abf-d3f0-4faa-9768-57e6486a5a6c-insights-runtime-extractor-tls podName:23d40abf-d3f0-4faa-9768-57e6486a5a6c nodeName:}" failed. No retries permitted until 2026-07-08 18:40:52.169690577 +0000 UTC m=+63.007018263 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23d40abf-d3f0-4faa-9768-57e6486a5a6c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lz2g7" (UID: "23d40abf-d3f0-4faa-9768-57e6486a5a6c") : secret "insights-runtime-extractor-tls" not found Jul 08 18:40:48.495885 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:48.495853 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l582k_a2d88ce3-6a8c-4b89-8bcd-e5a8ead15f44/serve-healthcheck-canary/0.log" Jul 08 18:40:49.013984 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:49.013948 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-mp7f9" event={"ID":"9435fc52-9618-4a7e-9b4e-e09a7bb498c9","Type":"ContainerStarted","Data":"195ecfba5a8661d033ea85e15cba8b730ded0e5a453a33b78ac41b0f72f4c122"} Jul 08 18:40:49.014172 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:49.014000 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-mp7f9" event={"ID":"9435fc52-9618-4a7e-9b4e-e09a7bb498c9","Type":"ContainerStarted","Data":"591e556e36b5ca96ac08860f349b188ccb85ec7f99e75365bb1894e4ccf3c601"} Jul 08 18:40:49.039725 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:49.039678 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-mp7f9" podStartSLOduration=2.039666284 podStartE2EDuration="2.039666284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:40:49.038954248 +0000 UTC m=+59.876281983" watchObservedRunningTime="2026-07-08 18:40:49.039666284 +0000 UTC m=+59.876993992" Jul 08 18:40:52.204605 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:52.204557 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23d40abf-d3f0-4faa-9768-57e6486a5a6c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lz2g7\" (UID: \"23d40abf-d3f0-4faa-9768-57e6486a5a6c\") " pod="openshift-insights/insights-runtime-extractor-lz2g7" Jul 08 18:40:52.205098 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:52.204653 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23d40abf-d3f0-4faa-9768-57e6486a5a6c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lz2g7\" (UID: \"23d40abf-d3f0-4faa-9768-57e6486a5a6c\") " pod="openshift-insights/insights-runtime-extractor-lz2g7" Jul 08 18:40:52.205098 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:52.204678 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 18:40:52.205098 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:52.204798 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 18:40:52.205098 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:52.204802 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/23d40abf-d3f0-4faa-9768-57e6486a5a6c-kube-rbac-proxy-cm podName:23d40abf-d3f0-4faa-9768-57e6486a5a6c nodeName:}" failed. No retries permitted until 2026-07-08 18:41:00.204783124 +0000 UTC m=+71.042110831 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/23d40abf-d3f0-4faa-9768-57e6486a5a6c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lz2g7" (UID: "23d40abf-d3f0-4faa-9768-57e6486a5a6c") : configmap "kube-rbac-proxy" not found Jul 08 18:40:52.205098 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:52.204849 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23d40abf-d3f0-4faa-9768-57e6486a5a6c-insights-runtime-extractor-tls podName:23d40abf-d3f0-4faa-9768-57e6486a5a6c nodeName:}" failed. No retries permitted until 2026-07-08 18:41:00.204837781 +0000 UTC m=+71.042165467 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/23d40abf-d3f0-4faa-9768-57e6486a5a6c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lz2g7" (UID: "23d40abf-d3f0-4faa-9768-57e6486a5a6c") : secret "insights-runtime-extractor-tls" not found Jul 08 18:40:52.936204 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:52.936168 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jrzrj" Jul 08 18:40:55.431631 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:55.431579 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee39c730-7ca1-4260-afea-a884940c00fc-metrics-certs\") pod \"network-metrics-daemon-252x5\" (UID: \"ee39c730-7ca1-4260-afea-a884940c00fc\") " pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:40:55.431631 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:55.431632 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/610063ae-dbb1-4a58-bacd-06bf90518168-original-pull-secret\") pod \"global-pull-secret-syncer-szlwf\" (UID: \"610063ae-dbb1-4a58-bacd-06bf90518168\") " pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:40:55.434946 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:55.434927 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 18:40:55.434999 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:55.434952 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 18:40:55.442048 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:55.442031 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 08 18:40:55.442101 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:55.442090 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee39c730-7ca1-4260-afea-a884940c00fc-metrics-certs podName:ee39c730-7ca1-4260-afea-a884940c00fc nodeName:}" failed. No retries permitted until 2026-07-08 18:41:59.442074942 +0000 UTC m=+130.279402641 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee39c730-7ca1-4260-afea-a884940c00fc-metrics-certs") pod "network-metrics-daemon-252x5" (UID: "ee39c730-7ca1-4260-afea-a884940c00fc") : secret "metrics-daemon-secret" not found Jul 08 18:40:55.444105 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:55.444085 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/610063ae-dbb1-4a58-bacd-06bf90518168-original-pull-secret\") pod \"global-pull-secret-syncer-szlwf\" (UID: \"610063ae-dbb1-4a58-bacd-06bf90518168\") " pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:40:55.483023 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:55.482991 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szlwf" Jul 08 18:40:55.532845 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:55.532770 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zwd9\" (UniqueName: \"kubernetes.io/projected/ff43f31c-c779-406d-8352-4590967387a5-kube-api-access-2zwd9\") pod \"network-check-target-pwffr\" (UID: \"ff43f31c-c779-406d-8352-4590967387a5\") " pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:40:55.532982 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:55.532861 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-7drlq\" (UID: \"3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7drlq" Jul 08 18:40:55.533043 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:55.533033 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 18:40:55.533672 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:55.533099 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7-cluster-monitoring-operator-tls podName:3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7 nodeName:}" failed. No retries permitted until 2026-07-08 18:41:11.53307938 +0000 UTC m=+82.370407071 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-7drlq" (UID: "3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7") : secret "cluster-monitoring-operator-tls" not found Jul 08 18:40:55.536582 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:55.536554 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 18:40:55.547083 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:55.547058 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 18:40:55.557863 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:55.557774 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zwd9\" (UniqueName: \"kubernetes.io/projected/ff43f31c-c779-406d-8352-4590967387a5-kube-api-access-2zwd9\") pod \"network-check-target-pwffr\" (UID: \"ff43f31c-c779-406d-8352-4590967387a5\") " pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:40:55.612777 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:55.612747 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-szlwf"] Jul 08 18:40:55.615593 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:40:55.615568 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610063ae_dbb1_4a58_bacd_06bf90518168.slice/crio-62c07cbe515caaa19a434c11e35045498e2f2959604cb5429cf66c2f6cc641f3 WatchSource:0}: Error finding container 62c07cbe515caaa19a434c11e35045498e2f2959604cb5429cf66c2f6cc641f3: Status 404 returned error can't find the container with id 62c07cbe515caaa19a434c11e35045498e2f2959604cb5429cf66c2f6cc641f3 Jul 08 18:40:55.633379 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:55.633358 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f18a01b1-7095-480e-ac45-e3a3cce3cfe5-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-qcqpm\" (UID: \"f18a01b1-7095-480e-ac45-e3a3cce3cfe5\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-qcqpm" Jul 08 18:40:55.633471 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:55.633390 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6cb0e900-30c7-416f-a732-2f590e174b8f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-88bdr\" (UID: \"6cb0e900-30c7-416f-a732-2f590e174b8f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-88bdr" Jul 08 18:40:55.633511 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:55.633493 2575 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 18:40:55.633560 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:40:55.633549 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f18a01b1-7095-480e-ac45-e3a3cce3cfe5-networking-console-plugin-cert podName:f18a01b1-7095-480e-ac45-e3a3cce3cfe5 nodeName:}" failed. No retries permitted until 2026-07-08 18:41:11.633533171 +0000 UTC m=+82.470860872 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f18a01b1-7095-480e-ac45-e3a3cce3cfe5-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-qcqpm" (UID: "f18a01b1-7095-480e-ac45-e3a3cce3cfe5") : secret "networking-console-plugin-cert" not found Jul 08 18:40:55.635839 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:55.635819 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/6cb0e900-30c7-416f-a732-2f590e174b8f-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-88bdr\" (UID: \"6cb0e900-30c7-416f-a732-2f590e174b8f\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-88bdr" Jul 08 18:40:55.640260 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:55.640242 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-z2lrl\"" Jul 08 18:40:55.648417 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:55.648400 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-88bdr" Jul 08 18:40:55.778339 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:55.778316 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-88bdr"] Jul 08 18:40:55.791542 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:55.791519 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-msx7w\"" Jul 08 18:40:55.799022 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:55.799001 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:40:55.922754 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:55.922670 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pwffr"] Jul 08 18:40:55.926069 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:40:55.926047 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff43f31c_c779_406d_8352_4590967387a5.slice/crio-a59b3e1bc2ac16f520bd372cc7ebf360a9f61756e53ae763087ce3f91bab90ba WatchSource:0}: Error finding container a59b3e1bc2ac16f520bd372cc7ebf360a9f61756e53ae763087ce3f91bab90ba: Status 404 returned error can't find the container with id a59b3e1bc2ac16f520bd372cc7ebf360a9f61756e53ae763087ce3f91bab90ba Jul 08 18:40:56.033188 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:56.033100 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-88bdr" event={"ID":"6cb0e900-30c7-416f-a732-2f590e174b8f","Type":"ContainerStarted","Data":"2f0532d5558a4d2cce3793a5349b2eb671c60de422161d19a156f6c495b188f6"} Jul 08 18:40:56.034287 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:56.034257 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pwffr" event={"ID":"ff43f31c-c779-406d-8352-4590967387a5","Type":"ContainerStarted","Data":"a59b3e1bc2ac16f520bd372cc7ebf360a9f61756e53ae763087ce3f91bab90ba"} Jul 08 18:40:56.035395 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:40:56.035374 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-szlwf" event={"ID":"610063ae-dbb1-4a58-bacd-06bf90518168","Type":"ContainerStarted","Data":"62c07cbe515caaa19a434c11e35045498e2f2959604cb5429cf66c2f6cc641f3"} Jul 08 18:41:00.277801 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:00.277749 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23d40abf-d3f0-4faa-9768-57e6486a5a6c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lz2g7\" (UID: \"23d40abf-d3f0-4faa-9768-57e6486a5a6c\") " pod="openshift-insights/insights-runtime-extractor-lz2g7" Jul 08 18:41:00.278273 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:00.277847 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23d40abf-d3f0-4faa-9768-57e6486a5a6c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lz2g7\" (UID: \"23d40abf-d3f0-4faa-9768-57e6486a5a6c\") " pod="openshift-insights/insights-runtime-extractor-lz2g7" Jul 08 18:41:00.278482 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:00.278447 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/23d40abf-d3f0-4faa-9768-57e6486a5a6c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lz2g7\" (UID: \"23d40abf-d3f0-4faa-9768-57e6486a5a6c\") " pod="openshift-insights/insights-runtime-extractor-lz2g7" Jul 08 18:41:00.280489 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:00.280463 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/23d40abf-d3f0-4faa-9768-57e6486a5a6c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lz2g7\" (UID: \"23d40abf-d3f0-4faa-9768-57e6486a5a6c\") " pod="openshift-insights/insights-runtime-extractor-lz2g7" Jul 08 18:41:00.445196 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:00.445118 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jtszc\"" Jul 08 18:41:00.452572 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:00.452543 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lz2g7" Jul 08 18:41:00.480033 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:00.480000 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b2b7480-7b33-4fd9-9993-71441bbb6c78-metrics-tls\") pod \"dns-default-ljtbq\" (UID: \"0b2b7480-7b33-4fd9-9993-71441bbb6c78\") " pod="openshift-dns/dns-default-ljtbq" Jul 08 18:41:00.482680 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:00.482645 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0b2b7480-7b33-4fd9-9993-71441bbb6c78-metrics-tls\") pod \"dns-default-ljtbq\" (UID: \"0b2b7480-7b33-4fd9-9993-71441bbb6c78\") " pod="openshift-dns/dns-default-ljtbq" Jul 08 18:41:00.660662 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:00.660628 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rmskx\"" Jul 08 18:41:00.668278 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:00.668254 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ljtbq" Jul 08 18:41:01.271056 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:01.270894 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ljtbq"] Jul 08 18:41:01.274167 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:41:01.274125 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b2b7480_7b33_4fd9_9993_71441bbb6c78.slice/crio-3ec9a22585dc6d75b16459b415922591eb63121a093f878443189239372cc297 WatchSource:0}: Error finding container 3ec9a22585dc6d75b16459b415922591eb63121a093f878443189239372cc297: Status 404 returned error can't find the container with id 3ec9a22585dc6d75b16459b415922591eb63121a093f878443189239372cc297 Jul 08 18:41:01.295903 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:01.295583 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lz2g7"] Jul 08 18:41:01.299426 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:41:01.299360 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d40abf_d3f0_4faa_9768_57e6486a5a6c.slice/crio-2b75b1e2be64e12d2cb9edef7a9fd61ee26a1d5c74f080575ffae2809dda2a4e WatchSource:0}: Error finding container 2b75b1e2be64e12d2cb9edef7a9fd61ee26a1d5c74f080575ffae2809dda2a4e: Status 404 returned error can't find the container with id 2b75b1e2be64e12d2cb9edef7a9fd61ee26a1d5c74f080575ffae2809dda2a4e Jul 08 18:41:02.060284 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:02.059585 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-88bdr" event={"ID":"6cb0e900-30c7-416f-a732-2f590e174b8f","Type":"ContainerStarted","Data":"d3f0f2109c35e82f6dbdd92deeb15f96101e8f87210a57b73194d6da597a9708"} Jul 08 18:41:02.060284 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:02.059653 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-88bdr" event={"ID":"6cb0e900-30c7-416f-a732-2f590e174b8f","Type":"ContainerStarted","Data":"e1a4408e1a6b8c8f3fa2defdcc0a937495810d5ed6e50050a10f3ba06ed77690"} Jul 08 18:41:02.065798 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:02.065654 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pwffr" event={"ID":"ff43f31c-c779-406d-8352-4590967387a5","Type":"ContainerStarted","Data":"1ca5daf02bc75f2fc471b3ce175cdbf0801bab8a8f8aa06305b23cc9e691c8e4"} Jul 08 18:41:02.065798 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:02.065756 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:41:02.070173 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:02.069749 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-szlwf" event={"ID":"610063ae-dbb1-4a58-bacd-06bf90518168","Type":"ContainerStarted","Data":"004c41cd170eb96bdc299d30965a60a54dafb00d34871ccc4d8cc11ff2dc68f5"} Jul 08 18:41:02.071952 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:02.071875 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ljtbq" event={"ID":"0b2b7480-7b33-4fd9-9993-71441bbb6c78","Type":"ContainerStarted","Data":"3ec9a22585dc6d75b16459b415922591eb63121a093f878443189239372cc297"} Jul 08 18:41:02.075221 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:02.075145 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lz2g7" event={"ID":"23d40abf-d3f0-4faa-9768-57e6486a5a6c","Type":"ContainerStarted","Data":"2b75b1e2be64e12d2cb9edef7a9fd61ee26a1d5c74f080575ffae2809dda2a4e"} Jul 08 18:41:02.082330 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:02.081644 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-88bdr" podStartSLOduration=17.804368519 podStartE2EDuration="23.081626989s" totalImagesPullingTime="5.27725847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:41:02.081377097 +0000 UTC m=+72.918704806" watchObservedRunningTime="2026-07-08 18:41:02.081626989 +0000 UTC m=+72.918954699" Jul 08 18:41:02.100243 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:02.099595 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-szlwf" podStartSLOduration=66.604405109 podStartE2EDuration="1m12.099579294s" totalImagesPullingTime="5.495174185s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:41:02.098434945 +0000 UTC m=+72.935762653" watchObservedRunningTime="2026-07-08 18:41:02.099579294 +0000 UTC m=+72.936907004" Jul 08 18:41:06.092639 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:06.092601 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ljtbq" event={"ID":"0b2b7480-7b33-4fd9-9993-71441bbb6c78","Type":"ContainerStarted","Data":"bbae0d4ef28f1aab527eb3df2ca457c7298fb04f5a2e190fc7824949ddfd7885"} Jul 08 18:41:06.093025 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:06.092664 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ljtbq" event={"ID":"0b2b7480-7b33-4fd9-9993-71441bbb6c78","Type":"ContainerStarted","Data":"869282964ecc4cd60a37c80cb9f14d99327ff2c3525e2c1fc5b5b078a7e3e9ff"} Jul 08 18:41:06.093025 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:06.092741 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ljtbq" Jul 08 18:41:06.094048 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:06.094026 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lz2g7" event={"ID":"23d40abf-d3f0-4faa-9768-57e6486a5a6c","Type":"ContainerStarted","Data":"12bd5cf9774f2387e3b0b0fb48911c3807ebaa1896aac87b4fe2aa66bc7e0cf1"} Jul 08 18:41:06.112129 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:06.112088 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pwffr" podStartSLOduration=70.921285077 podStartE2EDuration="1m16.112073258s" totalImagesPullingTime="5.190788181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:41:02.117566733 +0000 UTC m=+72.954894442" watchObservedRunningTime="2026-07-08 18:41:06.112073258 +0000 UTC m=+76.949400966" Jul 08 18:41:06.112670 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:06.112645 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ljtbq" podStartSLOduration=33.983554204 podStartE2EDuration="38.112635844s" totalImagesPullingTime="4.12908164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:41:06.110792296 +0000 UTC m=+76.948120005" watchObservedRunningTime="2026-07-08 18:41:06.112635844 +0000 UTC m=+76.949963552" Jul 08 18:41:07.101766 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:07.101727 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lz2g7" event={"ID":"23d40abf-d3f0-4faa-9768-57e6486a5a6c","Type":"ContainerStarted","Data":"de4650f0792880483b329156ade43bcf92c980795ac5e75a13b0937d4370a87a"} Jul 08 18:41:09.111623 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:09.111588 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lz2g7" event={"ID":"23d40abf-d3f0-4faa-9768-57e6486a5a6c","Type":"ContainerStarted","Data":"44c20d0494debcebee0cd33a2a5a88968dfbb85e2665b1480936b229685ea38b"} Jul 08 18:41:09.133784 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:09.133736 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lz2g7" podStartSLOduration=17.968991761 podStartE2EDuration="25.133722182s" totalImagesPullingTime="7.164730421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:40:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:41:09.132083267 +0000 UTC m=+79.969411008" watchObservedRunningTime="2026-07-08 18:41:09.133722182 +0000 UTC m=+79.971049889" Jul 08 18:41:10.374353 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.374318 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b9679dd58-rrf82"] Jul 08 18:41:10.377310 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.377278 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-864d54dd8d-k7l9n"] Jul 08 18:41:10.377418 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.377390 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b9679dd58-rrf82" Jul 08 18:41:10.380361 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.380343 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864d54dd8d-k7l9n" Jul 08 18:41:10.381664 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.381650 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 08 18:41:10.381733 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.381674 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-vjsnc\"" Jul 08 18:41:10.382077 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.382059 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 08 18:41:10.382475 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.382455 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 08 18:41:10.382933 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.382916 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 08 18:41:10.393035 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.393011 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b9679dd58-rrf82"] Jul 08 18:41:10.394154 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.394128 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-864d54dd8d-k7l9n"] Jul 08 18:41:10.394318 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:41:10.394207 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"work-manager-hub-kubeconfig\" is forbidden: User \"system:node:ip-10-0-131-102.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-131-102.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" type="*v1.Secret" Jul 08 18:41:10.460805 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.460776 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsgvt\" (UniqueName: \"kubernetes.io/projected/8274723b-a4ff-4cea-8697-c9d972e321ac-kube-api-access-lsgvt\") pod \"managed-serviceaccount-addon-agent-7b9679dd58-rrf82\" (UID: \"8274723b-a4ff-4cea-8697-c9d972e321ac\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b9679dd58-rrf82" Jul 08 18:41:10.460992 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.460833 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8274723b-a4ff-4cea-8697-c9d972e321ac-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7b9679dd58-rrf82\" (UID: \"8274723b-a4ff-4cea-8697-c9d972e321ac\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b9679dd58-rrf82" Jul 08 18:41:10.460992 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.460862 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce8ea612-0ed6-41b9-a621-a1411751786b-tmp\") pod \"klusterlet-addon-workmgr-864d54dd8d-k7l9n\" (UID: \"ce8ea612-0ed6-41b9-a621-a1411751786b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864d54dd8d-k7l9n" Jul 08 18:41:10.460992 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.460906 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ce8ea612-0ed6-41b9-a621-a1411751786b-klusterlet-config\") pod \"klusterlet-addon-workmgr-864d54dd8d-k7l9n\" (UID: \"ce8ea612-0ed6-41b9-a621-a1411751786b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864d54dd8d-k7l9n" Jul 08 18:41:10.460992 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.460981 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x2pf\" (UniqueName: \"kubernetes.io/projected/ce8ea612-0ed6-41b9-a621-a1411751786b-kube-api-access-4x2pf\") pod \"klusterlet-addon-workmgr-864d54dd8d-k7l9n\" (UID: \"ce8ea612-0ed6-41b9-a621-a1411751786b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864d54dd8d-k7l9n" Jul 08 18:41:10.472594 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.472566 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d6d887bf8-98j5w"] Jul 08 18:41:10.475784 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.475765 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:10.478853 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.478826 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 08 18:41:10.478994 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.478919 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n6k5m\"" Jul 08 18:41:10.478994 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.478928 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 08 18:41:10.479173 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.479154 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 08 18:41:10.486050 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.486034 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 08 18:41:10.497529 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.497506 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d6d887bf8-98j5w"] Jul 08 18:41:10.561951 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.561921 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8274723b-a4ff-4cea-8697-c9d972e321ac-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7b9679dd58-rrf82\" (UID: \"8274723b-a4ff-4cea-8697-c9d972e321ac\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b9679dd58-rrf82" Jul 08 18:41:10.561951 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.561954 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2s4w\" (UniqueName: \"kubernetes.io/projected/3bf87a41-e859-4773-a975-7b8633d52799-kube-api-access-p2s4w\") pod \"image-registry-5d6d887bf8-98j5w\" (UID: \"3bf87a41-e859-4773-a975-7b8633d52799\") " pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:10.562165 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.561976 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce8ea612-0ed6-41b9-a621-a1411751786b-tmp\") pod \"klusterlet-addon-workmgr-864d54dd8d-k7l9n\" (UID: \"ce8ea612-0ed6-41b9-a621-a1411751786b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864d54dd8d-k7l9n" Jul 08 18:41:10.562165 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.562000 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3bf87a41-e859-4773-a975-7b8633d52799-image-registry-private-configuration\") pod \"image-registry-5d6d887bf8-98j5w\" (UID: \"3bf87a41-e859-4773-a975-7b8633d52799\") " pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:10.562165 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.562025 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ce8ea612-0ed6-41b9-a621-a1411751786b-klusterlet-config\") pod \"klusterlet-addon-workmgr-864d54dd8d-k7l9n\" (UID: \"ce8ea612-0ed6-41b9-a621-a1411751786b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864d54dd8d-k7l9n" Jul 08 18:41:10.562165 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.562045 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3bf87a41-e859-4773-a975-7b8633d52799-ca-trust-extracted\") pod \"image-registry-5d6d887bf8-98j5w\" (UID: \"3bf87a41-e859-4773-a975-7b8633d52799\") " pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:10.562165 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.562075 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x2pf\" (UniqueName: \"kubernetes.io/projected/ce8ea612-0ed6-41b9-a621-a1411751786b-kube-api-access-4x2pf\") pod \"klusterlet-addon-workmgr-864d54dd8d-k7l9n\" (UID: \"ce8ea612-0ed6-41b9-a621-a1411751786b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864d54dd8d-k7l9n" Jul 08 18:41:10.562165 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.562098 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3bf87a41-e859-4773-a975-7b8633d52799-registry-certificates\") pod \"image-registry-5d6d887bf8-98j5w\" (UID: \"3bf87a41-e859-4773-a975-7b8633d52799\") " pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:10.562511 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.562219 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3bf87a41-e859-4773-a975-7b8633d52799-installation-pull-secrets\") pod \"image-registry-5d6d887bf8-98j5w\" (UID: \"3bf87a41-e859-4773-a975-7b8633d52799\") " pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:10.562511 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.562275 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bf87a41-e859-4773-a975-7b8633d52799-registry-tls\") pod \"image-registry-5d6d887bf8-98j5w\" (UID: \"3bf87a41-e859-4773-a975-7b8633d52799\") " pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:10.562511 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.562326 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3bf87a41-e859-4773-a975-7b8633d52799-trusted-ca\") pod \"image-registry-5d6d887bf8-98j5w\" (UID: \"3bf87a41-e859-4773-a975-7b8633d52799\") " pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:10.562511 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.562349 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce8ea612-0ed6-41b9-a621-a1411751786b-tmp\") pod \"klusterlet-addon-workmgr-864d54dd8d-k7l9n\" (UID: \"ce8ea612-0ed6-41b9-a621-a1411751786b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864d54dd8d-k7l9n" Jul 08 18:41:10.562511 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.562418 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsgvt\" (UniqueName: \"kubernetes.io/projected/8274723b-a4ff-4cea-8697-c9d972e321ac-kube-api-access-lsgvt\") pod \"managed-serviceaccount-addon-agent-7b9679dd58-rrf82\" (UID: \"8274723b-a4ff-4cea-8697-c9d972e321ac\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b9679dd58-rrf82" Jul 08 18:41:10.562511 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.562449 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bf87a41-e859-4773-a975-7b8633d52799-bound-sa-token\") pod \"image-registry-5d6d887bf8-98j5w\" (UID: \"3bf87a41-e859-4773-a975-7b8633d52799\") " pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:10.564610 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.564589 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8274723b-a4ff-4cea-8697-c9d972e321ac-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7b9679dd58-rrf82\" (UID: \"8274723b-a4ff-4cea-8697-c9d972e321ac\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b9679dd58-rrf82" Jul 08 18:41:10.573181 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.573153 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x2pf\" (UniqueName: \"kubernetes.io/projected/ce8ea612-0ed6-41b9-a621-a1411751786b-kube-api-access-4x2pf\") pod \"klusterlet-addon-workmgr-864d54dd8d-k7l9n\" (UID: \"ce8ea612-0ed6-41b9-a621-a1411751786b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864d54dd8d-k7l9n" Jul 08 18:41:10.573286 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.573208 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsgvt\" (UniqueName: \"kubernetes.io/projected/8274723b-a4ff-4cea-8697-c9d972e321ac-kube-api-access-lsgvt\") pod \"managed-serviceaccount-addon-agent-7b9679dd58-rrf82\" (UID: \"8274723b-a4ff-4cea-8697-c9d972e321ac\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b9679dd58-rrf82" Jul 08 18:41:10.663179 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.663093 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3bf87a41-e859-4773-a975-7b8633d52799-ca-trust-extracted\") pod \"image-registry-5d6d887bf8-98j5w\" (UID: \"3bf87a41-e859-4773-a975-7b8633d52799\") " pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:10.663179 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.663144 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3bf87a41-e859-4773-a975-7b8633d52799-registry-certificates\") pod \"image-registry-5d6d887bf8-98j5w\" (UID: \"3bf87a41-e859-4773-a975-7b8633d52799\") " pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:10.663436 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.663272 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3bf87a41-e859-4773-a975-7b8633d52799-installation-pull-secrets\") pod \"image-registry-5d6d887bf8-98j5w\" (UID: \"3bf87a41-e859-4773-a975-7b8633d52799\") " pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:10.663436 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.663342 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bf87a41-e859-4773-a975-7b8633d52799-registry-tls\") pod \"image-registry-5d6d887bf8-98j5w\" (UID: \"3bf87a41-e859-4773-a975-7b8633d52799\") " pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:10.663436 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.663368 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3bf87a41-e859-4773-a975-7b8633d52799-trusted-ca\") pod \"image-registry-5d6d887bf8-98j5w\" (UID: \"3bf87a41-e859-4773-a975-7b8633d52799\") " pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:10.663436 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.663426 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bf87a41-e859-4773-a975-7b8633d52799-bound-sa-token\") pod \"image-registry-5d6d887bf8-98j5w\" (UID: \"3bf87a41-e859-4773-a975-7b8633d52799\") " pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:10.663625 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.663551 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3bf87a41-e859-4773-a975-7b8633d52799-ca-trust-extracted\") pod \"image-registry-5d6d887bf8-98j5w\" (UID: \"3bf87a41-e859-4773-a975-7b8633d52799\") " pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:10.663625 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.663565 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2s4w\" (UniqueName: \"kubernetes.io/projected/3bf87a41-e859-4773-a975-7b8633d52799-kube-api-access-p2s4w\") pod \"image-registry-5d6d887bf8-98j5w\" (UID: \"3bf87a41-e859-4773-a975-7b8633d52799\") " pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:10.663731 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.663630 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3bf87a41-e859-4773-a975-7b8633d52799-image-registry-private-configuration\") pod \"image-registry-5d6d887bf8-98j5w\" (UID: \"3bf87a41-e859-4773-a975-7b8633d52799\") " pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:10.664010 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.663986 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3bf87a41-e859-4773-a975-7b8633d52799-registry-certificates\") pod \"image-registry-5d6d887bf8-98j5w\" (UID: \"3bf87a41-e859-4773-a975-7b8633d52799\") " pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:10.664278 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.664257 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3bf87a41-e859-4773-a975-7b8633d52799-trusted-ca\") pod \"image-registry-5d6d887bf8-98j5w\" (UID: \"3bf87a41-e859-4773-a975-7b8633d52799\") " pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:10.665820 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.665800 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3bf87a41-e859-4773-a975-7b8633d52799-installation-pull-secrets\") pod \"image-registry-5d6d887bf8-98j5w\" (UID: \"3bf87a41-e859-4773-a975-7b8633d52799\") " pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:10.665902 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.665843 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3bf87a41-e859-4773-a975-7b8633d52799-registry-tls\") pod \"image-registry-5d6d887bf8-98j5w\" (UID: \"3bf87a41-e859-4773-a975-7b8633d52799\") " pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:10.665902 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.665886 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3bf87a41-e859-4773-a975-7b8633d52799-image-registry-private-configuration\") pod \"image-registry-5d6d887bf8-98j5w\" (UID: \"3bf87a41-e859-4773-a975-7b8633d52799\") " pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:10.673194 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.673174 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2s4w\" (UniqueName: \"kubernetes.io/projected/3bf87a41-e859-4773-a975-7b8633d52799-kube-api-access-p2s4w\") pod \"image-registry-5d6d887bf8-98j5w\" (UID: \"3bf87a41-e859-4773-a975-7b8633d52799\") " pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:10.673364 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.673347 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bf87a41-e859-4773-a975-7b8633d52799-bound-sa-token\") pod \"image-registry-5d6d887bf8-98j5w\" (UID: \"3bf87a41-e859-4773-a975-7b8633d52799\") " pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:10.694306 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.694277 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b9679dd58-rrf82" Jul 08 18:41:10.785718 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.785678 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:10.828677 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.828651 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b9679dd58-rrf82"] Jul 08 18:41:10.831097 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:41:10.831072 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8274723b_a4ff_4cea_8697_c9d972e321ac.slice/crio-015a3a68136eb805b13878a479c8ec7e32bf74a1eae3c471ec766d97fcfa7b60 WatchSource:0}: Error finding container 015a3a68136eb805b13878a479c8ec7e32bf74a1eae3c471ec766d97fcfa7b60: Status 404 returned error can't find the container with id 015a3a68136eb805b13878a479c8ec7e32bf74a1eae3c471ec766d97fcfa7b60 Jul 08 18:41:10.939785 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:10.939721 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d6d887bf8-98j5w"] Jul 08 18:41:10.942184 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:41:10.942159 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf87a41_e859_4773_a975_7b8633d52799.slice/crio-283afa2685f7559e4db9011544d485d6a46c1cc49d926af3c3f7fca2f08420ca WatchSource:0}: Error finding container 283afa2685f7559e4db9011544d485d6a46c1cc49d926af3c3f7fca2f08420ca: Status 404 returned error can't find the container with id 283afa2685f7559e4db9011544d485d6a46c1cc49d926af3c3f7fca2f08420ca Jul 08 18:41:11.123603 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:11.123170 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" event={"ID":"3bf87a41-e859-4773-a975-7b8633d52799","Type":"ContainerStarted","Data":"a83d47a8c3fed9a42ab50f481bf5a8c8fab425772feebe5889fd880852ad958a"} Jul 08 18:41:11.123603 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:11.123261 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:11.123603 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:11.123282 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" event={"ID":"3bf87a41-e859-4773-a975-7b8633d52799","Type":"ContainerStarted","Data":"283afa2685f7559e4db9011544d485d6a46c1cc49d926af3c3f7fca2f08420ca"} Jul 08 18:41:11.124503 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:11.124480 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b9679dd58-rrf82" event={"ID":"8274723b-a4ff-4cea-8697-c9d972e321ac","Type":"ContainerStarted","Data":"015a3a68136eb805b13878a479c8ec7e32bf74a1eae3c471ec766d97fcfa7b60"} Jul 08 18:41:11.152319 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:11.152257 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" podStartSLOduration=1.15224269 podStartE2EDuration="1.15224269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:41:11.151386478 +0000 UTC m=+81.988714183" watchObservedRunningTime="2026-07-08 18:41:11.15224269 +0000 UTC m=+81.989570398" Jul 08 18:41:11.563231 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:41:11.563195 2575 secret.go:189] Couldn't get secret open-cluster-management-agent-addon/work-manager-hub-kubeconfig: failed to sync secret cache: timed out waiting for the condition Jul 08 18:41:11.563688 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:41:11.563287 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce8ea612-0ed6-41b9-a621-a1411751786b-klusterlet-config podName:ce8ea612-0ed6-41b9-a621-a1411751786b nodeName:}" failed. No retries permitted until 2026-07-08 18:41:12.063264494 +0000 UTC m=+82.900592180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "klusterlet-config" (UniqueName: "kubernetes.io/secret/ce8ea612-0ed6-41b9-a621-a1411751786b-klusterlet-config") pod "klusterlet-addon-workmgr-864d54dd8d-k7l9n" (UID: "ce8ea612-0ed6-41b9-a621-a1411751786b") : failed to sync secret cache: timed out waiting for the condition Jul 08 18:41:11.572526 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:11.572485 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-7drlq\" (UID: \"3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7drlq" Jul 08 18:41:11.575078 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:11.575048 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-7drlq\" (UID: \"3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7drlq" Jul 08 18:41:11.575392 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:11.575374 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 08 18:41:11.673381 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:11.673349 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f18a01b1-7095-480e-ac45-e3a3cce3cfe5-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-qcqpm\" (UID: \"f18a01b1-7095-480e-ac45-e3a3cce3cfe5\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-qcqpm" Jul 08 18:41:11.675719 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:11.675702 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f18a01b1-7095-480e-ac45-e3a3cce3cfe5-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-qcqpm\" (UID: \"f18a01b1-7095-480e-ac45-e3a3cce3cfe5\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-qcqpm" Jul 08 18:41:11.731664 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:11.731632 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cg9vg\"" Jul 08 18:41:11.738963 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:11.738945 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7drlq" Jul 08 18:41:11.832310 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:11.832276 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-qzbn7\"" Jul 08 18:41:11.839638 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:11.839611 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-qcqpm" Jul 08 18:41:11.865035 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:11.865006 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7drlq"] Jul 08 18:41:11.868257 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:41:11.868227 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cdaf3eb_a0ab_4001_a1d6_489fad86e0a7.slice/crio-f61212126f07edf6173c50643c613c4c36eec732e927592b8122eab2f82f53b5 WatchSource:0}: Error finding container f61212126f07edf6173c50643c613c4c36eec732e927592b8122eab2f82f53b5: Status 404 returned error can't find the container with id f61212126f07edf6173c50643c613c4c36eec732e927592b8122eab2f82f53b5 Jul 08 18:41:11.964547 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:11.964502 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-qcqpm"] Jul 08 18:41:11.967234 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:41:11.967209 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf18a01b1_7095_480e_ac45_e3a3cce3cfe5.slice/crio-df7b2f992097497f7801364fcefccf02f08e4b9255012fcfd4583932ab66fa8f WatchSource:0}: Error finding container df7b2f992097497f7801364fcefccf02f08e4b9255012fcfd4583932ab66fa8f: Status 404 returned error can't find the container with id df7b2f992097497f7801364fcefccf02f08e4b9255012fcfd4583932ab66fa8f Jul 08 18:41:12.077168 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:12.077083 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ce8ea612-0ed6-41b9-a621-a1411751786b-klusterlet-config\") pod \"klusterlet-addon-workmgr-864d54dd8d-k7l9n\" (UID: \"ce8ea612-0ed6-41b9-a621-a1411751786b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864d54dd8d-k7l9n" Jul 08 18:41:12.079841 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:12.079810 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ce8ea612-0ed6-41b9-a621-a1411751786b-klusterlet-config\") pod \"klusterlet-addon-workmgr-864d54dd8d-k7l9n\" (UID: \"ce8ea612-0ed6-41b9-a621-a1411751786b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864d54dd8d-k7l9n" Jul 08 18:41:12.130505 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:12.130459 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7drlq" event={"ID":"3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7","Type":"ContainerStarted","Data":"f61212126f07edf6173c50643c613c4c36eec732e927592b8122eab2f82f53b5"} Jul 08 18:41:12.132001 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:12.131970 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-qcqpm" event={"ID":"f18a01b1-7095-480e-ac45-e3a3cce3cfe5","Type":"ContainerStarted","Data":"df7b2f992097497f7801364fcefccf02f08e4b9255012fcfd4583932ab66fa8f"} Jul 08 18:41:12.200796 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:12.200630 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864d54dd8d-k7l9n" Jul 08 18:41:12.357586 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:12.357520 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-864d54dd8d-k7l9n"] Jul 08 18:41:12.361575 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:41:12.361543 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce8ea612_0ed6_41b9_a621_a1411751786b.slice/crio-8e66f235c6c096340be6f541ca64adae5dbaf664b29c84e0964a77a98f0748f8 WatchSource:0}: Error finding container 8e66f235c6c096340be6f541ca64adae5dbaf664b29c84e0964a77a98f0748f8: Status 404 returned error can't find the container with id 8e66f235c6c096340be6f541ca64adae5dbaf664b29c84e0964a77a98f0748f8 Jul 08 18:41:13.138494 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:13.138447 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864d54dd8d-k7l9n" event={"ID":"ce8ea612-0ed6-41b9-a621-a1411751786b","Type":"ContainerStarted","Data":"8e66f235c6c096340be6f541ca64adae5dbaf664b29c84e0964a77a98f0748f8"} Jul 08 18:41:15.149278 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:15.148855 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b9679dd58-rrf82" event={"ID":"8274723b-a4ff-4cea-8697-c9d972e321ac","Type":"ContainerStarted","Data":"d09800639897529f13bda3108ac14cff73aebeb3a56473a6e2c5d5bf68fbc7c3"} Jul 08 18:41:15.150887 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:15.150858 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7drlq" event={"ID":"3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7","Type":"ContainerStarted","Data":"b09b39ef306466801e7741136a8bb12fb80115b270504c03c9aa3b0e70b93cfa"} Jul 08 18:41:15.152630 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:15.152583 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-qcqpm" event={"ID":"f18a01b1-7095-480e-ac45-e3a3cce3cfe5","Type":"ContainerStarted","Data":"7d371f1603c364e9ba65f99ece29b0b41717c1c1d626efdcd134afa045e8a3e5"} Jul 08 18:41:15.165359 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:15.165276 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b9679dd58-rrf82" podStartSLOduration=1.12379981 podStartE2EDuration="5.165258365s" totalImagesPullingTime="4.041458555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:41:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:41:15.164374874 +0000 UTC m=+86.001702607" watchObservedRunningTime="2026-07-08 18:41:15.165258365 +0000 UTC m=+86.002586075" Jul 08 18:41:15.192232 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:15.192175 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-qcqpm" podStartSLOduration=33.283636876 podStartE2EDuration="36.192154649s" totalImagesPullingTime="2.908517773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:41:15.191680681 +0000 UTC m=+86.029008390" watchObservedRunningTime="2026-07-08 18:41:15.192154649 +0000 UTC m=+86.029482352" Jul 08 18:41:15.218782 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:15.218691 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-7drlq" podStartSLOduration=33.211111612 podStartE2EDuration="36.218670313s" totalImagesPullingTime="3.007558701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:41:15.217272356 +0000 UTC m=+86.054600082" watchObservedRunningTime="2026-07-08 18:41:15.218670313 +0000 UTC m=+86.055998021" Jul 08 18:41:16.104980 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:16.104943 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ljtbq" Jul 08 18:41:17.162207 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:17.162152 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864d54dd8d-k7l9n" event={"ID":"ce8ea612-0ed6-41b9-a621-a1411751786b","Type":"ContainerStarted","Data":"41bff4a576a4a76e71f25290fe2c2ad37a87680c7c2f97cf413e468870725884"} Jul 08 18:41:17.162666 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:17.162330 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864d54dd8d-k7l9n" Jul 08 18:41:17.164056 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:17.164035 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864d54dd8d-k7l9n" Jul 08 18:41:17.195766 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:17.195712 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-864d54dd8d-k7l9n" podStartSLOduration=2.900604701 podStartE2EDuration="7.195694463s" totalImagesPullingTime="4.295089762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:41:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:41:17.194422106 +0000 UTC m=+88.031749834" watchObservedRunningTime="2026-07-08 18:41:17.195694463 +0000 UTC m=+88.033022170" Jul 08 18:41:23.961348 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:23.961311 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gx7nm"] Jul 08 18:41:23.966422 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:23.966398 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:23.972965 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:23.972943 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 08 18:41:23.973368 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:23.973343 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 08 18:41:23.974654 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:23.974622 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 08 18:41:23.974654 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:23.974638 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-846kr\"" Jul 08 18:41:23.974816 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:23.974741 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 08 18:41:24.081071 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.081039 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-sys\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.081269 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.081081 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.081269 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.081108 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-node-exporter-accelerators-collector-config\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.081269 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.081199 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpz7z\" (UniqueName: \"kubernetes.io/projected/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-kube-api-access-jpz7z\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.081269 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.081250 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-root\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.081507 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.081374 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-metrics-client-ca\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.081507 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.081408 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-node-exporter-wtmp\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.081507 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.081426 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-node-exporter-textfile\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.081507 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.081449 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-node-exporter-tls\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.182487 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.182442 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-sys\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.182734 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.182511 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.182734 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.182542 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-node-exporter-accelerators-collector-config\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.182734 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.182561 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-sys\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.182734 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.182577 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpz7z\" (UniqueName: \"kubernetes.io/projected/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-kube-api-access-jpz7z\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.182734 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.182636 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-root\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.182734 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.182686 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-metrics-client-ca\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.182734 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.182713 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-root\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.182734 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.182734 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-node-exporter-wtmp\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.183120 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.182761 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-node-exporter-textfile\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.183120 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.182791 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-node-exporter-tls\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.183120 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:41:24.182892 2575 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 08 18:41:24.183120 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.182921 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-node-exporter-wtmp\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.183120 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:41:24.182949 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-node-exporter-tls podName:6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286 nodeName:}" failed. No retries permitted until 2026-07-08 18:41:24.682930113 +0000 UTC m=+95.520257813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-node-exporter-tls") pod "node-exporter-gx7nm" (UID: "6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286") : secret "node-exporter-tls" not found Jul 08 18:41:24.183120 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.183101 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-node-exporter-textfile\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.183461 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.183328 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-node-exporter-accelerators-collector-config\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.183547 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.183529 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-metrics-client-ca\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.185507 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.185483 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.192722 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.192695 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpz7z\" (UniqueName: \"kubernetes.io/projected/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-kube-api-access-jpz7z\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.686804 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.686762 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-node-exporter-tls\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.689143 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.689116 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286-node-exporter-tls\") pod \"node-exporter-gx7nm\" (UID: \"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286\") " pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.877368 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:24.877340 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gx7nm" Jul 08 18:41:24.892661 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:41:24.892632 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e28f6a7_ea12_4dfb_a1f1_fb7a1e2d4286.slice/crio-55da49011a3d293d2520b5e83b7534152ccbf01e6f20b1dfb4db98592f9575e3 WatchSource:0}: Error finding container 55da49011a3d293d2520b5e83b7534152ccbf01e6f20b1dfb4db98592f9575e3: Status 404 returned error can't find the container with id 55da49011a3d293d2520b5e83b7534152ccbf01e6f20b1dfb4db98592f9575e3 Jul 08 18:41:25.191016 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:25.190975 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gx7nm" event={"ID":"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286","Type":"ContainerStarted","Data":"55da49011a3d293d2520b5e83b7534152ccbf01e6f20b1dfb4db98592f9575e3"} Jul 08 18:41:26.196523 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:26.196487 2575 generic.go:356] "Generic (PLEG): container finished" podID="6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286" containerID="5830cb5e2eb96e4e12e00185b3307eab0accaf18f190e8a1907beadd7ee1ba08" exitCode=0 Jul 08 18:41:26.196894 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:26.196569 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gx7nm" event={"ID":"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286","Type":"ContainerDied","Data":"5830cb5e2eb96e4e12e00185b3307eab0accaf18f190e8a1907beadd7ee1ba08"} Jul 08 18:41:27.201768 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:27.201733 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gx7nm" event={"ID":"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286","Type":"ContainerStarted","Data":"cb4fe0c0ae2a302d7cbcc0c6e1a3a66d8c92ff80b2f9001623dc7841ba617528"} Jul 08 18:41:27.202242 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:27.201793 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gx7nm" event={"ID":"6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286","Type":"ContainerStarted","Data":"5a867ca761d3bd1e6d020f651c0020c265e7b1cf98ba3422be5c83ab975f4c74"} Jul 08 18:41:27.227504 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:27.227455 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gx7nm" podStartSLOduration=3.026542337 podStartE2EDuration="4.227439917s" totalImagesPullingTime="1.138860459s" totalInitContainerRuntime="62.037121ms" isStatefulPod=true podCreationTimestamp="2026-07-08 18:41:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:41:27.226630174 +0000 UTC m=+98.063957881" watchObservedRunningTime="2026-07-08 18:41:27.227439917 +0000 UTC m=+98.064767624" Jul 08 18:41:28.697203 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:28.697166 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-hckvj"] Jul 08 18:41:28.700450 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:28.700430 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-hckvj" Jul 08 18:41:28.703407 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:28.703379 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 08 18:41:28.703536 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:28.703449 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-gqfw4\"" Jul 08 18:41:28.711068 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:28.711047 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-hckvj"] Jul 08 18:41:28.716110 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:28.716082 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ls6l\" (UniqueName: \"kubernetes.io/projected/dc93f7a0-7dd6-4c0e-b213-68624306f061-kube-api-access-7ls6l\") pod \"monitoring-plugin-d5dcff7cf-hckvj\" (UID: \"dc93f7a0-7dd6-4c0e-b213-68624306f061\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-hckvj" Jul 08 18:41:28.716220 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:28.716188 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc93f7a0-7dd6-4c0e-b213-68624306f061-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-hckvj\" (UID: \"dc93f7a0-7dd6-4c0e-b213-68624306f061\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-hckvj" Jul 08 18:41:28.816627 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:28.816591 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ls6l\" (UniqueName: \"kubernetes.io/projected/dc93f7a0-7dd6-4c0e-b213-68624306f061-kube-api-access-7ls6l\") pod \"monitoring-plugin-d5dcff7cf-hckvj\" (UID: \"dc93f7a0-7dd6-4c0e-b213-68624306f061\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-hckvj" Jul 08 18:41:28.816800 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:28.816650 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc93f7a0-7dd6-4c0e-b213-68624306f061-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-hckvj\" (UID: \"dc93f7a0-7dd6-4c0e-b213-68624306f061\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-hckvj" Jul 08 18:41:28.816800 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:41:28.816744 2575 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 08 18:41:28.816800 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:41:28.816800 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc93f7a0-7dd6-4c0e-b213-68624306f061-monitoring-plugin-cert podName:dc93f7a0-7dd6-4c0e-b213-68624306f061 nodeName:}" failed. No retries permitted until 2026-07-08 18:41:29.316781938 +0000 UTC m=+100.154109642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/dc93f7a0-7dd6-4c0e-b213-68624306f061-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-hckvj" (UID: "dc93f7a0-7dd6-4c0e-b213-68624306f061") : secret "monitoring-plugin-cert" not found Jul 08 18:41:28.836085 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:28.836046 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ls6l\" (UniqueName: \"kubernetes.io/projected/dc93f7a0-7dd6-4c0e-b213-68624306f061-kube-api-access-7ls6l\") pod \"monitoring-plugin-d5dcff7cf-hckvj\" (UID: \"dc93f7a0-7dd6-4c0e-b213-68624306f061\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-hckvj" Jul 08 18:41:29.320559 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:29.320525 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc93f7a0-7dd6-4c0e-b213-68624306f061-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-hckvj\" (UID: \"dc93f7a0-7dd6-4c0e-b213-68624306f061\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-hckvj" Jul 08 18:41:29.323024 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:29.322999 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc93f7a0-7dd6-4c0e-b213-68624306f061-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-hckvj\" (UID: \"dc93f7a0-7dd6-4c0e-b213-68624306f061\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-hckvj" Jul 08 18:41:29.609948 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:29.609916 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-hckvj" Jul 08 18:41:29.745998 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:29.745975 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-hckvj"] Jul 08 18:41:29.748358 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:41:29.748331 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc93f7a0_7dd6_4c0e_b213_68624306f061.slice/crio-72f9c3767863f9d79c93c01d7a353e295cd9145f1364cdb48690fd1890b15f0a WatchSource:0}: Error finding container 72f9c3767863f9d79c93c01d7a353e295cd9145f1364cdb48690fd1890b15f0a: Status 404 returned error can't find the container with id 72f9c3767863f9d79c93c01d7a353e295cd9145f1364cdb48690fd1890b15f0a Jul 08 18:41:30.213070 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:30.213041 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-hckvj" event={"ID":"dc93f7a0-7dd6-4c0e-b213-68624306f061","Type":"ContainerStarted","Data":"72f9c3767863f9d79c93c01d7a353e295cd9145f1364cdb48690fd1890b15f0a"} Jul 08 18:41:32.136029 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:32.136001 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d6d887bf8-98j5w" Jul 08 18:41:32.222708 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:32.222674 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-hckvj" event={"ID":"dc93f7a0-7dd6-4c0e-b213-68624306f061","Type":"ContainerStarted","Data":"7ffdc183b4192bb1b60720406aa6ef67bec2525341f67f07066d7e346af86777"} Jul 08 18:41:32.222871 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:32.222822 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-hckvj" Jul 08 18:41:32.227429 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:32.227406 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-hckvj" Jul 08 18:41:32.244999 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:32.244937 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-hckvj" podStartSLOduration=2.8255922079999998 podStartE2EDuration="4.244920878s" totalImagesPullingTime="1.41932867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:41:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:41:32.243370777 +0000 UTC m=+103.080698485" watchObservedRunningTime="2026-07-08 18:41:32.244920878 +0000 UTC m=+103.082248633" Jul 08 18:41:33.081748 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:33.081718 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pwffr" Jul 08 18:41:55.308172 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:55.308134 2575 generic.go:356] "Generic (PLEG): container finished" podID="aedf6d82-9258-424b-a9f0-af1f0154058b" containerID="f6d9faae29caca4b72e3ba0e7503a56ccc4fbfed817dc4c2e1c3d498aaa9d6f9" exitCode=0 Jul 08 18:41:55.308650 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:55.308208 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2wffx" event={"ID":"aedf6d82-9258-424b-a9f0-af1f0154058b","Type":"ContainerDied","Data":"f6d9faae29caca4b72e3ba0e7503a56ccc4fbfed817dc4c2e1c3d498aaa9d6f9"} Jul 08 18:41:55.308650 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:55.308565 2575 scope.go:117] "RemoveContainer" containerID="f6d9faae29caca4b72e3ba0e7503a56ccc4fbfed817dc4c2e1c3d498aaa9d6f9" Jul 08 18:41:55.309923 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:55.309792 2575 generic.go:356] "Generic (PLEG): container finished" podID="0e3be5af-3630-43f1-80b1-daf23fb9b78b" containerID="98f4737e5eb0409b81179a069c1ff34c55f621d18e84067f9d2bfb7bb3b04498" exitCode=0 Jul 08 18:41:55.309923 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:55.309833 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-p4gn9" event={"ID":"0e3be5af-3630-43f1-80b1-daf23fb9b78b","Type":"ContainerDied","Data":"98f4737e5eb0409b81179a069c1ff34c55f621d18e84067f9d2bfb7bb3b04498"} Jul 08 18:41:55.310084 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:55.310068 2575 scope.go:117] "RemoveContainer" containerID="98f4737e5eb0409b81179a069c1ff34c55f621d18e84067f9d2bfb7bb3b04498" Jul 08 18:41:56.315347 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:56.315306 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-2wffx" event={"ID":"aedf6d82-9258-424b-a9f0-af1f0154058b","Type":"ContainerStarted","Data":"98536c5bffbd440f8df374ab43bd93967172d7b33eabb13cc9a792d6a647459d"} Jul 08 18:41:56.317041 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:56.317010 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-p4gn9" event={"ID":"0e3be5af-3630-43f1-80b1-daf23fb9b78b","Type":"ContainerStarted","Data":"a700446dcf75e248e96f4f9f68ae80edc3e752529b428c0ddc764bec5240fb66"} Jul 08 18:41:59.462933 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:59.462895 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee39c730-7ca1-4260-afea-a884940c00fc-metrics-certs\") pod \"network-metrics-daemon-252x5\" (UID: \"ee39c730-7ca1-4260-afea-a884940c00fc\") " pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:41:59.465195 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:59.465166 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee39c730-7ca1-4260-afea-a884940c00fc-metrics-certs\") pod \"network-metrics-daemon-252x5\" (UID: \"ee39c730-7ca1-4260-afea-a884940c00fc\") " pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:41:59.696480 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:59.696448 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sc5vc\"" Jul 08 18:41:59.703896 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:59.703868 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-252x5" Jul 08 18:41:59.829257 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:41:59.829227 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-252x5"] Jul 08 18:41:59.831978 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:41:59.831951 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee39c730_7ca1_4260_afea_a884940c00fc.slice/crio-0a80f69daf64084c4c9f1714ae50922fb0bd146b0d1fd3ba24a9ea76c81f888f WatchSource:0}: Error finding container 0a80f69daf64084c4c9f1714ae50922fb0bd146b0d1fd3ba24a9ea76c81f888f: Status 404 returned error can't find the container with id 0a80f69daf64084c4c9f1714ae50922fb0bd146b0d1fd3ba24a9ea76c81f888f Jul 08 18:42:00.334356 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:42:00.334320 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-252x5" event={"ID":"ee39c730-7ca1-4260-afea-a884940c00fc","Type":"ContainerStarted","Data":"0a80f69daf64084c4c9f1714ae50922fb0bd146b0d1fd3ba24a9ea76c81f888f"} Jul 08 18:42:01.340573 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:42:01.340482 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-252x5" event={"ID":"ee39c730-7ca1-4260-afea-a884940c00fc","Type":"ContainerStarted","Data":"079367b65a235e912c258ea2cb33e2d3599a5f4d556c059fc1f72a2e9831c42d"} Jul 08 18:42:01.340573 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:42:01.340533 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-252x5" event={"ID":"ee39c730-7ca1-4260-afea-a884940c00fc","Type":"ContainerStarted","Data":"ba575f188753196afa68cd31d91b30b19d321bad6e2736df966a1d5cdec92dc2"} Jul 08 18:42:01.357888 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:42:01.357840 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-252x5" podStartSLOduration=130.156411959 podStartE2EDuration="2m11.357825293s" totalImagesPullingTime="1.201413334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:39:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:42:01.357281203 +0000 UTC m=+132.194608911" watchObservedRunningTime="2026-07-08 18:42:01.357825293 +0000 UTC m=+132.195153002" Jul 08 18:44:49.637635 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:44:49.637609 2575 kubelet.go:1648] "Image garbage collection succeeded" Jul 08 18:45:45.755954 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:45:45.755922 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4pw65"] Jul 08 18:45:45.758879 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:45:45.758862 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4pw65" Jul 08 18:45:45.761594 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:45:45.761571 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 08 18:45:45.761713 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:45:45.761632 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-s4k8n\"" Jul 08 18:45:45.761713 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:45:45.761634 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 08 18:45:45.769872 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:45:45.769847 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4pw65"] Jul 08 18:45:45.860934 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:45:45.860892 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qt9r\" (UniqueName: \"kubernetes.io/projected/b1f22463-b8a0-4a4d-973b-21f1a8db4c30-kube-api-access-4qt9r\") pod \"openshift-lws-operator-fd8ccff4c-4pw65\" (UID: \"b1f22463-b8a0-4a4d-973b-21f1a8db4c30\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4pw65" Jul 08 18:45:45.861124 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:45:45.860992 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1f22463-b8a0-4a4d-973b-21f1a8db4c30-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4pw65\" (UID: \"b1f22463-b8a0-4a4d-973b-21f1a8db4c30\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4pw65" Jul 08 18:45:45.961994 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:45:45.961954 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qt9r\" (UniqueName: \"kubernetes.io/projected/b1f22463-b8a0-4a4d-973b-21f1a8db4c30-kube-api-access-4qt9r\") pod \"openshift-lws-operator-fd8ccff4c-4pw65\" (UID: \"b1f22463-b8a0-4a4d-973b-21f1a8db4c30\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4pw65" Jul 08 18:45:45.962155 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:45:45.962003 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1f22463-b8a0-4a4d-973b-21f1a8db4c30-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4pw65\" (UID: \"b1f22463-b8a0-4a4d-973b-21f1a8db4c30\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4pw65" Jul 08 18:45:45.962335 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:45:45.962318 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1f22463-b8a0-4a4d-973b-21f1a8db4c30-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4pw65\" (UID: \"b1f22463-b8a0-4a4d-973b-21f1a8db4c30\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4pw65" Jul 08 18:45:45.971144 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:45:45.971120 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qt9r\" (UniqueName: \"kubernetes.io/projected/b1f22463-b8a0-4a4d-973b-21f1a8db4c30-kube-api-access-4qt9r\") pod \"openshift-lws-operator-fd8ccff4c-4pw65\" (UID: \"b1f22463-b8a0-4a4d-973b-21f1a8db4c30\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4pw65" Jul 08 18:45:46.068072 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:45:46.067987 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4pw65" Jul 08 18:45:46.206701 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:45:46.206665 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4pw65"] Jul 08 18:45:46.209818 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:45:46.209788 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1f22463_b8a0_4a4d_973b_21f1a8db4c30.slice/crio-69727cb5af13a37890ac59f8430ef825dfa5a97af451021681e4bb2387490f00 WatchSource:0}: Error finding container 69727cb5af13a37890ac59f8430ef825dfa5a97af451021681e4bb2387490f00: Status 404 returned error can't find the container with id 69727cb5af13a37890ac59f8430ef825dfa5a97af451021681e4bb2387490f00 Jul 08 18:45:46.211324 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:45:46.211284 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 18:45:47.143165 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:45:47.143116 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4pw65" event={"ID":"b1f22463-b8a0-4a4d-973b-21f1a8db4c30","Type":"ContainerStarted","Data":"69727cb5af13a37890ac59f8430ef825dfa5a97af451021681e4bb2387490f00"} Jul 08 18:45:49.153167 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:45:49.153129 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4pw65" event={"ID":"b1f22463-b8a0-4a4d-973b-21f1a8db4c30","Type":"ContainerStarted","Data":"b4a4ff21e23b7978a841edd1c4b95d8ada171de08382b6ecd162a7745da462a3"} Jul 08 18:45:49.178013 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:45:49.177960 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4pw65" podStartSLOduration=1.5354570459999999 podStartE2EDuration="4.177945421s" totalImagesPullingTime="2.642488375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:45:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:45:49.175748583 +0000 UTC m=+360.013076292" watchObservedRunningTime="2026-07-08 18:45:49.177945421 +0000 UTC m=+360.015273129" Jul 08 18:46:07.438282 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:07.438249 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c5cf545b9-r59jn"] Jul 08 18:46:07.442382 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:07.442361 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c5cf545b9-r59jn" Jul 08 18:46:07.445184 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:07.445163 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 08 18:46:07.445310 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:07.445223 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-56jvk\"" Jul 08 18:46:07.445687 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:07.445670 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 08 18:46:07.445732 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:07.445686 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 08 18:46:07.445777 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:07.445737 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 08 18:46:07.455151 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:07.455131 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c5cf545b9-r59jn"] Jul 08 18:46:07.528332 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:07.528279 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/34dd0db6-dd56-49aa-8302-1d90fd9c31cf-manifests\") pod \"opendatahub-operator-controller-manager-6c5cf545b9-r59jn\" (UID: \"34dd0db6-dd56-49aa-8302-1d90fd9c31cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5cf545b9-r59jn" Jul 08 18:46:07.528332 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:07.528335 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/34dd0db6-dd56-49aa-8302-1d90fd9c31cf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c5cf545b9-r59jn\" (UID: \"34dd0db6-dd56-49aa-8302-1d90fd9c31cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5cf545b9-r59jn" Jul 08 18:46:07.528535 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:07.528360 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n647t\" (UniqueName: \"kubernetes.io/projected/34dd0db6-dd56-49aa-8302-1d90fd9c31cf-kube-api-access-n647t\") pod \"opendatahub-operator-controller-manager-6c5cf545b9-r59jn\" (UID: \"34dd0db6-dd56-49aa-8302-1d90fd9c31cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5cf545b9-r59jn" Jul 08 18:46:07.528535 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:07.528391 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34dd0db6-dd56-49aa-8302-1d90fd9c31cf-tmp\") pod \"opendatahub-operator-controller-manager-6c5cf545b9-r59jn\" (UID: \"34dd0db6-dd56-49aa-8302-1d90fd9c31cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5cf545b9-r59jn" Jul 08 18:46:07.528535 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:07.528465 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34dd0db6-dd56-49aa-8302-1d90fd9c31cf-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c5cf545b9-r59jn\" (UID: \"34dd0db6-dd56-49aa-8302-1d90fd9c31cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5cf545b9-r59jn" Jul 08 18:46:07.628836 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:07.628799 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34dd0db6-dd56-49aa-8302-1d90fd9c31cf-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c5cf545b9-r59jn\" (UID: \"34dd0db6-dd56-49aa-8302-1d90fd9c31cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5cf545b9-r59jn" Jul 08 18:46:07.628992 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:07.628863 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/34dd0db6-dd56-49aa-8302-1d90fd9c31cf-manifests\") pod \"opendatahub-operator-controller-manager-6c5cf545b9-r59jn\" (UID: \"34dd0db6-dd56-49aa-8302-1d90fd9c31cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5cf545b9-r59jn" Jul 08 18:46:07.628992 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:07.628894 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/34dd0db6-dd56-49aa-8302-1d90fd9c31cf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c5cf545b9-r59jn\" (UID: \"34dd0db6-dd56-49aa-8302-1d90fd9c31cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5cf545b9-r59jn" Jul 08 18:46:07.628992 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:07.628924 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n647t\" (UniqueName: \"kubernetes.io/projected/34dd0db6-dd56-49aa-8302-1d90fd9c31cf-kube-api-access-n647t\") pod \"opendatahub-operator-controller-manager-6c5cf545b9-r59jn\" (UID: \"34dd0db6-dd56-49aa-8302-1d90fd9c31cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5cf545b9-r59jn" Jul 08 18:46:07.628992 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:07.628972 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34dd0db6-dd56-49aa-8302-1d90fd9c31cf-tmp\") pod \"opendatahub-operator-controller-manager-6c5cf545b9-r59jn\" (UID: \"34dd0db6-dd56-49aa-8302-1d90fd9c31cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5cf545b9-r59jn" Jul 08 18:46:07.629277 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:07.629256 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/34dd0db6-dd56-49aa-8302-1d90fd9c31cf-manifests\") pod \"opendatahub-operator-controller-manager-6c5cf545b9-r59jn\" (UID: \"34dd0db6-dd56-49aa-8302-1d90fd9c31cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5cf545b9-r59jn" Jul 08 18:46:07.631688 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:07.631660 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/34dd0db6-dd56-49aa-8302-1d90fd9c31cf-tmp\") pod \"opendatahub-operator-controller-manager-6c5cf545b9-r59jn\" (UID: \"34dd0db6-dd56-49aa-8302-1d90fd9c31cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5cf545b9-r59jn" Jul 08 18:46:07.631839 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:07.631814 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/34dd0db6-dd56-49aa-8302-1d90fd9c31cf-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c5cf545b9-r59jn\" (UID: \"34dd0db6-dd56-49aa-8302-1d90fd9c31cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5cf545b9-r59jn" Jul 08 18:46:07.631937 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:07.631922 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/34dd0db6-dd56-49aa-8302-1d90fd9c31cf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c5cf545b9-r59jn\" (UID: \"34dd0db6-dd56-49aa-8302-1d90fd9c31cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5cf545b9-r59jn" Jul 08 18:46:07.644502 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:07.644481 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n647t\" (UniqueName: \"kubernetes.io/projected/34dd0db6-dd56-49aa-8302-1d90fd9c31cf-kube-api-access-n647t\") pod \"opendatahub-operator-controller-manager-6c5cf545b9-r59jn\" (UID: \"34dd0db6-dd56-49aa-8302-1d90fd9c31cf\") " pod="opendatahub/opendatahub-operator-controller-manager-6c5cf545b9-r59jn" Jul 08 18:46:07.752728 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:07.752642 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c5cf545b9-r59jn" Jul 08 18:46:07.893568 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:07.893544 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c5cf545b9-r59jn"] Jul 08 18:46:07.895058 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:46:07.895020 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34dd0db6_dd56_49aa_8302_1d90fd9c31cf.slice/crio-bbfd3935c36e1e432bebc0e6956dcec13b483bf63820f0c0eb358aaffe2ca9ba WatchSource:0}: Error finding container bbfd3935c36e1e432bebc0e6956dcec13b483bf63820f0c0eb358aaffe2ca9ba: Status 404 returned error can't find the container with id bbfd3935c36e1e432bebc0e6956dcec13b483bf63820f0c0eb358aaffe2ca9ba Jul 08 18:46:08.222526 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:08.222489 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c5cf545b9-r59jn" event={"ID":"34dd0db6-dd56-49aa-8302-1d90fd9c31cf","Type":"ContainerStarted","Data":"bbfd3935c36e1e432bebc0e6956dcec13b483bf63820f0c0eb358aaffe2ca9ba"} Jul 08 18:46:11.236211 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:11.236180 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c5cf545b9-r59jn" event={"ID":"34dd0db6-dd56-49aa-8302-1d90fd9c31cf","Type":"ContainerStarted","Data":"20155c577232913e3015aa161ad63ef53349399107658886bb76de565a46de75"} Jul 08 18:46:13.247921 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:13.247885 2575 generic.go:356] "Generic (PLEG): container finished" podID="34dd0db6-dd56-49aa-8302-1d90fd9c31cf" containerID="20155c577232913e3015aa161ad63ef53349399107658886bb76de565a46de75" exitCode=0 Jul 08 18:46:13.248439 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:13.247940 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c5cf545b9-r59jn" event={"ID":"34dd0db6-dd56-49aa-8302-1d90fd9c31cf","Type":"ContainerDied","Data":"20155c577232913e3015aa161ad63ef53349399107658886bb76de565a46de75"} Jul 08 18:46:14.253440 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:14.253407 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c5cf545b9-r59jn" event={"ID":"34dd0db6-dd56-49aa-8302-1d90fd9c31cf","Type":"ContainerStarted","Data":"3c098410159721a63f83e289a226904ed8b3d679f7bf7f226cb9fc17526e91d9"} Jul 08 18:46:14.253828 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:14.253617 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6c5cf545b9-r59jn" Jul 08 18:46:25.259639 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:25.259604 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6c5cf545b9-r59jn" Jul 08 18:46:25.287007 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:25.286954 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6c5cf545b9-r59jn" podStartSLOduration=13.684364749 podStartE2EDuration="18.286938173s" totalImagesPullingTime="3.515203265s" totalInitContainerRuntime="1.087370159s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:46:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:14.304690655 +0000 UTC m=+385.142018363" watchObservedRunningTime="2026-07-08 18:46:25.286938173 +0000 UTC m=+396.124265881" Jul 08 18:46:27.565354 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:27.565312 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-8b75dd56d-wk2ph"] Jul 08 18:46:27.573864 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:27.573836 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8b75dd56d-wk2ph" Jul 08 18:46:27.578854 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:27.578826 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 08 18:46:27.578854 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:27.578838 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 08 18:46:27.578854 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:27.578826 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 08 18:46:27.579062 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:27.578827 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 08 18:46:27.579062 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:27.578827 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-dd6dt\"" Jul 08 18:46:27.598256 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:27.598234 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8b75dd56d-wk2ph"] Jul 08 18:46:27.686104 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:27.686069 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxb5j\" (UniqueName: \"kubernetes.io/projected/dc05552e-bd5b-46e6-95df-b70171c218f5-kube-api-access-zxb5j\") pod \"kube-auth-proxy-8b75dd56d-wk2ph\" (UID: \"dc05552e-bd5b-46e6-95df-b70171c218f5\") " pod="openshift-ingress/kube-auth-proxy-8b75dd56d-wk2ph" Jul 08 18:46:27.686285 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:27.686121 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc05552e-bd5b-46e6-95df-b70171c218f5-tls-certs\") pod \"kube-auth-proxy-8b75dd56d-wk2ph\" (UID: \"dc05552e-bd5b-46e6-95df-b70171c218f5\") " pod="openshift-ingress/kube-auth-proxy-8b75dd56d-wk2ph" Jul 08 18:46:27.686285 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:27.686145 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc05552e-bd5b-46e6-95df-b70171c218f5-tmp\") pod \"kube-auth-proxy-8b75dd56d-wk2ph\" (UID: \"dc05552e-bd5b-46e6-95df-b70171c218f5\") " pod="openshift-ingress/kube-auth-proxy-8b75dd56d-wk2ph" Jul 08 18:46:27.786661 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:27.786630 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc05552e-bd5b-46e6-95df-b70171c218f5-tmp\") pod \"kube-auth-proxy-8b75dd56d-wk2ph\" (UID: \"dc05552e-bd5b-46e6-95df-b70171c218f5\") " pod="openshift-ingress/kube-auth-proxy-8b75dd56d-wk2ph" Jul 08 18:46:27.786820 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:27.786701 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxb5j\" (UniqueName: \"kubernetes.io/projected/dc05552e-bd5b-46e6-95df-b70171c218f5-kube-api-access-zxb5j\") pod \"kube-auth-proxy-8b75dd56d-wk2ph\" (UID: \"dc05552e-bd5b-46e6-95df-b70171c218f5\") " pod="openshift-ingress/kube-auth-proxy-8b75dd56d-wk2ph" Jul 08 18:46:27.786820 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:27.786762 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc05552e-bd5b-46e6-95df-b70171c218f5-tls-certs\") pod \"kube-auth-proxy-8b75dd56d-wk2ph\" (UID: \"dc05552e-bd5b-46e6-95df-b70171c218f5\") " pod="openshift-ingress/kube-auth-proxy-8b75dd56d-wk2ph" Jul 08 18:46:27.788847 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:27.788811 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc05552e-bd5b-46e6-95df-b70171c218f5-tmp\") pod \"kube-auth-proxy-8b75dd56d-wk2ph\" (UID: \"dc05552e-bd5b-46e6-95df-b70171c218f5\") " pod="openshift-ingress/kube-auth-proxy-8b75dd56d-wk2ph" Jul 08 18:46:27.789051 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:27.789034 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc05552e-bd5b-46e6-95df-b70171c218f5-tls-certs\") pod \"kube-auth-proxy-8b75dd56d-wk2ph\" (UID: \"dc05552e-bd5b-46e6-95df-b70171c218f5\") " pod="openshift-ingress/kube-auth-proxy-8b75dd56d-wk2ph" Jul 08 18:46:27.822460 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:27.822400 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxb5j\" (UniqueName: \"kubernetes.io/projected/dc05552e-bd5b-46e6-95df-b70171c218f5-kube-api-access-zxb5j\") pod \"kube-auth-proxy-8b75dd56d-wk2ph\" (UID: \"dc05552e-bd5b-46e6-95df-b70171c218f5\") " pod="openshift-ingress/kube-auth-proxy-8b75dd56d-wk2ph" Jul 08 18:46:27.883273 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:27.883241 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8b75dd56d-wk2ph" Jul 08 18:46:27.995783 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:27.995734 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8vw4w"] Jul 08 18:46:27.999976 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:27.999955 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8vw4w" Jul 08 18:46:28.012077 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:28.012053 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-qb8jn\"" Jul 08 18:46:28.014691 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:28.014671 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 08 18:46:28.019526 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:28.019503 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8vw4w"] Jul 08 18:46:28.043793 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:28.043629 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8b75dd56d-wk2ph"] Jul 08 18:46:28.046001 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:46:28.045977 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc05552e_bd5b_46e6_95df_b70171c218f5.slice/crio-e66635d7537babc6c9e825f50fbf8f06f4e65724960ebcaeeb6acc67ff260d3f WatchSource:0}: Error finding container e66635d7537babc6c9e825f50fbf8f06f4e65724960ebcaeeb6acc67ff260d3f: Status 404 returned error can't find the container with id e66635d7537babc6c9e825f50fbf8f06f4e65724960ebcaeeb6acc67ff260d3f Jul 08 18:46:28.189832 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:28.189800 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtw45\" (UniqueName: \"kubernetes.io/projected/eeae891b-9bad-4f61-915c-f0f4cac1bc18-kube-api-access-qtw45\") pod \"odh-model-controller-7998c67dc4-8vw4w\" (UID: \"eeae891b-9bad-4f61-915c-f0f4cac1bc18\") " pod="opendatahub/odh-model-controller-7998c67dc4-8vw4w" Jul 08 18:46:28.189982 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:28.189845 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eeae891b-9bad-4f61-915c-f0f4cac1bc18-cert\") pod \"odh-model-controller-7998c67dc4-8vw4w\" (UID: \"eeae891b-9bad-4f61-915c-f0f4cac1bc18\") " pod="opendatahub/odh-model-controller-7998c67dc4-8vw4w" Jul 08 18:46:28.290806 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:28.290767 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtw45\" (UniqueName: \"kubernetes.io/projected/eeae891b-9bad-4f61-915c-f0f4cac1bc18-kube-api-access-qtw45\") pod \"odh-model-controller-7998c67dc4-8vw4w\" (UID: \"eeae891b-9bad-4f61-915c-f0f4cac1bc18\") " pod="opendatahub/odh-model-controller-7998c67dc4-8vw4w" Jul 08 18:46:28.290982 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:28.290819 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eeae891b-9bad-4f61-915c-f0f4cac1bc18-cert\") pod \"odh-model-controller-7998c67dc4-8vw4w\" (UID: \"eeae891b-9bad-4f61-915c-f0f4cac1bc18\") " pod="opendatahub/odh-model-controller-7998c67dc4-8vw4w" Jul 08 18:46:28.293426 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:28.293405 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eeae891b-9bad-4f61-915c-f0f4cac1bc18-cert\") pod \"odh-model-controller-7998c67dc4-8vw4w\" (UID: \"eeae891b-9bad-4f61-915c-f0f4cac1bc18\") " pod="opendatahub/odh-model-controller-7998c67dc4-8vw4w" Jul 08 18:46:28.304482 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:28.304433 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtw45\" (UniqueName: \"kubernetes.io/projected/eeae891b-9bad-4f61-915c-f0f4cac1bc18-kube-api-access-qtw45\") pod \"odh-model-controller-7998c67dc4-8vw4w\" (UID: \"eeae891b-9bad-4f61-915c-f0f4cac1bc18\") " pod="opendatahub/odh-model-controller-7998c67dc4-8vw4w" Jul 08 18:46:28.309646 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:28.309623 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8vw4w" Jul 08 18:46:28.310645 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:28.310622 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8b75dd56d-wk2ph" event={"ID":"dc05552e-bd5b-46e6-95df-b70171c218f5","Type":"ContainerStarted","Data":"e66635d7537babc6c9e825f50fbf8f06f4e65724960ebcaeeb6acc67ff260d3f"} Jul 08 18:46:28.654144 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:28.654025 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8vw4w"] Jul 08 18:46:28.656312 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:46:28.656271 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeae891b_9bad_4f61_915c_f0f4cac1bc18.slice/crio-ad4a5e424b5b3a16132fd6a8dbcae66f44b518da66298e4293521554e0b1670a WatchSource:0}: Error finding container ad4a5e424b5b3a16132fd6a8dbcae66f44b518da66298e4293521554e0b1670a: Status 404 returned error can't find the container with id ad4a5e424b5b3a16132fd6a8dbcae66f44b518da66298e4293521554e0b1670a Jul 08 18:46:29.221262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.221228 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7"] Jul 08 18:46:29.224289 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.224273 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" Jul 08 18:46:29.227262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.227243 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 08 18:46:29.227411 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.227269 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 08 18:46:29.227411 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.227284 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 18:46:29.227411 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.227243 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 08 18:46:29.227562 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.227423 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gnwwh\"" Jul 08 18:46:29.239965 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.239942 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7"] Jul 08 18:46:29.317956 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.317916 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8vw4w" event={"ID":"eeae891b-9bad-4f61-915c-f0f4cac1bc18","Type":"ContainerStarted","Data":"ad4a5e424b5b3a16132fd6a8dbcae66f44b518da66298e4293521554e0b1670a"} Jul 08 18:46:29.401457 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.401422 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-cacerts\") pod \"istiod-openshift-gateway-7574459656-fkfd7\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" Jul 08 18:46:29.401647 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.401481 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-local-certs\") pod \"istiod-openshift-gateway-7574459656-fkfd7\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" Jul 08 18:46:29.401647 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.401521 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-fkfd7\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" Jul 08 18:46:29.401647 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.401567 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rnlg\" (UniqueName: \"kubernetes.io/projected/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-kube-api-access-4rnlg\") pod \"istiod-openshift-gateway-7574459656-fkfd7\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" Jul 08 18:46:29.401826 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.401676 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-fkfd7\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" Jul 08 18:46:29.401826 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.401706 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-istio-token\") pod \"istiod-openshift-gateway-7574459656-fkfd7\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" Jul 08 18:46:29.401826 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.401744 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-fkfd7\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" Jul 08 18:46:29.502756 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.502674 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-local-certs\") pod \"istiod-openshift-gateway-7574459656-fkfd7\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" Jul 08 18:46:29.502756 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.502737 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-fkfd7\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" Jul 08 18:46:29.502977 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.502789 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rnlg\" (UniqueName: \"kubernetes.io/projected/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-kube-api-access-4rnlg\") pod \"istiod-openshift-gateway-7574459656-fkfd7\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" Jul 08 18:46:29.502977 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.502837 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-fkfd7\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" Jul 08 18:46:29.502977 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.502868 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-istio-token\") pod \"istiod-openshift-gateway-7574459656-fkfd7\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" Jul 08 18:46:29.502977 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.502910 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-fkfd7\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" Jul 08 18:46:29.502977 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.502955 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-cacerts\") pod \"istiod-openshift-gateway-7574459656-fkfd7\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" Jul 08 18:46:29.503848 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.503776 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-fkfd7\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" Jul 08 18:46:29.505851 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.505824 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-local-certs\") pod \"istiod-openshift-gateway-7574459656-fkfd7\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" Jul 08 18:46:29.506112 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.506071 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-fkfd7\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" Jul 08 18:46:29.506211 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.506107 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-fkfd7\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" Jul 08 18:46:29.506773 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.506732 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-cacerts\") pod \"istiod-openshift-gateway-7574459656-fkfd7\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" Jul 08 18:46:29.519981 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.519948 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-istio-token\") pod \"istiod-openshift-gateway-7574459656-fkfd7\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" Jul 08 18:46:29.520105 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.520081 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rnlg\" (UniqueName: \"kubernetes.io/projected/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-kube-api-access-4rnlg\") pod \"istiod-openshift-gateway-7574459656-fkfd7\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" Jul 08 18:46:29.533940 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.533912 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" Jul 08 18:46:29.694956 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:29.694920 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7"] Jul 08 18:46:29.698802 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:46:29.698770 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a278aa5_0fc7_4e1e_9466_9f97ab0d63af.slice/crio-826b676669b8c02bfdeb9f2d40e36e76bffe3b122b9bbfca42202ceeb1826d6f WatchSource:0}: Error finding container 826b676669b8c02bfdeb9f2d40e36e76bffe3b122b9bbfca42202ceeb1826d6f: Status 404 returned error can't find the container with id 826b676669b8c02bfdeb9f2d40e36e76bffe3b122b9bbfca42202ceeb1826d6f Jul 08 18:46:30.324454 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:30.324417 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" event={"ID":"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af","Type":"ContainerStarted","Data":"826b676669b8c02bfdeb9f2d40e36e76bffe3b122b9bbfca42202ceeb1826d6f"} Jul 08 18:46:33.272373 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:33.272264 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 18:46:33.272747 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:33.272383 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 18:46:33.339331 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:33.339281 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8b75dd56d-wk2ph" event={"ID":"dc05552e-bd5b-46e6-95df-b70171c218f5","Type":"ContainerStarted","Data":"d5fe0bdf0a3911676fc3bcc11bc7b888f15db9ef5a5c3ee53d0787edc2701448"} Jul 08 18:46:33.341112 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:33.341086 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" event={"ID":"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af","Type":"ContainerStarted","Data":"ce3c210a0d2e562b5bedc17ebc09a4d6197d23843ba957302b922c672b06a72d"} Jul 08 18:46:33.341216 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:33.341148 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" Jul 08 18:46:33.342791 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:33.342768 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8vw4w" event={"ID":"eeae891b-9bad-4f61-915c-f0f4cac1bc18","Type":"ContainerStarted","Data":"e5cde856cc9f56144f6423234a3769b68f2549242d94e066ead84a46127ea1b2"} Jul 08 18:46:33.342923 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:33.342906 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-8vw4w" Jul 08 18:46:33.358269 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:33.358219 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-8b75dd56d-wk2ph" podStartSLOduration=1.193748113 podStartE2EDuration="6.35820243s" totalImagesPullingTime="5.164454317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:33.35736496 +0000 UTC m=+404.194692670" watchObservedRunningTime="2026-07-08 18:46:33.35820243 +0000 UTC m=+404.195530138" Jul 08 18:46:33.379178 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:33.379125 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-8vw4w" podStartSLOduration=1.813878353 podStartE2EDuration="6.379107203s" totalImagesPullingTime="4.56522885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:33.379082819 +0000 UTC m=+404.216410528" watchObservedRunningTime="2026-07-08 18:46:33.379107203 +0000 UTC m=+404.216434953" Jul 08 18:46:34.348230 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:34.348190 2575 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-fkfd7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 08 18:46:34.348614 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:34.348249 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" podUID="7a278aa5-0fc7-4e1e-9466-9f97ab0d63af" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 18:46:37.348342 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.348235 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" Jul 08 18:46:37.361262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.361217 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" podStartSLOduration=4.79083787 podStartE2EDuration="8.361201493s" totalImagesPullingTime="3.570363623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:33.410794875 +0000 UTC m=+404.248122582" watchObservedRunningTime="2026-07-08 18:46:37.361201493 +0000 UTC m=+408.198529201" Jul 08 18:46:37.361435 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.361422 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6"] Jul 08 18:46:37.364788 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.364769 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" Jul 08 18:46:37.393028 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.393005 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6"] Jul 08 18:46:37.463651 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.463621 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/362ce324-f216-489b-89c5-ddd7244f8fbd-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-w9rg6\" (UID: \"362ce324-f216-489b-89c5-ddd7244f8fbd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" Jul 08 18:46:37.463858 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.463672 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/362ce324-f216-489b-89c5-ddd7244f8fbd-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-w9rg6\" (UID: \"362ce324-f216-489b-89c5-ddd7244f8fbd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" Jul 08 18:46:37.463858 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.463739 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/362ce324-f216-489b-89c5-ddd7244f8fbd-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-w9rg6\" (UID: \"362ce324-f216-489b-89c5-ddd7244f8fbd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" Jul 08 18:46:37.463858 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.463765 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/362ce324-f216-489b-89c5-ddd7244f8fbd-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-w9rg6\" (UID: \"362ce324-f216-489b-89c5-ddd7244f8fbd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" Jul 08 18:46:37.463858 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.463814 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg5j5\" (UniqueName: \"kubernetes.io/projected/362ce324-f216-489b-89c5-ddd7244f8fbd-kube-api-access-sg5j5\") pod \"istiod-openshift-gateway-94bb8fbfd-w9rg6\" (UID: \"362ce324-f216-489b-89c5-ddd7244f8fbd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" Jul 08 18:46:37.463858 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.463838 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/362ce324-f216-489b-89c5-ddd7244f8fbd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-w9rg6\" (UID: \"362ce324-f216-489b-89c5-ddd7244f8fbd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" Jul 08 18:46:37.464040 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.463860 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/362ce324-f216-489b-89c5-ddd7244f8fbd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-w9rg6\" (UID: \"362ce324-f216-489b-89c5-ddd7244f8fbd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" Jul 08 18:46:37.564952 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.564913 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/362ce324-f216-489b-89c5-ddd7244f8fbd-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-w9rg6\" (UID: \"362ce324-f216-489b-89c5-ddd7244f8fbd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" Jul 08 18:46:37.565148 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.564977 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/362ce324-f216-489b-89c5-ddd7244f8fbd-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-w9rg6\" (UID: \"362ce324-f216-489b-89c5-ddd7244f8fbd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" Jul 08 18:46:37.565148 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.565000 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/362ce324-f216-489b-89c5-ddd7244f8fbd-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-w9rg6\" (UID: \"362ce324-f216-489b-89c5-ddd7244f8fbd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" Jul 08 18:46:37.565148 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.565027 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg5j5\" (UniqueName: \"kubernetes.io/projected/362ce324-f216-489b-89c5-ddd7244f8fbd-kube-api-access-sg5j5\") pod \"istiod-openshift-gateway-94bb8fbfd-w9rg6\" (UID: \"362ce324-f216-489b-89c5-ddd7244f8fbd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" Jul 08 18:46:37.565148 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.565054 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/362ce324-f216-489b-89c5-ddd7244f8fbd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-w9rg6\" (UID: \"362ce324-f216-489b-89c5-ddd7244f8fbd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" Jul 08 18:46:37.565148 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.565078 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/362ce324-f216-489b-89c5-ddd7244f8fbd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-w9rg6\" (UID: \"362ce324-f216-489b-89c5-ddd7244f8fbd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" Jul 08 18:46:37.565148 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.565130 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/362ce324-f216-489b-89c5-ddd7244f8fbd-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-w9rg6\" (UID: \"362ce324-f216-489b-89c5-ddd7244f8fbd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" Jul 08 18:46:37.565958 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.565925 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/362ce324-f216-489b-89c5-ddd7244f8fbd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-w9rg6\" (UID: \"362ce324-f216-489b-89c5-ddd7244f8fbd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" Jul 08 18:46:37.567426 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.567400 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/362ce324-f216-489b-89c5-ddd7244f8fbd-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-w9rg6\" (UID: \"362ce324-f216-489b-89c5-ddd7244f8fbd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" Jul 08 18:46:37.567539 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.567412 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/362ce324-f216-489b-89c5-ddd7244f8fbd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-w9rg6\" (UID: \"362ce324-f216-489b-89c5-ddd7244f8fbd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" Jul 08 18:46:37.567735 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.567715 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/362ce324-f216-489b-89c5-ddd7244f8fbd-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-w9rg6\" (UID: \"362ce324-f216-489b-89c5-ddd7244f8fbd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" Jul 08 18:46:37.567807 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.567784 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/362ce324-f216-489b-89c5-ddd7244f8fbd-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-w9rg6\" (UID: \"362ce324-f216-489b-89c5-ddd7244f8fbd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" Jul 08 18:46:37.576537 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.576507 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/362ce324-f216-489b-89c5-ddd7244f8fbd-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-w9rg6\" (UID: \"362ce324-f216-489b-89c5-ddd7244f8fbd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" Jul 08 18:46:37.576753 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.576731 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg5j5\" (UniqueName: \"kubernetes.io/projected/362ce324-f216-489b-89c5-ddd7244f8fbd-kube-api-access-sg5j5\") pod \"istiod-openshift-gateway-94bb8fbfd-w9rg6\" (UID: \"362ce324-f216-489b-89c5-ddd7244f8fbd\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" Jul 08 18:46:37.673450 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.673418 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" Jul 08 18:46:37.822964 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.822932 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6"] Jul 08 18:46:37.826076 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:46:37.826051 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362ce324_f216_489b_89c5_ddd7244f8fbd.slice/crio-7bbe085d199305879f3f17d1bc8be636f17ca551e21d2fd91d551e9441fcf071 WatchSource:0}: Error finding container 7bbe085d199305879f3f17d1bc8be636f17ca551e21d2fd91d551e9441fcf071: Status 404 returned error can't find the container with id 7bbe085d199305879f3f17d1bc8be636f17ca551e21d2fd91d551e9441fcf071 Jul 08 18:46:37.829009 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.828978 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 18:46:37.829079 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:37.829050 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 18:46:38.364789 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:38.364753 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" event={"ID":"362ce324-f216-489b-89c5-ddd7244f8fbd","Type":"ContainerStarted","Data":"b8ff50ea426deb6e728e68c268eee4d0cd299ee71e193a6f72cf9c5faf5ad400"} Jul 08 18:46:38.365187 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:38.364801 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" event={"ID":"362ce324-f216-489b-89c5-ddd7244f8fbd","Type":"ContainerStarted","Data":"7bbe085d199305879f3f17d1bc8be636f17ca551e21d2fd91d551e9441fcf071"} Jul 08 18:46:38.365187 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:38.364898 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" Jul 08 18:46:38.366569 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:38.366537 2575 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-w9rg6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 08 18:46:38.366668 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:38.366584 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" podUID="362ce324-f216-489b-89c5-ddd7244f8fbd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 18:46:38.399712 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:38.399666 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" podStartSLOduration=1.399652016 podStartE2EDuration="1.399652016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:46:38.397570194 +0000 UTC m=+409.234897925" watchObservedRunningTime="2026-07-08 18:46:38.399652016 +0000 UTC m=+409.236979724" Jul 08 18:46:39.370099 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:39.370070 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-w9rg6" Jul 08 18:46:39.518740 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:39.518712 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7"] Jul 08 18:46:39.518907 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:39.518885 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" podUID="7a278aa5-0fc7-4e1e-9466-9f97ab0d63af" containerName="discovery" containerID="cri-o://ce3c210a0d2e562b5bedc17ebc09a4d6197d23843ba957302b922c672b06a72d" gracePeriod=30 Jul 08 18:46:39.759633 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:39.759603 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" Jul 08 18:46:39.784366 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:39.784085 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-istio-token\") pod \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " Jul 08 18:46:39.784366 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:39.784147 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rnlg\" (UniqueName: \"kubernetes.io/projected/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-kube-api-access-4rnlg\") pod \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " Jul 08 18:46:39.784366 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:39.784214 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-istio-kubeconfig\") pod \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " Jul 08 18:46:39.784366 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:39.784245 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-local-certs\") pod \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " Jul 08 18:46:39.784366 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:39.784281 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-istio-csr-dns-cert\") pod \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " Jul 08 18:46:39.784366 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:39.784335 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-istio-csr-ca-configmap\") pod \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " Jul 08 18:46:39.784366 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:39.784370 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-cacerts\") pod \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\" (UID: \"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af\") " Jul 08 18:46:39.787232 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:39.787204 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "7a278aa5-0fc7-4e1e-9466-9f97ab0d63af" (UID: "7a278aa5-0fc7-4e1e-9466-9f97ab0d63af"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:46:39.788041 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:39.788009 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-cacerts" (OuterVolumeSpecName: "cacerts") pod "7a278aa5-0fc7-4e1e-9466-9f97ab0d63af" (UID: "7a278aa5-0fc7-4e1e-9466-9f97ab0d63af"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:46:39.788254 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:39.788207 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-local-certs" (OuterVolumeSpecName: "local-certs") pod "7a278aa5-0fc7-4e1e-9466-9f97ab0d63af" (UID: "7a278aa5-0fc7-4e1e-9466-9f97ab0d63af"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:46:39.788468 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:39.788444 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-istio-token" (OuterVolumeSpecName: "istio-token") pod "7a278aa5-0fc7-4e1e-9466-9f97ab0d63af" (UID: "7a278aa5-0fc7-4e1e-9466-9f97ab0d63af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:46:39.788568 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:39.788460 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-kube-api-access-4rnlg" (OuterVolumeSpecName: "kube-api-access-4rnlg") pod "7a278aa5-0fc7-4e1e-9466-9f97ab0d63af" (UID: "7a278aa5-0fc7-4e1e-9466-9f97ab0d63af"). InnerVolumeSpecName "kube-api-access-4rnlg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:46:39.789046 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:39.789018 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "7a278aa5-0fc7-4e1e-9466-9f97ab0d63af" (UID: "7a278aa5-0fc7-4e1e-9466-9f97ab0d63af"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:46:39.789418 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:39.789400 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "7a278aa5-0fc7-4e1e-9466-9f97ab0d63af" (UID: "7a278aa5-0fc7-4e1e-9466-9f97ab0d63af"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:46:39.885308 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:39.885269 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-istio-csr-ca-configmap\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 18:46:39.885471 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:39.885318 2575 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-cacerts\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 18:46:39.885471 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:39.885330 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-istio-token\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 18:46:39.885471 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:39.885339 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rnlg\" (UniqueName: \"kubernetes.io/projected/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-kube-api-access-4rnlg\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 18:46:39.885471 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:39.885347 2575 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-istio-kubeconfig\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 18:46:39.885471 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:39.885356 2575 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-local-certs\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 18:46:39.885471 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:39.885363 2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af-istio-csr-dns-cert\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 18:46:40.375575 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:40.375541 2575 generic.go:356] "Generic (PLEG): container finished" podID="7a278aa5-0fc7-4e1e-9466-9f97ab0d63af" containerID="ce3c210a0d2e562b5bedc17ebc09a4d6197d23843ba957302b922c672b06a72d" exitCode=0 Jul 08 18:46:40.376050 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:40.375616 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" event={"ID":"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af","Type":"ContainerDied","Data":"ce3c210a0d2e562b5bedc17ebc09a4d6197d23843ba957302b922c672b06a72d"} Jul 08 18:46:40.376050 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:40.375645 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" Jul 08 18:46:40.376050 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:40.375709 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7" event={"ID":"7a278aa5-0fc7-4e1e-9466-9f97ab0d63af","Type":"ContainerDied","Data":"826b676669b8c02bfdeb9f2d40e36e76bffe3b122b9bbfca42202ceeb1826d6f"} Jul 08 18:46:40.376050 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:40.375740 2575 scope.go:117] "RemoveContainer" containerID="ce3c210a0d2e562b5bedc17ebc09a4d6197d23843ba957302b922c672b06a72d" Jul 08 18:46:40.384846 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:40.384829 2575 scope.go:117] "RemoveContainer" containerID="ce3c210a0d2e562b5bedc17ebc09a4d6197d23843ba957302b922c672b06a72d" Jul 08 18:46:40.385084 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:46:40.385065 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce3c210a0d2e562b5bedc17ebc09a4d6197d23843ba957302b922c672b06a72d\": container with ID starting with ce3c210a0d2e562b5bedc17ebc09a4d6197d23843ba957302b922c672b06a72d not found: ID does not exist" containerID="ce3c210a0d2e562b5bedc17ebc09a4d6197d23843ba957302b922c672b06a72d" Jul 08 18:46:40.385141 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:40.385092 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce3c210a0d2e562b5bedc17ebc09a4d6197d23843ba957302b922c672b06a72d"} err="failed to get container status \"ce3c210a0d2e562b5bedc17ebc09a4d6197d23843ba957302b922c672b06a72d\": rpc error: code = NotFound desc = could not find container \"ce3c210a0d2e562b5bedc17ebc09a4d6197d23843ba957302b922c672b06a72d\": container with ID starting with ce3c210a0d2e562b5bedc17ebc09a4d6197d23843ba957302b922c672b06a72d not found: ID does not exist" Jul 08 18:46:40.418265 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:40.418221 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7"] Jul 08 18:46:40.432818 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:40.432798 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-fkfd7"] Jul 08 18:46:41.779258 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:41.779214 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a278aa5-0fc7-4e1e-9466-9f97ab0d63af" path="/var/lib/kubelet/pods/7a278aa5-0fc7-4e1e-9466-9f97ab0d63af/volumes" Jul 08 18:46:44.349314 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:46:44.349260 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-8vw4w" Jul 08 18:47:28.684149 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:28.684067 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5bdwh"] Jul 08 18:47:28.684620 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:28.684364 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a278aa5-0fc7-4e1e-9466-9f97ab0d63af" containerName="discovery" Jul 08 18:47:28.684620 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:28.684375 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a278aa5-0fc7-4e1e-9466-9f97ab0d63af" containerName="discovery" Jul 08 18:47:28.684620 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:28.684433 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a278aa5-0fc7-4e1e-9466-9f97ab0d63af" containerName="discovery" Jul 08 18:47:28.686348 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:28.686326 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5bdwh" Jul 08 18:47:28.689718 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:28.689679 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 08 18:47:28.689855 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:28.689835 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnz6g\" (UniqueName: \"kubernetes.io/projected/b288953b-357a-409d-9f63-8e71c6ac0352-kube-api-access-qnz6g\") pod \"kuadrant-operator-catalog-5bdwh\" (UID: \"b288953b-357a-409d-9f63-8e71c6ac0352\") " pod="kuadrant-system/kuadrant-operator-catalog-5bdwh" Jul 08 18:47:28.691386 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:28.691370 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-b8pdm\"" Jul 08 18:47:28.691488 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:28.691412 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 08 18:47:28.698351 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:28.698327 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5bdwh"] Jul 08 18:47:28.790680 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:28.790645 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnz6g\" (UniqueName: \"kubernetes.io/projected/b288953b-357a-409d-9f63-8e71c6ac0352-kube-api-access-qnz6g\") pod \"kuadrant-operator-catalog-5bdwh\" (UID: \"b288953b-357a-409d-9f63-8e71c6ac0352\") " pod="kuadrant-system/kuadrant-operator-catalog-5bdwh" Jul 08 18:47:28.802231 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:28.802193 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnz6g\" (UniqueName: \"kubernetes.io/projected/b288953b-357a-409d-9f63-8e71c6ac0352-kube-api-access-qnz6g\") pod \"kuadrant-operator-catalog-5bdwh\" (UID: \"b288953b-357a-409d-9f63-8e71c6ac0352\") " pod="kuadrant-system/kuadrant-operator-catalog-5bdwh" Jul 08 18:47:28.996836 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:28.996746 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5bdwh" Jul 08 18:47:29.031200 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:29.031170 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5bdwh"] Jul 08 18:47:29.131702 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:29.131670 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5bdwh"] Jul 08 18:47:29.139722 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:47:29.139696 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb288953b_357a_409d_9f63_8e71c6ac0352.slice/crio-87f7468cc588189800f609945650625dd090401fb471ad6320a8b9458a7d9d11 WatchSource:0}: Error finding container 87f7468cc588189800f609945650625dd090401fb471ad6320a8b9458a7d9d11: Status 404 returned error can't find the container with id 87f7468cc588189800f609945650625dd090401fb471ad6320a8b9458a7d9d11 Jul 08 18:47:29.241908 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:29.241874 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9b9mq"] Jul 08 18:47:29.244525 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:29.244500 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9b9mq" Jul 08 18:47:29.255242 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:29.255217 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9b9mq"] Jul 08 18:47:29.294937 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:29.294904 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf9ht\" (UniqueName: \"kubernetes.io/projected/2b0b7393-a76e-44cd-82aa-3a19ed4ce0bc-kube-api-access-pf9ht\") pod \"kuadrant-operator-catalog-9b9mq\" (UID: \"2b0b7393-a76e-44cd-82aa-3a19ed4ce0bc\") " pod="kuadrant-system/kuadrant-operator-catalog-9b9mq" Jul 08 18:47:29.396097 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:29.396064 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf9ht\" (UniqueName: \"kubernetes.io/projected/2b0b7393-a76e-44cd-82aa-3a19ed4ce0bc-kube-api-access-pf9ht\") pod \"kuadrant-operator-catalog-9b9mq\" (UID: \"2b0b7393-a76e-44cd-82aa-3a19ed4ce0bc\") " pod="kuadrant-system/kuadrant-operator-catalog-9b9mq" Jul 08 18:47:29.405696 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:29.405663 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf9ht\" (UniqueName: \"kubernetes.io/projected/2b0b7393-a76e-44cd-82aa-3a19ed4ce0bc-kube-api-access-pf9ht\") pod \"kuadrant-operator-catalog-9b9mq\" (UID: \"2b0b7393-a76e-44cd-82aa-3a19ed4ce0bc\") " pod="kuadrant-system/kuadrant-operator-catalog-9b9mq" Jul 08 18:47:29.553622 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:29.553536 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9b9mq" Jul 08 18:47:29.588424 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:29.588376 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5bdwh" event={"ID":"b288953b-357a-409d-9f63-8e71c6ac0352","Type":"ContainerStarted","Data":"87f7468cc588189800f609945650625dd090401fb471ad6320a8b9458a7d9d11"} Jul 08 18:47:29.685429 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:29.685399 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9b9mq"] Jul 08 18:47:29.687621 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:47:29.687588 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b0b7393_a76e_44cd_82aa_3a19ed4ce0bc.slice/crio-e1228a922ba802c5bf580fa6ecaafae3dbd826d0f070f8be1e2f619696692bed WatchSource:0}: Error finding container e1228a922ba802c5bf580fa6ecaafae3dbd826d0f070f8be1e2f619696692bed: Status 404 returned error can't find the container with id e1228a922ba802c5bf580fa6ecaafae3dbd826d0f070f8be1e2f619696692bed Jul 08 18:47:30.595597 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:30.595556 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9b9mq" event={"ID":"2b0b7393-a76e-44cd-82aa-3a19ed4ce0bc","Type":"ContainerStarted","Data":"e1228a922ba802c5bf580fa6ecaafae3dbd826d0f070f8be1e2f619696692bed"} Jul 08 18:47:31.603458 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:31.603417 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9b9mq" event={"ID":"2b0b7393-a76e-44cd-82aa-3a19ed4ce0bc","Type":"ContainerStarted","Data":"9adc3522d3f2c537941cd36f8576775291e459a88e84be19d3911d5c12cfcdb6"} Jul 08 18:47:31.605058 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:31.605034 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5bdwh" event={"ID":"b288953b-357a-409d-9f63-8e71c6ac0352","Type":"ContainerStarted","Data":"21815933adcd01c8d646ce81d9e391773d2d0bceff707a1b6743fa9b5bd12678"} Jul 08 18:47:31.605194 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:31.605123 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-5bdwh" podUID="b288953b-357a-409d-9f63-8e71c6ac0352" containerName="registry-server" containerID="cri-o://21815933adcd01c8d646ce81d9e391773d2d0bceff707a1b6743fa9b5bd12678" gracePeriod=2 Jul 08 18:47:31.622338 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:31.622256 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-9b9mq" podStartSLOduration=0.841263859 podStartE2EDuration="2.622240535s" totalImagesPullingTime="1.780976676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:47:31.620896912 +0000 UTC m=+462.458224619" watchObservedRunningTime="2026-07-08 18:47:31.622240535 +0000 UTC m=+462.459568243" Jul 08 18:47:31.638755 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:31.638702 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-5bdwh" podStartSLOduration=1.3125935420000001 podStartE2EDuration="3.638682484s" totalImagesPullingTime="2.326088942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:47:31.636921064 +0000 UTC m=+462.474248773" watchObservedRunningTime="2026-07-08 18:47:31.638682484 +0000 UTC m=+462.476010192" Jul 08 18:47:31.841200 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:31.841177 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-5bdwh_b288953b-357a-409d-9f63-8e71c6ac0352/registry-server/0.log" Jul 08 18:47:31.843649 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:31.843631 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5bdwh" Jul 08 18:47:31.919481 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:31.919451 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnz6g\" (UniqueName: \"kubernetes.io/projected/b288953b-357a-409d-9f63-8e71c6ac0352-kube-api-access-qnz6g\") pod \"b288953b-357a-409d-9f63-8e71c6ac0352\" (UID: \"b288953b-357a-409d-9f63-8e71c6ac0352\") " Jul 08 18:47:31.921558 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:31.921524 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b288953b-357a-409d-9f63-8e71c6ac0352-kube-api-access-qnz6g" (OuterVolumeSpecName: "kube-api-access-qnz6g") pod "b288953b-357a-409d-9f63-8e71c6ac0352" (UID: "b288953b-357a-409d-9f63-8e71c6ac0352"). InnerVolumeSpecName "kube-api-access-qnz6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:47:32.020134 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:32.020050 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnz6g\" (UniqueName: \"kubernetes.io/projected/b288953b-357a-409d-9f63-8e71c6ac0352-kube-api-access-qnz6g\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 18:47:32.610365 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:32.610338 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-5bdwh_b288953b-357a-409d-9f63-8e71c6ac0352/registry-server/0.log" Jul 08 18:47:32.610843 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:32.610376 2575 generic.go:356] "Generic (PLEG): container finished" podID="b288953b-357a-409d-9f63-8e71c6ac0352" containerID="21815933adcd01c8d646ce81d9e391773d2d0bceff707a1b6743fa9b5bd12678" exitCode=1 Jul 08 18:47:32.610843 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:32.610446 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5bdwh" Jul 08 18:47:32.610843 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:32.610468 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5bdwh" event={"ID":"b288953b-357a-409d-9f63-8e71c6ac0352","Type":"ContainerDied","Data":"21815933adcd01c8d646ce81d9e391773d2d0bceff707a1b6743fa9b5bd12678"} Jul 08 18:47:32.610843 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:32.610518 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5bdwh" event={"ID":"b288953b-357a-409d-9f63-8e71c6ac0352","Type":"ContainerDied","Data":"87f7468cc588189800f609945650625dd090401fb471ad6320a8b9458a7d9d11"} Jul 08 18:47:32.610843 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:32.610538 2575 scope.go:117] "RemoveContainer" containerID="21815933adcd01c8d646ce81d9e391773d2d0bceff707a1b6743fa9b5bd12678" Jul 08 18:47:32.619236 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:32.619221 2575 scope.go:117] "RemoveContainer" containerID="21815933adcd01c8d646ce81d9e391773d2d0bceff707a1b6743fa9b5bd12678" Jul 08 18:47:32.619482 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:47:32.619465 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21815933adcd01c8d646ce81d9e391773d2d0bceff707a1b6743fa9b5bd12678\": container with ID starting with 21815933adcd01c8d646ce81d9e391773d2d0bceff707a1b6743fa9b5bd12678 not found: ID does not exist" containerID="21815933adcd01c8d646ce81d9e391773d2d0bceff707a1b6743fa9b5bd12678" Jul 08 18:47:32.619543 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:32.619491 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21815933adcd01c8d646ce81d9e391773d2d0bceff707a1b6743fa9b5bd12678"} err="failed to get container status \"21815933adcd01c8d646ce81d9e391773d2d0bceff707a1b6743fa9b5bd12678\": rpc error: code = NotFound desc = could not find container \"21815933adcd01c8d646ce81d9e391773d2d0bceff707a1b6743fa9b5bd12678\": container with ID starting with 21815933adcd01c8d646ce81d9e391773d2d0bceff707a1b6743fa9b5bd12678 not found: ID does not exist" Jul 08 18:47:32.631681 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:32.631655 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5bdwh"] Jul 08 18:47:32.634812 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:32.634792 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5bdwh"] Jul 08 18:47:33.780015 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:33.779981 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b288953b-357a-409d-9f63-8e71c6ac0352" path="/var/lib/kubelet/pods/b288953b-357a-409d-9f63-8e71c6ac0352/volumes" Jul 08 18:47:39.553798 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:39.553761 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-9b9mq" Jul 08 18:47:39.554266 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:39.553838 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-9b9mq" Jul 08 18:47:39.576463 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:39.576430 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-9b9mq" Jul 08 18:47:39.662971 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:47:39.662943 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-9b9mq" Jul 08 18:48:01.604345 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:01.604288 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-2zjk4"] Jul 08 18:48:01.604848 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:01.604605 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b288953b-357a-409d-9f63-8e71c6ac0352" containerName="registry-server" Jul 08 18:48:01.604848 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:01.604616 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b288953b-357a-409d-9f63-8e71c6ac0352" containerName="registry-server" Jul 08 18:48:01.604848 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:01.604666 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b288953b-357a-409d-9f63-8e71c6ac0352" containerName="registry-server" Jul 08 18:48:01.608774 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:01.608752 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2zjk4" Jul 08 18:48:01.612257 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:01.612237 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-w4ksz\"" Jul 08 18:48:01.626619 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:01.626591 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-2zjk4"] Jul 08 18:48:01.764191 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:01.764160 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph5xv\" (UniqueName: \"kubernetes.io/projected/66b826e7-33cf-4a39-a0b9-d9615dd3f360-kube-api-access-ph5xv\") pod \"authorino-operator-75cc8d5cb7-2zjk4\" (UID: \"66b826e7-33cf-4a39-a0b9-d9615dd3f360\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-2zjk4" Jul 08 18:48:01.864682 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:01.864593 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph5xv\" (UniqueName: \"kubernetes.io/projected/66b826e7-33cf-4a39-a0b9-d9615dd3f360-kube-api-access-ph5xv\") pod \"authorino-operator-75cc8d5cb7-2zjk4\" (UID: \"66b826e7-33cf-4a39-a0b9-d9615dd3f360\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-2zjk4" Jul 08 18:48:01.874562 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:01.874541 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph5xv\" (UniqueName: \"kubernetes.io/projected/66b826e7-33cf-4a39-a0b9-d9615dd3f360-kube-api-access-ph5xv\") pod \"authorino-operator-75cc8d5cb7-2zjk4\" (UID: \"66b826e7-33cf-4a39-a0b9-d9615dd3f360\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-2zjk4" Jul 08 18:48:01.918474 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:01.918441 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2zjk4" Jul 08 18:48:02.056884 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:02.056863 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-2zjk4"] Jul 08 18:48:02.059229 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:48:02.059197 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b826e7_33cf_4a39_a0b9_d9615dd3f360.slice/crio-901d4fb2a951971cfeb18dae531cec1cb53ed0fe32c0fdbeda22bab812ca6bda WatchSource:0}: Error finding container 901d4fb2a951971cfeb18dae531cec1cb53ed0fe32c0fdbeda22bab812ca6bda: Status 404 returned error can't find the container with id 901d4fb2a951971cfeb18dae531cec1cb53ed0fe32c0fdbeda22bab812ca6bda Jul 08 18:48:02.745175 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:02.745138 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2zjk4" event={"ID":"66b826e7-33cf-4a39-a0b9-d9615dd3f360","Type":"ContainerStarted","Data":"901d4fb2a951971cfeb18dae531cec1cb53ed0fe32c0fdbeda22bab812ca6bda"} Jul 08 18:48:04.851192 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:04.851151 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-f4ps5"] Jul 08 18:48:04.854685 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:04.854664 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-f4ps5" Jul 08 18:48:04.857975 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:04.857949 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 08 18:48:04.858095 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:04.858034 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zljlk\"" Jul 08 18:48:04.864262 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:04.864238 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-f4ps5"] Jul 08 18:48:04.992845 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:04.992806 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8tm9\" (UniqueName: \"kubernetes.io/projected/5238549b-03a9-48be-844a-9c9e0fc9d722-kube-api-access-m8tm9\") pod \"dns-operator-controller-manager-864dd78c4c-f4ps5\" (UID: \"5238549b-03a9-48be-844a-9c9e0fc9d722\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-f4ps5" Jul 08 18:48:05.094339 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:05.094281 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8tm9\" (UniqueName: \"kubernetes.io/projected/5238549b-03a9-48be-844a-9c9e0fc9d722-kube-api-access-m8tm9\") pod \"dns-operator-controller-manager-864dd78c4c-f4ps5\" (UID: \"5238549b-03a9-48be-844a-9c9e0fc9d722\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-f4ps5" Jul 08 18:48:05.108659 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:05.108629 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8tm9\" (UniqueName: \"kubernetes.io/projected/5238549b-03a9-48be-844a-9c9e0fc9d722-kube-api-access-m8tm9\") pod \"dns-operator-controller-manager-864dd78c4c-f4ps5\" (UID: \"5238549b-03a9-48be-844a-9c9e0fc9d722\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-f4ps5" Jul 08 18:48:05.167589 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:05.167549 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-f4ps5" Jul 08 18:48:05.317766 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:05.317701 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-f4ps5"] Jul 08 18:48:05.319895 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:48:05.319867 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5238549b_03a9_48be_844a_9c9e0fc9d722.slice/crio-0fa60e37fd1b86a1983740d5554bd03205e1b65a886dcb1584b79e4609d171b1 WatchSource:0}: Error finding container 0fa60e37fd1b86a1983740d5554bd03205e1b65a886dcb1584b79e4609d171b1: Status 404 returned error can't find the container with id 0fa60e37fd1b86a1983740d5554bd03205e1b65a886dcb1584b79e4609d171b1 Jul 08 18:48:05.760710 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:05.760669 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-f4ps5" event={"ID":"5238549b-03a9-48be-844a-9c9e0fc9d722","Type":"ContainerStarted","Data":"0fa60e37fd1b86a1983740d5554bd03205e1b65a886dcb1584b79e4609d171b1"} Jul 08 18:48:05.762111 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:05.762083 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2zjk4" event={"ID":"66b826e7-33cf-4a39-a0b9-d9615dd3f360","Type":"ContainerStarted","Data":"542699b67f6d26988c8630310aa25c54d4606aeb3e31d171f022e28cedf26c39"} Jul 08 18:48:05.762227 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:05.762211 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2zjk4" Jul 08 18:48:05.779161 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:05.779102 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2zjk4" podStartSLOduration=1.604400144 podStartE2EDuration="4.779081713s" totalImagesPullingTime="3.174681569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:48:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:48:05.778654109 +0000 UTC m=+496.615981819" watchObservedRunningTime="2026-07-08 18:48:05.779081713 +0000 UTC m=+496.616409420" Jul 08 18:48:08.267693 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:08.267654 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mx8dh"] Jul 08 18:48:08.270121 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:08.270101 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mx8dh" Jul 08 18:48:08.273309 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:08.273272 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-8fwgv\"" Jul 08 18:48:08.281731 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:08.281696 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mx8dh"] Jul 08 18:48:08.421895 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:08.421873 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x64h9\" (UniqueName: \"kubernetes.io/projected/f60bfd06-38a2-4300-9f2d-650da5666eff-kube-api-access-x64h9\") pod \"limitador-operator-controller-manager-859d7b78bd-mx8dh\" (UID: \"f60bfd06-38a2-4300-9f2d-650da5666eff\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mx8dh" Jul 08 18:48:08.522375 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:08.522272 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x64h9\" (UniqueName: \"kubernetes.io/projected/f60bfd06-38a2-4300-9f2d-650da5666eff-kube-api-access-x64h9\") pod \"limitador-operator-controller-manager-859d7b78bd-mx8dh\" (UID: \"f60bfd06-38a2-4300-9f2d-650da5666eff\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mx8dh" Jul 08 18:48:08.532314 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:08.532278 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x64h9\" (UniqueName: \"kubernetes.io/projected/f60bfd06-38a2-4300-9f2d-650da5666eff-kube-api-access-x64h9\") pod \"limitador-operator-controller-manager-859d7b78bd-mx8dh\" (UID: \"f60bfd06-38a2-4300-9f2d-650da5666eff\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mx8dh" Jul 08 18:48:08.585437 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:08.585405 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mx8dh" Jul 08 18:48:08.731040 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:08.731006 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mx8dh"] Jul 08 18:48:08.759388 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:48:08.759351 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60bfd06_38a2_4300_9f2d_650da5666eff.slice/crio-927ec87b42eca577ba3c0daedf78caf537519a79f1654298940ef2848025d0ae WatchSource:0}: Error finding container 927ec87b42eca577ba3c0daedf78caf537519a79f1654298940ef2848025d0ae: Status 404 returned error can't find the container with id 927ec87b42eca577ba3c0daedf78caf537519a79f1654298940ef2848025d0ae Jul 08 18:48:08.775667 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:08.775582 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-f4ps5" event={"ID":"5238549b-03a9-48be-844a-9c9e0fc9d722","Type":"ContainerStarted","Data":"dc5914cd361ca68dd1ac58ce2fa79188bbe0f86569e0230c0404d2ae5e7bac81"} Jul 08 18:48:08.775820 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:08.775677 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-f4ps5" Jul 08 18:48:08.776826 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:08.776801 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mx8dh" event={"ID":"f60bfd06-38a2-4300-9f2d-650da5666eff","Type":"ContainerStarted","Data":"927ec87b42eca577ba3c0daedf78caf537519a79f1654298940ef2848025d0ae"} Jul 08 18:48:08.801263 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:08.801206 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-f4ps5" podStartSLOduration=1.767616564 podStartE2EDuration="4.80119152s" totalImagesPullingTime="3.033574956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:48:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:48:08.800014932 +0000 UTC m=+499.637342641" watchObservedRunningTime="2026-07-08 18:48:08.80119152 +0000 UTC m=+499.638519228" Jul 08 18:48:10.790423 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:10.790327 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mx8dh" event={"ID":"f60bfd06-38a2-4300-9f2d-650da5666eff","Type":"ContainerStarted","Data":"1facafe1015cc5c0249a3e8694ebd5f4a9324a5b7b9ddbc7236926064e294a14"} Jul 08 18:48:10.790806 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:10.790458 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mx8dh" Jul 08 18:48:10.812649 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:10.812600 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mx8dh" podStartSLOduration=1.149723308 podStartE2EDuration="2.812584224s" totalImagesPullingTime="1.662860916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:48:10.810500809 +0000 UTC m=+501.647828524" watchObservedRunningTime="2026-07-08 18:48:10.812584224 +0000 UTC m=+501.649911932" Jul 08 18:48:16.768452 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:16.768419 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-2zjk4" Jul 08 18:48:19.786851 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:19.786820 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-f4ps5" Jul 08 18:48:21.797656 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:21.797626 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mx8dh" Jul 08 18:48:33.335751 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:33.335718 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sxg5c"] Jul 08 18:48:33.338927 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:33.338905 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sxg5c" Jul 08 18:48:33.341614 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:33.341584 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qw22r\"" Jul 08 18:48:33.351795 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:33.351769 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sxg5c"] Jul 08 18:48:33.416755 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:33.416710 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f7b64e2-0a20-463f-be66-d5ad7b8da3c7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-sxg5c\" (UID: \"1f7b64e2-0a20-463f-be66-d5ad7b8da3c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sxg5c" Jul 08 18:48:33.416939 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:33.416832 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm8s4\" (UniqueName: \"kubernetes.io/projected/1f7b64e2-0a20-463f-be66-d5ad7b8da3c7-kube-api-access-zm8s4\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-sxg5c\" (UID: \"1f7b64e2-0a20-463f-be66-d5ad7b8da3c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sxg5c" Jul 08 18:48:33.518095 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:33.518049 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm8s4\" (UniqueName: \"kubernetes.io/projected/1f7b64e2-0a20-463f-be66-d5ad7b8da3c7-kube-api-access-zm8s4\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-sxg5c\" (UID: \"1f7b64e2-0a20-463f-be66-d5ad7b8da3c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sxg5c" Jul 08 18:48:33.518279 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:33.518116 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f7b64e2-0a20-463f-be66-d5ad7b8da3c7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-sxg5c\" (UID: \"1f7b64e2-0a20-463f-be66-d5ad7b8da3c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sxg5c" Jul 08 18:48:33.518544 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:33.518521 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f7b64e2-0a20-463f-be66-d5ad7b8da3c7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-sxg5c\" (UID: \"1f7b64e2-0a20-463f-be66-d5ad7b8da3c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sxg5c" Jul 08 18:48:33.527394 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:33.527374 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm8s4\" (UniqueName: \"kubernetes.io/projected/1f7b64e2-0a20-463f-be66-d5ad7b8da3c7-kube-api-access-zm8s4\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-sxg5c\" (UID: \"1f7b64e2-0a20-463f-be66-d5ad7b8da3c7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sxg5c" Jul 08 18:48:33.649339 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:33.649281 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sxg5c" Jul 08 18:48:33.789542 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:33.789520 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sxg5c"] Jul 08 18:48:33.791962 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:48:33.791930 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f7b64e2_0a20_463f_be66_d5ad7b8da3c7.slice/crio-924e7f5e8f722eb9bcfac2f143ad552c8a97fc6b2091bcb649d5dec889d659b1 WatchSource:0}: Error finding container 924e7f5e8f722eb9bcfac2f143ad552c8a97fc6b2091bcb649d5dec889d659b1: Status 404 returned error can't find the container with id 924e7f5e8f722eb9bcfac2f143ad552c8a97fc6b2091bcb649d5dec889d659b1 Jul 08 18:48:33.892440 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:33.892404 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sxg5c" event={"ID":"1f7b64e2-0a20-463f-be66-d5ad7b8da3c7","Type":"ContainerStarted","Data":"924e7f5e8f722eb9bcfac2f143ad552c8a97fc6b2091bcb649d5dec889d659b1"} Jul 08 18:48:34.116722 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.116692 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sxg5c"] Jul 08 18:48:34.123121 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.123078 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sxg5c"] Jul 08 18:48:34.144286 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.144258 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hcs7p"] Jul 08 18:48:34.148766 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.148746 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hcs7p" Jul 08 18:48:34.168393 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.168366 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mx8dh"] Jul 08 18:48:34.168553 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.168533 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mx8dh" podUID="f60bfd06-38a2-4300-9f2d-650da5666eff" containerName="manager" containerID="cri-o://1facafe1015cc5c0249a3e8694ebd5f4a9324a5b7b9ddbc7236926064e294a14" gracePeriod=2 Jul 08 18:48:34.180086 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.180054 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mx8dh"] Jul 08 18:48:34.182568 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.182466 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hcs7p"] Jul 08 18:48:34.215952 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:48:34.215904 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-mx8dh\" is forbidden: User \"system:node:ip-10-0-131-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-102.ec2.internal' and this object" podUID="f60bfd06-38a2-4300-9f2d-650da5666eff" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mx8dh" Jul 08 18:48:34.222601 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.222571 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9kdf\" (UniqueName: \"kubernetes.io/projected/3fe6ea9d-0390-4585-b10d-894de55d5405-kube-api-access-m9kdf\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hcs7p\" (UID: \"3fe6ea9d-0390-4585-b10d-894de55d5405\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hcs7p" Jul 08 18:48:34.222715 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.222674 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fe6ea9d-0390-4585-b10d-894de55d5405-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hcs7p\" (UID: \"3fe6ea9d-0390-4585-b10d-894de55d5405\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hcs7p" Jul 08 18:48:34.236953 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.236927 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m8gzx"] Jul 08 18:48:34.237290 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.237274 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f60bfd06-38a2-4300-9f2d-650da5666eff" containerName="manager" Jul 08 18:48:34.237290 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.237290 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f60bfd06-38a2-4300-9f2d-650da5666eff" containerName="manager" Jul 08 18:48:34.237431 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.237381 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f60bfd06-38a2-4300-9f2d-650da5666eff" containerName="manager" Jul 08 18:48:34.240322 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.240306 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m8gzx" Jul 08 18:48:34.262709 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.262685 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m8gzx"] Jul 08 18:48:34.281455 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:48:34.281427 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-mx8dh\" is forbidden: User \"system:node:ip-10-0-131-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-102.ec2.internal' and this object" podUID="f60bfd06-38a2-4300-9f2d-650da5666eff" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mx8dh" Jul 08 18:48:34.323106 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.323066 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fe6ea9d-0390-4585-b10d-894de55d5405-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hcs7p\" (UID: \"3fe6ea9d-0390-4585-b10d-894de55d5405\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hcs7p" Jul 08 18:48:34.323266 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.323132 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6g55\" (UniqueName: \"kubernetes.io/projected/ec105bbe-ad50-426a-bf2d-c73942258586-kube-api-access-s6g55\") pod \"limitador-operator-controller-manager-859d7b78bd-m8gzx\" (UID: \"ec105bbe-ad50-426a-bf2d-c73942258586\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m8gzx" Jul 08 18:48:34.323266 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.323171 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9kdf\" (UniqueName: \"kubernetes.io/projected/3fe6ea9d-0390-4585-b10d-894de55d5405-kube-api-access-m9kdf\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hcs7p\" (UID: \"3fe6ea9d-0390-4585-b10d-894de55d5405\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hcs7p" Jul 08 18:48:34.323456 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.323440 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fe6ea9d-0390-4585-b10d-894de55d5405-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hcs7p\" (UID: \"3fe6ea9d-0390-4585-b10d-894de55d5405\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hcs7p" Jul 08 18:48:34.340818 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.340788 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9kdf\" (UniqueName: \"kubernetes.io/projected/3fe6ea9d-0390-4585-b10d-894de55d5405-kube-api-access-m9kdf\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-hcs7p\" (UID: \"3fe6ea9d-0390-4585-b10d-894de55d5405\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hcs7p" Jul 08 18:48:34.392448 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.392427 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mx8dh" Jul 08 18:48:34.395168 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:48:34.395141 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-mx8dh\" is forbidden: User \"system:node:ip-10-0-131-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-102.ec2.internal' and this object" podUID="f60bfd06-38a2-4300-9f2d-650da5666eff" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mx8dh" Jul 08 18:48:34.425134 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.424953 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6g55\" (UniqueName: \"kubernetes.io/projected/ec105bbe-ad50-426a-bf2d-c73942258586-kube-api-access-s6g55\") pod \"limitador-operator-controller-manager-859d7b78bd-m8gzx\" (UID: \"ec105bbe-ad50-426a-bf2d-c73942258586\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m8gzx" Jul 08 18:48:34.443396 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.443364 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6g55\" (UniqueName: \"kubernetes.io/projected/ec105bbe-ad50-426a-bf2d-c73942258586-kube-api-access-s6g55\") pod \"limitador-operator-controller-manager-859d7b78bd-m8gzx\" (UID: \"ec105bbe-ad50-426a-bf2d-c73942258586\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m8gzx" Jul 08 18:48:34.459006 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.458979 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hcs7p" Jul 08 18:48:34.525543 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.525506 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x64h9\" (UniqueName: \"kubernetes.io/projected/f60bfd06-38a2-4300-9f2d-650da5666eff-kube-api-access-x64h9\") pod \"f60bfd06-38a2-4300-9f2d-650da5666eff\" (UID: \"f60bfd06-38a2-4300-9f2d-650da5666eff\") " Jul 08 18:48:34.529981 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.529943 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f60bfd06-38a2-4300-9f2d-650da5666eff-kube-api-access-x64h9" (OuterVolumeSpecName: "kube-api-access-x64h9") pod "f60bfd06-38a2-4300-9f2d-650da5666eff" (UID: "f60bfd06-38a2-4300-9f2d-650da5666eff"). InnerVolumeSpecName "kube-api-access-x64h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:48:34.558604 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.558562 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m8gzx" Jul 08 18:48:34.621676 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.621612 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hcs7p"] Jul 08 18:48:34.625990 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:48:34.625958 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe6ea9d_0390_4585_b10d_894de55d5405.slice/crio-22e7da6c8d1ffaab8646ae4e870d9cfa5262b71f6fd54bb4f96e2f25bd364a4d WatchSource:0}: Error finding container 22e7da6c8d1ffaab8646ae4e870d9cfa5262b71f6fd54bb4f96e2f25bd364a4d: Status 404 returned error can't find the container with id 22e7da6c8d1ffaab8646ae4e870d9cfa5262b71f6fd54bb4f96e2f25bd364a4d Jul 08 18:48:34.626418 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.626386 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x64h9\" (UniqueName: \"kubernetes.io/projected/f60bfd06-38a2-4300-9f2d-650da5666eff-kube-api-access-x64h9\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 18:48:34.706267 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.706235 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m8gzx"] Jul 08 18:48:34.710398 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:48:34.710363 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec105bbe_ad50_426a_bf2d_c73942258586.slice/crio-d058b2356e38b1f9164068fef3e25a49b7c3697ef55a8d7ef158887e291b508b WatchSource:0}: Error finding container d058b2356e38b1f9164068fef3e25a49b7c3697ef55a8d7ef158887e291b508b: Status 404 returned error can't find the container with id d058b2356e38b1f9164068fef3e25a49b7c3697ef55a8d7ef158887e291b508b Jul 08 18:48:34.902699 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.902600 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hcs7p" event={"ID":"3fe6ea9d-0390-4585-b10d-894de55d5405","Type":"ContainerStarted","Data":"22e7da6c8d1ffaab8646ae4e870d9cfa5262b71f6fd54bb4f96e2f25bd364a4d"} Jul 08 18:48:34.904101 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.904062 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m8gzx" event={"ID":"ec105bbe-ad50-426a-bf2d-c73942258586","Type":"ContainerStarted","Data":"c0ed94aa31536f22144f802abbb7cebe72460d98ad684c62f0991bf3c2c51a9f"} Jul 08 18:48:34.904101 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.904104 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m8gzx" event={"ID":"ec105bbe-ad50-426a-bf2d-c73942258586","Type":"ContainerStarted","Data":"d058b2356e38b1f9164068fef3e25a49b7c3697ef55a8d7ef158887e291b508b"} Jul 08 18:48:34.904274 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.904251 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m8gzx" Jul 08 18:48:34.905347 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.905319 2575 generic.go:356] "Generic (PLEG): container finished" podID="f60bfd06-38a2-4300-9f2d-650da5666eff" containerID="1facafe1015cc5c0249a3e8694ebd5f4a9324a5b7b9ddbc7236926064e294a14" exitCode=0 Jul 08 18:48:34.905447 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.905349 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mx8dh" Jul 08 18:48:34.905447 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.905403 2575 scope.go:117] "RemoveContainer" containerID="1facafe1015cc5c0249a3e8694ebd5f4a9324a5b7b9ddbc7236926064e294a14" Jul 08 18:48:34.915186 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.915169 2575 scope.go:117] "RemoveContainer" containerID="1facafe1015cc5c0249a3e8694ebd5f4a9324a5b7b9ddbc7236926064e294a14" Jul 08 18:48:34.915480 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:48:34.915454 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1facafe1015cc5c0249a3e8694ebd5f4a9324a5b7b9ddbc7236926064e294a14\": container with ID starting with 1facafe1015cc5c0249a3e8694ebd5f4a9324a5b7b9ddbc7236926064e294a14 not found: ID does not exist" containerID="1facafe1015cc5c0249a3e8694ebd5f4a9324a5b7b9ddbc7236926064e294a14" Jul 08 18:48:34.915535 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.915482 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1facafe1015cc5c0249a3e8694ebd5f4a9324a5b7b9ddbc7236926064e294a14"} err="failed to get container status \"1facafe1015cc5c0249a3e8694ebd5f4a9324a5b7b9ddbc7236926064e294a14\": rpc error: code = NotFound desc = could not find container \"1facafe1015cc5c0249a3e8694ebd5f4a9324a5b7b9ddbc7236926064e294a14\": container with ID starting with 1facafe1015cc5c0249a3e8694ebd5f4a9324a5b7b9ddbc7236926064e294a14 not found: ID does not exist" Jul 08 18:48:34.924250 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:34.924209 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m8gzx" podStartSLOduration=0.924194232 podStartE2EDuration="924.194232ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:48:34.923074511 +0000 UTC m=+525.760402220" watchObservedRunningTime="2026-07-08 18:48:34.924194232 +0000 UTC m=+525.761521996" Jul 08 18:48:34.925280 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:48:34.925255 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-mx8dh\" is forbidden: User \"system:node:ip-10-0-131-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-102.ec2.internal' and this object" podUID="f60bfd06-38a2-4300-9f2d-650da5666eff" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mx8dh" Jul 08 18:48:34.928718 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:48:34.928688 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-mx8dh\" is forbidden: User \"system:node:ip-10-0-131-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-102.ec2.internal' and this object" podUID="f60bfd06-38a2-4300-9f2d-650da5666eff" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-mx8dh" Jul 08 18:48:35.779966 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:35.779927 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f60bfd06-38a2-4300-9f2d-650da5666eff" path="/var/lib/kubelet/pods/f60bfd06-38a2-4300-9f2d-650da5666eff/volumes" Jul 08 18:48:39.937539 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:39.937276 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sxg5c" podUID="1f7b64e2-0a20-463f-be66-d5ad7b8da3c7" containerName="manager" containerID="cri-o://39153364884ee86a6d31047788ab6ee7fc6a4b465989f33b0688a355895a7793" gracePeriod=2 Jul 08 18:48:39.939285 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:39.939255 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hcs7p" event={"ID":"3fe6ea9d-0390-4585-b10d-894de55d5405","Type":"ContainerStarted","Data":"2b8d45337b7a6526f40fb0e1b166cdf0cb4784b38be0c8fac0a6ad35254d6165"} Jul 08 18:48:39.939442 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:39.939417 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hcs7p" Jul 08 18:48:39.939847 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:48:39.939821 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-sxg5c\" is forbidden: User \"system:node:ip-10-0-131-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-102.ec2.internal' and this object" podUID="1f7b64e2-0a20-463f-be66-d5ad7b8da3c7" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sxg5c" Jul 08 18:48:39.963221 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:39.963168 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hcs7p" podStartSLOduration=1.6562430479999999 podStartE2EDuration="5.963148643s" totalImagesPullingTime="4.306905595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:48:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:48:39.961735681 +0000 UTC m=+530.799063413" watchObservedRunningTime="2026-07-08 18:48:39.963148643 +0000 UTC m=+530.800476415" Jul 08 18:48:39.963925 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:48:39.963896 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-sxg5c\" is forbidden: User \"system:node:ip-10-0-131-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-102.ec2.internal' and this object" podUID="1f7b64e2-0a20-463f-be66-d5ad7b8da3c7" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sxg5c" Jul 08 18:48:40.184972 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:40.184945 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sxg5c" Jul 08 18:48:40.187389 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:48:40.187367 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-sxg5c\" is forbidden: User \"system:node:ip-10-0-131-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-102.ec2.internal' and this object" podUID="1f7b64e2-0a20-463f-be66-d5ad7b8da3c7" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sxg5c" Jul 08 18:48:40.269288 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:40.269218 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f7b64e2-0a20-463f-be66-d5ad7b8da3c7-extensions-socket-volume\") pod \"1f7b64e2-0a20-463f-be66-d5ad7b8da3c7\" (UID: \"1f7b64e2-0a20-463f-be66-d5ad7b8da3c7\") " Jul 08 18:48:40.269288 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:40.269252 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm8s4\" (UniqueName: \"kubernetes.io/projected/1f7b64e2-0a20-463f-be66-d5ad7b8da3c7-kube-api-access-zm8s4\") pod \"1f7b64e2-0a20-463f-be66-d5ad7b8da3c7\" (UID: \"1f7b64e2-0a20-463f-be66-d5ad7b8da3c7\") " Jul 08 18:48:40.269553 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:40.269526 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f7b64e2-0a20-463f-be66-d5ad7b8da3c7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1f7b64e2-0a20-463f-be66-d5ad7b8da3c7" (UID: "1f7b64e2-0a20-463f-be66-d5ad7b8da3c7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:48:40.271349 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:40.271317 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f7b64e2-0a20-463f-be66-d5ad7b8da3c7-kube-api-access-zm8s4" (OuterVolumeSpecName: "kube-api-access-zm8s4") pod "1f7b64e2-0a20-463f-be66-d5ad7b8da3c7" (UID: "1f7b64e2-0a20-463f-be66-d5ad7b8da3c7"). InnerVolumeSpecName "kube-api-access-zm8s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:48:40.370326 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:40.370281 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f7b64e2-0a20-463f-be66-d5ad7b8da3c7-extensions-socket-volume\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 18:48:40.370326 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:40.370320 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm8s4\" (UniqueName: \"kubernetes.io/projected/1f7b64e2-0a20-463f-be66-d5ad7b8da3c7-kube-api-access-zm8s4\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 18:48:40.945811 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:40.945778 2575 generic.go:356] "Generic (PLEG): container finished" podID="1f7b64e2-0a20-463f-be66-d5ad7b8da3c7" containerID="39153364884ee86a6d31047788ab6ee7fc6a4b465989f33b0688a355895a7793" exitCode=2 Jul 08 18:48:40.946289 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:40.945820 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sxg5c" Jul 08 18:48:40.946289 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:40.945873 2575 scope.go:117] "RemoveContainer" containerID="39153364884ee86a6d31047788ab6ee7fc6a4b465989f33b0688a355895a7793" Jul 08 18:48:40.948440 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:48:40.948405 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-sxg5c\" is forbidden: User \"system:node:ip-10-0-131-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-102.ec2.internal' and this object" podUID="1f7b64e2-0a20-463f-be66-d5ad7b8da3c7" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sxg5c" Jul 08 18:48:40.955260 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:40.955240 2575 scope.go:117] "RemoveContainer" containerID="39153364884ee86a6d31047788ab6ee7fc6a4b465989f33b0688a355895a7793" Jul 08 18:48:40.955524 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:48:40.955503 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39153364884ee86a6d31047788ab6ee7fc6a4b465989f33b0688a355895a7793\": container with ID starting with 39153364884ee86a6d31047788ab6ee7fc6a4b465989f33b0688a355895a7793 not found: ID does not exist" containerID="39153364884ee86a6d31047788ab6ee7fc6a4b465989f33b0688a355895a7793" Jul 08 18:48:40.955597 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:40.955535 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39153364884ee86a6d31047788ab6ee7fc6a4b465989f33b0688a355895a7793"} err="failed to get container status \"39153364884ee86a6d31047788ab6ee7fc6a4b465989f33b0688a355895a7793\": rpc error: code = NotFound desc = could not find container \"39153364884ee86a6d31047788ab6ee7fc6a4b465989f33b0688a355895a7793\": container with ID starting with 39153364884ee86a6d31047788ab6ee7fc6a4b465989f33b0688a355895a7793 not found: ID does not exist" Jul 08 18:48:40.956939 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:48:40.956906 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-sxg5c\" is forbidden: User \"system:node:ip-10-0-131-102.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-102.ec2.internal' and this object" podUID="1f7b64e2-0a20-463f-be66-d5ad7b8da3c7" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-sxg5c" Jul 08 18:48:41.780374 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:41.780332 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f7b64e2-0a20-463f-be66-d5ad7b8da3c7" path="/var/lib/kubelet/pods/1f7b64e2-0a20-463f-be66-d5ad7b8da3c7/volumes" Jul 08 18:48:45.915601 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:45.915572 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-m8gzx" Jul 08 18:48:50.947685 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:48:50.947658 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hcs7p" Jul 08 18:49:03.673452 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:03.673417 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hcs7p"] Jul 08 18:49:03.673871 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:03.673616 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hcs7p" podUID="3fe6ea9d-0390-4585-b10d-894de55d5405" containerName="manager" containerID="cri-o://2b8d45337b7a6526f40fb0e1b166cdf0cb4784b38be0c8fac0a6ad35254d6165" gracePeriod=10 Jul 08 18:49:03.924488 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:03.924426 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hcs7p" Jul 08 18:49:04.044025 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:04.043995 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9kdf\" (UniqueName: \"kubernetes.io/projected/3fe6ea9d-0390-4585-b10d-894de55d5405-kube-api-access-m9kdf\") pod \"3fe6ea9d-0390-4585-b10d-894de55d5405\" (UID: \"3fe6ea9d-0390-4585-b10d-894de55d5405\") " Jul 08 18:49:04.044223 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:04.044040 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fe6ea9d-0390-4585-b10d-894de55d5405-extensions-socket-volume\") pod \"3fe6ea9d-0390-4585-b10d-894de55d5405\" (UID: \"3fe6ea9d-0390-4585-b10d-894de55d5405\") " Jul 08 18:49:04.044495 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:04.044466 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe6ea9d-0390-4585-b10d-894de55d5405-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3fe6ea9d-0390-4585-b10d-894de55d5405" (UID: "3fe6ea9d-0390-4585-b10d-894de55d5405"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:49:04.046197 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:04.046167 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe6ea9d-0390-4585-b10d-894de55d5405-kube-api-access-m9kdf" (OuterVolumeSpecName: "kube-api-access-m9kdf") pod "3fe6ea9d-0390-4585-b10d-894de55d5405" (UID: "3fe6ea9d-0390-4585-b10d-894de55d5405"). InnerVolumeSpecName "kube-api-access-m9kdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:49:04.050348 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:04.050319 2575 generic.go:356] "Generic (PLEG): container finished" podID="3fe6ea9d-0390-4585-b10d-894de55d5405" containerID="2b8d45337b7a6526f40fb0e1b166cdf0cb4784b38be0c8fac0a6ad35254d6165" exitCode=0 Jul 08 18:49:04.050459 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:04.050365 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hcs7p" Jul 08 18:49:04.050459 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:04.050404 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hcs7p" event={"ID":"3fe6ea9d-0390-4585-b10d-894de55d5405","Type":"ContainerDied","Data":"2b8d45337b7a6526f40fb0e1b166cdf0cb4784b38be0c8fac0a6ad35254d6165"} Jul 08 18:49:04.050540 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:04.050465 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hcs7p" event={"ID":"3fe6ea9d-0390-4585-b10d-894de55d5405","Type":"ContainerDied","Data":"22e7da6c8d1ffaab8646ae4e870d9cfa5262b71f6fd54bb4f96e2f25bd364a4d"} Jul 08 18:49:04.050540 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:04.050487 2575 scope.go:117] "RemoveContainer" containerID="2b8d45337b7a6526f40fb0e1b166cdf0cb4784b38be0c8fac0a6ad35254d6165" Jul 08 18:49:04.059607 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:04.059591 2575 scope.go:117] "RemoveContainer" containerID="2b8d45337b7a6526f40fb0e1b166cdf0cb4784b38be0c8fac0a6ad35254d6165" Jul 08 18:49:04.059862 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:49:04.059846 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b8d45337b7a6526f40fb0e1b166cdf0cb4784b38be0c8fac0a6ad35254d6165\": container with ID starting with 2b8d45337b7a6526f40fb0e1b166cdf0cb4784b38be0c8fac0a6ad35254d6165 not found: ID does not exist" containerID="2b8d45337b7a6526f40fb0e1b166cdf0cb4784b38be0c8fac0a6ad35254d6165" Jul 08 18:49:04.059912 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:04.059873 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b8d45337b7a6526f40fb0e1b166cdf0cb4784b38be0c8fac0a6ad35254d6165"} err="failed to get container status \"2b8d45337b7a6526f40fb0e1b166cdf0cb4784b38be0c8fac0a6ad35254d6165\": rpc error: code = NotFound desc = could not find container \"2b8d45337b7a6526f40fb0e1b166cdf0cb4784b38be0c8fac0a6ad35254d6165\": container with ID starting with 2b8d45337b7a6526f40fb0e1b166cdf0cb4784b38be0c8fac0a6ad35254d6165 not found: ID does not exist" Jul 08 18:49:04.073844 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:04.073823 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hcs7p"] Jul 08 18:49:04.081336 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:04.081316 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-hcs7p"] Jul 08 18:49:04.145308 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:04.145265 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9kdf\" (UniqueName: \"kubernetes.io/projected/3fe6ea9d-0390-4585-b10d-894de55d5405-kube-api-access-m9kdf\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 18:49:04.145308 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:04.145291 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fe6ea9d-0390-4585-b10d-894de55d5405-extensions-socket-volume\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 18:49:05.779863 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:05.779831 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fe6ea9d-0390-4585-b10d-894de55d5405" path="/var/lib/kubelet/pods/3fe6ea9d-0390-4585-b10d-894de55d5405/volumes" Jul 08 18:49:07.411254 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.411222 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2"] Jul 08 18:49:07.411621 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.411563 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f7b64e2-0a20-463f-be66-d5ad7b8da3c7" containerName="manager" Jul 08 18:49:07.411621 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.411575 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f7b64e2-0a20-463f-be66-d5ad7b8da3c7" containerName="manager" Jul 08 18:49:07.411621 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.411599 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fe6ea9d-0390-4585-b10d-894de55d5405" containerName="manager" Jul 08 18:49:07.411621 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.411604 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fe6ea9d-0390-4585-b10d-894de55d5405" containerName="manager" Jul 08 18:49:07.411749 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.411650 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f7b64e2-0a20-463f-be66-d5ad7b8da3c7" containerName="manager" Jul 08 18:49:07.411749 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.411660 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fe6ea9d-0390-4585-b10d-894de55d5405" containerName="manager" Jul 08 18:49:07.416140 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.416118 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.418934 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.418917 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-h4p4v\"" Jul 08 18:49:07.426877 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.426830 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2"] Jul 08 18:49:07.471954 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.471919 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/926ae9db-d53a-44d9-93ec-c93c16c604b2-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-w6zn2\" (UID: \"926ae9db-d53a-44d9-93ec-c93c16c604b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.472104 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.471966 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t9r6\" (UniqueName: \"kubernetes.io/projected/926ae9db-d53a-44d9-93ec-c93c16c604b2-kube-api-access-9t9r6\") pod \"maas-default-gateway-openshift-default-5896d4dd49-w6zn2\" (UID: \"926ae9db-d53a-44d9-93ec-c93c16c604b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.472104 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.471996 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/926ae9db-d53a-44d9-93ec-c93c16c604b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-w6zn2\" (UID: \"926ae9db-d53a-44d9-93ec-c93c16c604b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.472104 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.472019 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/926ae9db-d53a-44d9-93ec-c93c16c604b2-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-w6zn2\" (UID: \"926ae9db-d53a-44d9-93ec-c93c16c604b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.472104 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.472067 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/926ae9db-d53a-44d9-93ec-c93c16c604b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-w6zn2\" (UID: \"926ae9db-d53a-44d9-93ec-c93c16c604b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.472360 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.472127 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/926ae9db-d53a-44d9-93ec-c93c16c604b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-w6zn2\" (UID: \"926ae9db-d53a-44d9-93ec-c93c16c604b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.472360 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.472165 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/926ae9db-d53a-44d9-93ec-c93c16c604b2-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-w6zn2\" (UID: \"926ae9db-d53a-44d9-93ec-c93c16c604b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.472360 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.472222 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/926ae9db-d53a-44d9-93ec-c93c16c604b2-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-w6zn2\" (UID: \"926ae9db-d53a-44d9-93ec-c93c16c604b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.472360 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.472254 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/926ae9db-d53a-44d9-93ec-c93c16c604b2-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-w6zn2\" (UID: \"926ae9db-d53a-44d9-93ec-c93c16c604b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.573093 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.573055 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/926ae9db-d53a-44d9-93ec-c93c16c604b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-w6zn2\" (UID: \"926ae9db-d53a-44d9-93ec-c93c16c604b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.573252 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.573108 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/926ae9db-d53a-44d9-93ec-c93c16c604b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-w6zn2\" (UID: \"926ae9db-d53a-44d9-93ec-c93c16c604b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.573252 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.573142 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/926ae9db-d53a-44d9-93ec-c93c16c604b2-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-w6zn2\" (UID: \"926ae9db-d53a-44d9-93ec-c93c16c604b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.573252 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.573181 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/926ae9db-d53a-44d9-93ec-c93c16c604b2-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-w6zn2\" (UID: \"926ae9db-d53a-44d9-93ec-c93c16c604b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.573445 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.573337 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/926ae9db-d53a-44d9-93ec-c93c16c604b2-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-w6zn2\" (UID: \"926ae9db-d53a-44d9-93ec-c93c16c604b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.573445 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.573407 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/926ae9db-d53a-44d9-93ec-c93c16c604b2-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-w6zn2\" (UID: \"926ae9db-d53a-44d9-93ec-c93c16c604b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.573547 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.573453 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t9r6\" (UniqueName: \"kubernetes.io/projected/926ae9db-d53a-44d9-93ec-c93c16c604b2-kube-api-access-9t9r6\") pod \"maas-default-gateway-openshift-default-5896d4dd49-w6zn2\" (UID: \"926ae9db-d53a-44d9-93ec-c93c16c604b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.573547 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.573486 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/926ae9db-d53a-44d9-93ec-c93c16c604b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-w6zn2\" (UID: \"926ae9db-d53a-44d9-93ec-c93c16c604b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.573547 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.573512 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/926ae9db-d53a-44d9-93ec-c93c16c604b2-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-w6zn2\" (UID: \"926ae9db-d53a-44d9-93ec-c93c16c604b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.573757 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.573632 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/926ae9db-d53a-44d9-93ec-c93c16c604b2-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-w6zn2\" (UID: \"926ae9db-d53a-44d9-93ec-c93c16c604b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.573887 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.573864 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/926ae9db-d53a-44d9-93ec-c93c16c604b2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-w6zn2\" (UID: \"926ae9db-d53a-44d9-93ec-c93c16c604b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.574154 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.574129 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/926ae9db-d53a-44d9-93ec-c93c16c604b2-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-w6zn2\" (UID: \"926ae9db-d53a-44d9-93ec-c93c16c604b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.574275 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.574135 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/926ae9db-d53a-44d9-93ec-c93c16c604b2-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-w6zn2\" (UID: \"926ae9db-d53a-44d9-93ec-c93c16c604b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.574275 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.574204 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/926ae9db-d53a-44d9-93ec-c93c16c604b2-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-w6zn2\" (UID: \"926ae9db-d53a-44d9-93ec-c93c16c604b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.576042 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.576021 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/926ae9db-d53a-44d9-93ec-c93c16c604b2-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-w6zn2\" (UID: \"926ae9db-d53a-44d9-93ec-c93c16c604b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.576167 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.576152 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/926ae9db-d53a-44d9-93ec-c93c16c604b2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-w6zn2\" (UID: \"926ae9db-d53a-44d9-93ec-c93c16c604b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.582129 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.582106 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/926ae9db-d53a-44d9-93ec-c93c16c604b2-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-w6zn2\" (UID: \"926ae9db-d53a-44d9-93ec-c93c16c604b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.582254 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.582236 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t9r6\" (UniqueName: \"kubernetes.io/projected/926ae9db-d53a-44d9-93ec-c93c16c604b2-kube-api-access-9t9r6\") pod \"maas-default-gateway-openshift-default-5896d4dd49-w6zn2\" (UID: \"926ae9db-d53a-44d9-93ec-c93c16c604b2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.727780 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.727681 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:07.865823 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:07.865794 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2"] Jul 08 18:49:07.868342 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:49:07.868312 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926ae9db_d53a_44d9_93ec_c93c16c604b2.slice/crio-7b54ceb188aa41b862b5144e1d6ba586b511a220d21c10576442f7eb8254c02b WatchSource:0}: Error finding container 7b54ceb188aa41b862b5144e1d6ba586b511a220d21c10576442f7eb8254c02b: Status 404 returned error can't find the container with id 7b54ceb188aa41b862b5144e1d6ba586b511a220d21c10576442f7eb8254c02b Jul 08 18:49:08.070820 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:08.070735 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" event={"ID":"926ae9db-d53a-44d9-93ec-c93c16c604b2","Type":"ContainerStarted","Data":"7b54ceb188aa41b862b5144e1d6ba586b511a220d21c10576442f7eb8254c02b"} Jul 08 18:49:10.982889 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:10.982842 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 18:49:10.983169 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:10.982948 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 18:49:10.983169 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:10.982998 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 18:49:11.088999 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:11.088959 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" event={"ID":"926ae9db-d53a-44d9-93ec-c93c16c604b2","Type":"ContainerStarted","Data":"9470bc15942c900ccb61a02b49df98e49f1c8e45e14b73f5a1c9b471fa44003e"} Jul 08 18:49:11.115052 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:11.114991 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" podStartSLOduration=1.002796532 podStartE2EDuration="4.114969653s" totalImagesPullingTime="3.112173121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:49:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:49:11.114451941 +0000 UTC m=+561.951779670" watchObservedRunningTime="2026-07-08 18:49:11.114969653 +0000 UTC m=+561.952297361" Jul 08 18:49:11.728495 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:11.728443 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:12.732690 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:12.732658 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:13.098451 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:13.098363 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:13.099251 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:13.099231 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:49:24.480649 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:24.480612 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jr7p9"] Jul 08 18:49:24.484771 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:24.484750 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jr7p9" Jul 08 18:49:24.487706 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:24.487686 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fzfvr\"" Jul 08 18:49:24.487818 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:24.487685 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 08 18:49:24.491620 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:24.491593 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jr7p9"] Jul 08 18:49:24.577502 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:24.577465 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jr7p9"] Jul 08 18:49:24.620244 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:24.620212 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1b9835ff-36fc-4e82-b944-5ac6d08fe864-config-file\") pod \"limitador-limitador-64c94fd64f-jr7p9\" (UID: \"1b9835ff-36fc-4e82-b944-5ac6d08fe864\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jr7p9" Jul 08 18:49:24.620439 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:24.620255 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl5r9\" (UniqueName: \"kubernetes.io/projected/1b9835ff-36fc-4e82-b944-5ac6d08fe864-kube-api-access-pl5r9\") pod \"limitador-limitador-64c94fd64f-jr7p9\" (UID: \"1b9835ff-36fc-4e82-b944-5ac6d08fe864\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jr7p9" Jul 08 18:49:24.721635 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:24.721581 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1b9835ff-36fc-4e82-b944-5ac6d08fe864-config-file\") pod \"limitador-limitador-64c94fd64f-jr7p9\" (UID: \"1b9835ff-36fc-4e82-b944-5ac6d08fe864\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jr7p9" Jul 08 18:49:24.721635 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:24.721636 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl5r9\" (UniqueName: \"kubernetes.io/projected/1b9835ff-36fc-4e82-b944-5ac6d08fe864-kube-api-access-pl5r9\") pod \"limitador-limitador-64c94fd64f-jr7p9\" (UID: \"1b9835ff-36fc-4e82-b944-5ac6d08fe864\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jr7p9" Jul 08 18:49:24.722265 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:24.722245 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1b9835ff-36fc-4e82-b944-5ac6d08fe864-config-file\") pod \"limitador-limitador-64c94fd64f-jr7p9\" (UID: \"1b9835ff-36fc-4e82-b944-5ac6d08fe864\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jr7p9" Jul 08 18:49:24.730345 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:24.730313 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl5r9\" (UniqueName: \"kubernetes.io/projected/1b9835ff-36fc-4e82-b944-5ac6d08fe864-kube-api-access-pl5r9\") pod \"limitador-limitador-64c94fd64f-jr7p9\" (UID: \"1b9835ff-36fc-4e82-b944-5ac6d08fe864\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jr7p9" Jul 08 18:49:24.796015 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:24.795932 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jr7p9" Jul 08 18:49:24.935374 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:24.935339 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jr7p9"] Jul 08 18:49:24.939118 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:49:24.939090 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b9835ff_36fc_4e82_b944_5ac6d08fe864.slice/crio-eed42cb13997a1d4d08a6034d2575ed4ae28e06b91acbd9926e245fceadac54e WatchSource:0}: Error finding container eed42cb13997a1d4d08a6034d2575ed4ae28e06b91acbd9926e245fceadac54e: Status 404 returned error can't find the container with id eed42cb13997a1d4d08a6034d2575ed4ae28e06b91acbd9926e245fceadac54e Jul 08 18:49:25.151918 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:25.151882 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jr7p9" event={"ID":"1b9835ff-36fc-4e82-b944-5ac6d08fe864","Type":"ContainerStarted","Data":"eed42cb13997a1d4d08a6034d2575ed4ae28e06b91acbd9926e245fceadac54e"} Jul 08 18:49:28.168609 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:28.168562 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jr7p9" event={"ID":"1b9835ff-36fc-4e82-b944-5ac6d08fe864","Type":"ContainerStarted","Data":"5d4ff36dba4acf17930baaecb3509508a95c839effd91c9176379865ebb6de4a"} Jul 08 18:49:28.169050 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:28.168628 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-jr7p9" Jul 08 18:49:28.188136 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:28.188079 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-jr7p9" podStartSLOduration=1.522666715 podStartE2EDuration="4.188064978s" totalImagesPullingTime="2.665398263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:49:28.186715447 +0000 UTC m=+579.024043155" watchObservedRunningTime="2026-07-08 18:49:28.188064978 +0000 UTC m=+579.025392686" Jul 08 18:49:39.174610 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:39.174576 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-jr7p9" Jul 08 18:49:39.367981 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:39.367946 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jr7p9"] Jul 08 18:49:39.368156 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:39.368131 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-jr7p9" podUID="1b9835ff-36fc-4e82-b944-5ac6d08fe864" containerName="limitador" containerID="cri-o://5d4ff36dba4acf17930baaecb3509508a95c839effd91c9176379865ebb6de4a" gracePeriod=30 Jul 08 18:49:39.922339 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:39.922309 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jr7p9" Jul 08 18:49:39.965363 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:39.965254 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pl5r9\" (UniqueName: \"kubernetes.io/projected/1b9835ff-36fc-4e82-b944-5ac6d08fe864-kube-api-access-pl5r9\") pod \"1b9835ff-36fc-4e82-b944-5ac6d08fe864\" (UID: \"1b9835ff-36fc-4e82-b944-5ac6d08fe864\") " Jul 08 18:49:39.967449 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:39.967421 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b9835ff-36fc-4e82-b944-5ac6d08fe864-kube-api-access-pl5r9" (OuterVolumeSpecName: "kube-api-access-pl5r9") pod "1b9835ff-36fc-4e82-b944-5ac6d08fe864" (UID: "1b9835ff-36fc-4e82-b944-5ac6d08fe864"). InnerVolumeSpecName "kube-api-access-pl5r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:49:40.066358 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:40.066323 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1b9835ff-36fc-4e82-b944-5ac6d08fe864-config-file\") pod \"1b9835ff-36fc-4e82-b944-5ac6d08fe864\" (UID: \"1b9835ff-36fc-4e82-b944-5ac6d08fe864\") " Jul 08 18:49:40.066514 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:40.066501 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl5r9\" (UniqueName: \"kubernetes.io/projected/1b9835ff-36fc-4e82-b944-5ac6d08fe864-kube-api-access-pl5r9\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 18:49:40.066710 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:40.066691 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b9835ff-36fc-4e82-b944-5ac6d08fe864-config-file" (OuterVolumeSpecName: "config-file") pod "1b9835ff-36fc-4e82-b944-5ac6d08fe864" (UID: "1b9835ff-36fc-4e82-b944-5ac6d08fe864"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:49:40.167114 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:40.167079 2575 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1b9835ff-36fc-4e82-b944-5ac6d08fe864-config-file\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 18:49:40.224206 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:40.224127 2575 generic.go:356] "Generic (PLEG): container finished" podID="1b9835ff-36fc-4e82-b944-5ac6d08fe864" containerID="5d4ff36dba4acf17930baaecb3509508a95c839effd91c9176379865ebb6de4a" exitCode=0 Jul 08 18:49:40.224206 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:40.224169 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jr7p9" Jul 08 18:49:40.224687 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:40.224211 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jr7p9" event={"ID":"1b9835ff-36fc-4e82-b944-5ac6d08fe864","Type":"ContainerDied","Data":"5d4ff36dba4acf17930baaecb3509508a95c839effd91c9176379865ebb6de4a"} Jul 08 18:49:40.224687 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:40.224263 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jr7p9" event={"ID":"1b9835ff-36fc-4e82-b944-5ac6d08fe864","Type":"ContainerDied","Data":"eed42cb13997a1d4d08a6034d2575ed4ae28e06b91acbd9926e245fceadac54e"} Jul 08 18:49:40.224687 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:40.224284 2575 scope.go:117] "RemoveContainer" containerID="5d4ff36dba4acf17930baaecb3509508a95c839effd91c9176379865ebb6de4a" Jul 08 18:49:40.232856 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:40.232841 2575 scope.go:117] "RemoveContainer" containerID="5d4ff36dba4acf17930baaecb3509508a95c839effd91c9176379865ebb6de4a" Jul 08 18:49:40.233098 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:49:40.233077 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d4ff36dba4acf17930baaecb3509508a95c839effd91c9176379865ebb6de4a\": container with ID starting with 5d4ff36dba4acf17930baaecb3509508a95c839effd91c9176379865ebb6de4a not found: ID does not exist" containerID="5d4ff36dba4acf17930baaecb3509508a95c839effd91c9176379865ebb6de4a" Jul 08 18:49:40.233142 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:40.233109 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d4ff36dba4acf17930baaecb3509508a95c839effd91c9176379865ebb6de4a"} err="failed to get container status \"5d4ff36dba4acf17930baaecb3509508a95c839effd91c9176379865ebb6de4a\": rpc error: code = NotFound desc = could not find container \"5d4ff36dba4acf17930baaecb3509508a95c839effd91c9176379865ebb6de4a\": container with ID starting with 5d4ff36dba4acf17930baaecb3509508a95c839effd91c9176379865ebb6de4a not found: ID does not exist" Jul 08 18:49:40.246105 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:40.246082 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jr7p9"] Jul 08 18:49:40.249257 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:40.249238 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jr7p9"] Jul 08 18:49:41.780140 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:49:41.780104 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b9835ff-36fc-4e82-b944-5ac6d08fe864" path="/var/lib/kubelet/pods/1b9835ff-36fc-4e82-b944-5ac6d08fe864/volumes" Jul 08 18:50:09.914145 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:09.914111 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b87cbb67f-q7zl8"] Jul 08 18:50:09.914604 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:09.914455 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b9835ff-36fc-4e82-b944-5ac6d08fe864" containerName="limitador" Jul 08 18:50:09.914604 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:09.914466 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b9835ff-36fc-4e82-b944-5ac6d08fe864" containerName="limitador" Jul 08 18:50:09.914604 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:09.914530 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b9835ff-36fc-4e82-b944-5ac6d08fe864" containerName="limitador" Jul 08 18:50:09.925104 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:09.925078 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b87cbb67f-q7zl8" Jul 08 18:50:09.928061 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:09.927921 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 08 18:50:09.928061 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:09.927936 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 08 18:50:09.928061 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:09.927973 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7drrl\"" Jul 08 18:50:09.932013 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:09.931949 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b87cbb67f-q7zl8"] Jul 08 18:50:10.007525 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:10.007484 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/51c9a450-d9ba-475d-b6bb-cce750bfdcf2-webhook-cert\") pod \"maas-controller-b87cbb67f-q7zl8\" (UID: \"51c9a450-d9ba-475d-b6bb-cce750bfdcf2\") " pod="opendatahub/maas-controller-b87cbb67f-q7zl8" Jul 08 18:50:10.007719 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:10.007541 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhs4z\" (UniqueName: \"kubernetes.io/projected/51c9a450-d9ba-475d-b6bb-cce750bfdcf2-kube-api-access-zhs4z\") pod \"maas-controller-b87cbb67f-q7zl8\" (UID: \"51c9a450-d9ba-475d-b6bb-cce750bfdcf2\") " pod="opendatahub/maas-controller-b87cbb67f-q7zl8" Jul 08 18:50:10.108833 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:10.108791 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/51c9a450-d9ba-475d-b6bb-cce750bfdcf2-webhook-cert\") pod \"maas-controller-b87cbb67f-q7zl8\" (UID: \"51c9a450-d9ba-475d-b6bb-cce750bfdcf2\") " pod="opendatahub/maas-controller-b87cbb67f-q7zl8" Jul 08 18:50:10.109005 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:10.108856 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhs4z\" (UniqueName: \"kubernetes.io/projected/51c9a450-d9ba-475d-b6bb-cce750bfdcf2-kube-api-access-zhs4z\") pod \"maas-controller-b87cbb67f-q7zl8\" (UID: \"51c9a450-d9ba-475d-b6bb-cce750bfdcf2\") " pod="opendatahub/maas-controller-b87cbb67f-q7zl8" Jul 08 18:50:10.109005 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:50:10.108949 2575 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 08 18:50:10.109084 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:50:10.109018 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51c9a450-d9ba-475d-b6bb-cce750bfdcf2-webhook-cert podName:51c9a450-d9ba-475d-b6bb-cce750bfdcf2 nodeName:}" failed. No retries permitted until 2026-07-08 18:50:10.608998821 +0000 UTC m=+621.446326511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/51c9a450-d9ba-475d-b6bb-cce750bfdcf2-webhook-cert") pod "maas-controller-b87cbb67f-q7zl8" (UID: "51c9a450-d9ba-475d-b6bb-cce750bfdcf2") : secret "maas-controller-webhook-cert" not found Jul 08 18:50:10.117897 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:10.117858 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhs4z\" (UniqueName: \"kubernetes.io/projected/51c9a450-d9ba-475d-b6bb-cce750bfdcf2-kube-api-access-zhs4z\") pod \"maas-controller-b87cbb67f-q7zl8\" (UID: \"51c9a450-d9ba-475d-b6bb-cce750bfdcf2\") " pod="opendatahub/maas-controller-b87cbb67f-q7zl8" Jul 08 18:50:10.619564 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:10.613968 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/51c9a450-d9ba-475d-b6bb-cce750bfdcf2-webhook-cert\") pod \"maas-controller-b87cbb67f-q7zl8\" (UID: \"51c9a450-d9ba-475d-b6bb-cce750bfdcf2\") " pod="opendatahub/maas-controller-b87cbb67f-q7zl8" Jul 08 18:50:10.619564 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:10.617983 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/51c9a450-d9ba-475d-b6bb-cce750bfdcf2-webhook-cert\") pod \"maas-controller-b87cbb67f-q7zl8\" (UID: \"51c9a450-d9ba-475d-b6bb-cce750bfdcf2\") " pod="opendatahub/maas-controller-b87cbb67f-q7zl8" Jul 08 18:50:10.837467 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:10.837433 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b87cbb67f-q7zl8" Jul 08 18:50:10.976350 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:10.976319 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b87cbb67f-q7zl8"] Jul 08 18:50:10.978263 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:50:10.978232 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c9a450_d9ba_475d_b6bb_cce750bfdcf2.slice/crio-a6a1edc5f43adbd74d30ae13255dd504e35919dcb134eef1cfb3efa5b3f7f9ae WatchSource:0}: Error finding container a6a1edc5f43adbd74d30ae13255dd504e35919dcb134eef1cfb3efa5b3f7f9ae: Status 404 returned error can't find the container with id a6a1edc5f43adbd74d30ae13255dd504e35919dcb134eef1cfb3efa5b3f7f9ae Jul 08 18:50:11.375205 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:11.375168 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b87cbb67f-q7zl8" event={"ID":"51c9a450-d9ba-475d-b6bb-cce750bfdcf2","Type":"ContainerStarted","Data":"a6a1edc5f43adbd74d30ae13255dd504e35919dcb134eef1cfb3efa5b3f7f9ae"} Jul 08 18:50:14.393467 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:14.393432 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b87cbb67f-q7zl8" event={"ID":"51c9a450-d9ba-475d-b6bb-cce750bfdcf2","Type":"ContainerStarted","Data":"d88ba856cdf61c0fb4e130dcf7dc7767952f0edd119c977943ac4b61006a2734"} Jul 08 18:50:14.393850 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:14.393500 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b87cbb67f-q7zl8" Jul 08 18:50:15.125933 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:15.125872 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-b87cbb67f-q7zl8" podStartSLOduration=3.397327431 podStartE2EDuration="6.12584957s" totalImagesPullingTime="2.728522139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:50:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:50:14.438048292 +0000 UTC m=+625.275376002" watchObservedRunningTime="2026-07-08 18:50:15.12584957 +0000 UTC m=+625.963177278" Jul 08 18:50:15.126151 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:15.126136 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6449898b5-8dc9t"] Jul 08 18:50:15.129690 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:15.129668 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6449898b5-8dc9t" Jul 08 18:50:15.144953 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:15.144910 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6449898b5-8dc9t"] Jul 08 18:50:15.252451 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:15.252412 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9hqm\" (UniqueName: \"kubernetes.io/projected/ecd2a433-4cca-4d9f-96db-cca824e24e64-kube-api-access-v9hqm\") pod \"maas-controller-6449898b5-8dc9t\" (UID: \"ecd2a433-4cca-4d9f-96db-cca824e24e64\") " pod="opendatahub/maas-controller-6449898b5-8dc9t" Jul 08 18:50:15.252644 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:15.252548 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecd2a433-4cca-4d9f-96db-cca824e24e64-webhook-cert\") pod \"maas-controller-6449898b5-8dc9t\" (UID: \"ecd2a433-4cca-4d9f-96db-cca824e24e64\") " pod="opendatahub/maas-controller-6449898b5-8dc9t" Jul 08 18:50:15.353023 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:15.352980 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecd2a433-4cca-4d9f-96db-cca824e24e64-webhook-cert\") pod \"maas-controller-6449898b5-8dc9t\" (UID: \"ecd2a433-4cca-4d9f-96db-cca824e24e64\") " pod="opendatahub/maas-controller-6449898b5-8dc9t" Jul 08 18:50:15.353207 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:15.353044 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9hqm\" (UniqueName: \"kubernetes.io/projected/ecd2a433-4cca-4d9f-96db-cca824e24e64-kube-api-access-v9hqm\") pod \"maas-controller-6449898b5-8dc9t\" (UID: \"ecd2a433-4cca-4d9f-96db-cca824e24e64\") " pod="opendatahub/maas-controller-6449898b5-8dc9t" Jul 08 18:50:15.355453 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:15.355427 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecd2a433-4cca-4d9f-96db-cca824e24e64-webhook-cert\") pod \"maas-controller-6449898b5-8dc9t\" (UID: \"ecd2a433-4cca-4d9f-96db-cca824e24e64\") " pod="opendatahub/maas-controller-6449898b5-8dc9t" Jul 08 18:50:15.368651 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:15.368622 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9hqm\" (UniqueName: \"kubernetes.io/projected/ecd2a433-4cca-4d9f-96db-cca824e24e64-kube-api-access-v9hqm\") pod \"maas-controller-6449898b5-8dc9t\" (UID: \"ecd2a433-4cca-4d9f-96db-cca824e24e64\") " pod="opendatahub/maas-controller-6449898b5-8dc9t" Jul 08 18:50:15.450513 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:15.450422 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6449898b5-8dc9t" Jul 08 18:50:15.590101 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:15.590028 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6449898b5-8dc9t"] Jul 08 18:50:15.592192 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:50:15.592164 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd2a433_4cca_4d9f_96db_cca824e24e64.slice/crio-5f33b45998b742dff949a32ca3af1a7862fe33d7db73caef44a190488612da21 WatchSource:0}: Error finding container 5f33b45998b742dff949a32ca3af1a7862fe33d7db73caef44a190488612da21: Status 404 returned error can't find the container with id 5f33b45998b742dff949a32ca3af1a7862fe33d7db73caef44a190488612da21 Jul 08 18:50:16.404160 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:16.404125 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6449898b5-8dc9t" event={"ID":"ecd2a433-4cca-4d9f-96db-cca824e24e64","Type":"ContainerStarted","Data":"9f3d9f9422e875bd66fb6a16d2f0c3eef2f109879e6b5f3f319c3deca95a2e02"} Jul 08 18:50:16.404371 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:16.404178 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6449898b5-8dc9t" event={"ID":"ecd2a433-4cca-4d9f-96db-cca824e24e64","Type":"ContainerStarted","Data":"5f33b45998b742dff949a32ca3af1a7862fe33d7db73caef44a190488612da21"} Jul 08 18:50:16.404371 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:16.404256 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6449898b5-8dc9t" Jul 08 18:50:16.423922 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:16.423868 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6449898b5-8dc9t" podStartSLOduration=1.025234984 podStartE2EDuration="1.423851989s" totalImagesPullingTime="398.617005ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:50:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:50:16.422039111 +0000 UTC m=+627.259366832" watchObservedRunningTime="2026-07-08 18:50:16.423851989 +0000 UTC m=+627.261179697" Jul 08 18:50:25.404270 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:25.404231 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b87cbb67f-q7zl8" Jul 08 18:50:27.418770 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:27.418732 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6449898b5-8dc9t" Jul 08 18:50:27.463878 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:27.463839 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b87cbb67f-q7zl8"] Jul 08 18:50:27.464081 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:27.464055 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-b87cbb67f-q7zl8" podUID="51c9a450-d9ba-475d-b6bb-cce750bfdcf2" containerName="manager" containerID="cri-o://d88ba856cdf61c0fb4e130dcf7dc7767952f0edd119c977943ac4b61006a2734" gracePeriod=30 Jul 08 18:50:27.706467 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:27.706439 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b87cbb67f-q7zl8" Jul 08 18:50:27.858945 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:27.858914 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhs4z\" (UniqueName: \"kubernetes.io/projected/51c9a450-d9ba-475d-b6bb-cce750bfdcf2-kube-api-access-zhs4z\") pod \"51c9a450-d9ba-475d-b6bb-cce750bfdcf2\" (UID: \"51c9a450-d9ba-475d-b6bb-cce750bfdcf2\") " Jul 08 18:50:27.859134 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:27.858990 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/51c9a450-d9ba-475d-b6bb-cce750bfdcf2-webhook-cert\") pod \"51c9a450-d9ba-475d-b6bb-cce750bfdcf2\" (UID: \"51c9a450-d9ba-475d-b6bb-cce750bfdcf2\") " Jul 08 18:50:27.861203 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:27.861166 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51c9a450-d9ba-475d-b6bb-cce750bfdcf2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "51c9a450-d9ba-475d-b6bb-cce750bfdcf2" (UID: "51c9a450-d9ba-475d-b6bb-cce750bfdcf2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:50:27.861203 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:27.861177 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c9a450-d9ba-475d-b6bb-cce750bfdcf2-kube-api-access-zhs4z" (OuterVolumeSpecName: "kube-api-access-zhs4z") pod "51c9a450-d9ba-475d-b6bb-cce750bfdcf2" (UID: "51c9a450-d9ba-475d-b6bb-cce750bfdcf2"). InnerVolumeSpecName "kube-api-access-zhs4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:50:27.959887 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:27.959794 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhs4z\" (UniqueName: \"kubernetes.io/projected/51c9a450-d9ba-475d-b6bb-cce750bfdcf2-kube-api-access-zhs4z\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 18:50:27.959887 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:27.959833 2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/51c9a450-d9ba-475d-b6bb-cce750bfdcf2-webhook-cert\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 18:50:28.459910 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:28.459873 2575 generic.go:356] "Generic (PLEG): container finished" podID="51c9a450-d9ba-475d-b6bb-cce750bfdcf2" containerID="d88ba856cdf61c0fb4e130dcf7dc7767952f0edd119c977943ac4b61006a2734" exitCode=0 Jul 08 18:50:28.460339 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:28.459920 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b87cbb67f-q7zl8" Jul 08 18:50:28.460339 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:28.459959 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b87cbb67f-q7zl8" event={"ID":"51c9a450-d9ba-475d-b6bb-cce750bfdcf2","Type":"ContainerDied","Data":"d88ba856cdf61c0fb4e130dcf7dc7767952f0edd119c977943ac4b61006a2734"} Jul 08 18:50:28.460339 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:28.460021 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b87cbb67f-q7zl8" event={"ID":"51c9a450-d9ba-475d-b6bb-cce750bfdcf2","Type":"ContainerDied","Data":"a6a1edc5f43adbd74d30ae13255dd504e35919dcb134eef1cfb3efa5b3f7f9ae"} Jul 08 18:50:28.460339 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:28.460049 2575 scope.go:117] "RemoveContainer" containerID="d88ba856cdf61c0fb4e130dcf7dc7767952f0edd119c977943ac4b61006a2734" Jul 08 18:50:28.469580 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:28.469411 2575 scope.go:117] "RemoveContainer" containerID="d88ba856cdf61c0fb4e130dcf7dc7767952f0edd119c977943ac4b61006a2734" Jul 08 18:50:28.469684 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:50:28.469666 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d88ba856cdf61c0fb4e130dcf7dc7767952f0edd119c977943ac4b61006a2734\": container with ID starting with d88ba856cdf61c0fb4e130dcf7dc7767952f0edd119c977943ac4b61006a2734 not found: ID does not exist" containerID="d88ba856cdf61c0fb4e130dcf7dc7767952f0edd119c977943ac4b61006a2734" Jul 08 18:50:28.469726 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:28.469696 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d88ba856cdf61c0fb4e130dcf7dc7767952f0edd119c977943ac4b61006a2734"} err="failed to get container status \"d88ba856cdf61c0fb4e130dcf7dc7767952f0edd119c977943ac4b61006a2734\": rpc error: code = NotFound desc = could not find container \"d88ba856cdf61c0fb4e130dcf7dc7767952f0edd119c977943ac4b61006a2734\": container with ID starting with d88ba856cdf61c0fb4e130dcf7dc7767952f0edd119c977943ac4b61006a2734 not found: ID does not exist" Jul 08 18:50:28.481671 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:28.481646 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b87cbb67f-q7zl8"] Jul 08 18:50:28.485426 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:28.485403 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-b87cbb67f-q7zl8"] Jul 08 18:50:29.780027 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:29.779991 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51c9a450-d9ba-475d-b6bb-cce750bfdcf2" path="/var/lib/kubelet/pods/51c9a450-d9ba-475d-b6bb-cce750bfdcf2/volumes" Jul 08 18:50:50.399016 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.398981 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-658c95c5dd-svgvn"] Jul 08 18:50:50.399586 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.399357 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51c9a450-d9ba-475d-b6bb-cce750bfdcf2" containerName="manager" Jul 08 18:50:50.399586 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.399371 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="51c9a450-d9ba-475d-b6bb-cce750bfdcf2" containerName="manager" Jul 08 18:50:50.399586 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.399441 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="51c9a450-d9ba-475d-b6bb-cce750bfdcf2" containerName="manager" Jul 08 18:50:50.402078 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.402060 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-658c95c5dd-svgvn" Jul 08 18:50:50.405109 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.405084 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 08 18:50:50.405247 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.405086 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 18:50:50.405517 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.405501 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kxbw6\"" Jul 08 18:50:50.406372 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.406360 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 18:50:50.415181 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.415158 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-658c95c5dd-svgvn"] Jul 08 18:50:50.453323 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.453267 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3abe33e-9ff1-4cac-8e3b-114fb6c63761-maas-api-tls\") pod \"maas-api-658c95c5dd-svgvn\" (UID: \"d3abe33e-9ff1-4cac-8e3b-114fb6c63761\") " pod="odh-ai-gateway-infra/maas-api-658c95c5dd-svgvn" Jul 08 18:50:50.453517 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.453380 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xthn9\" (UniqueName: \"kubernetes.io/projected/d3abe33e-9ff1-4cac-8e3b-114fb6c63761-kube-api-access-xthn9\") pod \"maas-api-658c95c5dd-svgvn\" (UID: \"d3abe33e-9ff1-4cac-8e3b-114fb6c63761\") " pod="odh-ai-gateway-infra/maas-api-658c95c5dd-svgvn" Jul 08 18:50:50.453517 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.453436 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3abe33e-9ff1-4cac-8e3b-114fb6c63761-openshift-service-ca\") pod \"maas-api-658c95c5dd-svgvn\" (UID: \"d3abe33e-9ff1-4cac-8e3b-114fb6c63761\") " pod="odh-ai-gateway-infra/maas-api-658c95c5dd-svgvn" Jul 08 18:50:50.554797 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.554760 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3abe33e-9ff1-4cac-8e3b-114fb6c63761-maas-api-tls\") pod \"maas-api-658c95c5dd-svgvn\" (UID: \"d3abe33e-9ff1-4cac-8e3b-114fb6c63761\") " pod="odh-ai-gateway-infra/maas-api-658c95c5dd-svgvn" Jul 08 18:50:50.555056 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.554815 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xthn9\" (UniqueName: \"kubernetes.io/projected/d3abe33e-9ff1-4cac-8e3b-114fb6c63761-kube-api-access-xthn9\") pod \"maas-api-658c95c5dd-svgvn\" (UID: \"d3abe33e-9ff1-4cac-8e3b-114fb6c63761\") " pod="odh-ai-gateway-infra/maas-api-658c95c5dd-svgvn" Jul 08 18:50:50.555056 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.554864 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3abe33e-9ff1-4cac-8e3b-114fb6c63761-openshift-service-ca\") pod \"maas-api-658c95c5dd-svgvn\" (UID: \"d3abe33e-9ff1-4cac-8e3b-114fb6c63761\") " pod="odh-ai-gateway-infra/maas-api-658c95c5dd-svgvn" Jul 08 18:50:50.555056 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:50:50.554927 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 08 18:50:50.555056 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:50:50.555000 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3abe33e-9ff1-4cac-8e3b-114fb6c63761-maas-api-tls podName:d3abe33e-9ff1-4cac-8e3b-114fb6c63761 nodeName:}" failed. No retries permitted until 2026-07-08 18:50:51.054980867 +0000 UTC m=+661.892308552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d3abe33e-9ff1-4cac-8e3b-114fb6c63761-maas-api-tls") pod "maas-api-658c95c5dd-svgvn" (UID: "d3abe33e-9ff1-4cac-8e3b-114fb6c63761") : secret "maas-api-serving-cert" not found Jul 08 18:50:50.555649 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.555628 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3abe33e-9ff1-4cac-8e3b-114fb6c63761-openshift-service-ca\") pod \"maas-api-658c95c5dd-svgvn\" (UID: \"d3abe33e-9ff1-4cac-8e3b-114fb6c63761\") " pod="odh-ai-gateway-infra/maas-api-658c95c5dd-svgvn" Jul 08 18:50:50.574028 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.573994 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xthn9\" (UniqueName: \"kubernetes.io/projected/d3abe33e-9ff1-4cac-8e3b-114fb6c63761-kube-api-access-xthn9\") pod \"maas-api-658c95c5dd-svgvn\" (UID: \"d3abe33e-9ff1-4cac-8e3b-114fb6c63761\") " pod="odh-ai-gateway-infra/maas-api-658c95c5dd-svgvn" Jul 08 18:50:50.931462 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.931431 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-plb8f"] Jul 08 18:50:50.934531 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.934509 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-plb8f" Jul 08 18:50:50.937950 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.937925 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 08 18:50:50.938084 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.938025 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-ljl9j\"" Jul 08 18:50:50.946589 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.946565 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-plb8f"] Jul 08 18:50:50.960024 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.959992 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckgtb\" (UniqueName: \"kubernetes.io/projected/dbed1f82-e4b6-4cdc-a68c-40dcf2448650-kube-api-access-ckgtb\") pod \"payload-pre-processing-7964457696-plb8f\" (UID: \"dbed1f82-e4b6-4cdc-a68c-40dcf2448650\") " pod="openshift-ingress/payload-pre-processing-7964457696-plb8f" Jul 08 18:50:50.960158 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.960077 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbed1f82-e4b6-4cdc-a68c-40dcf2448650-plugins-config-volume\") pod \"payload-pre-processing-7964457696-plb8f\" (UID: \"dbed1f82-e4b6-4cdc-a68c-40dcf2448650\") " pod="openshift-ingress/payload-pre-processing-7964457696-plb8f" Jul 08 18:50:50.966189 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.966166 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-qcc7s"] Jul 08 18:50:50.968758 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.968743 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-qcc7s" Jul 08 18:50:50.982642 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:50.982615 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-qcc7s"] Jul 08 18:50:51.061103 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:51.061064 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70b85c5d-9a96-4b41-a062-8de6a4809609-plugins-config-volume\") pod \"payload-processing-58f86c7798-qcc7s\" (UID: \"70b85c5d-9a96-4b41-a062-8de6a4809609\") " pod="openshift-ingress/payload-processing-58f86c7798-qcc7s" Jul 08 18:50:51.061103 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:51.061104 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsqss\" (UniqueName: \"kubernetes.io/projected/70b85c5d-9a96-4b41-a062-8de6a4809609-kube-api-access-tsqss\") pod \"payload-processing-58f86c7798-qcc7s\" (UID: \"70b85c5d-9a96-4b41-a062-8de6a4809609\") " pod="openshift-ingress/payload-processing-58f86c7798-qcc7s" Jul 08 18:50:51.061316 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:51.061202 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbed1f82-e4b6-4cdc-a68c-40dcf2448650-plugins-config-volume\") pod \"payload-pre-processing-7964457696-plb8f\" (UID: \"dbed1f82-e4b6-4cdc-a68c-40dcf2448650\") " pod="openshift-ingress/payload-pre-processing-7964457696-plb8f" Jul 08 18:50:51.061316 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:51.061259 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3abe33e-9ff1-4cac-8e3b-114fb6c63761-maas-api-tls\") pod \"maas-api-658c95c5dd-svgvn\" (UID: \"d3abe33e-9ff1-4cac-8e3b-114fb6c63761\") " pod="odh-ai-gateway-infra/maas-api-658c95c5dd-svgvn" Jul 08 18:50:51.061316 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:51.061289 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckgtb\" (UniqueName: \"kubernetes.io/projected/dbed1f82-e4b6-4cdc-a68c-40dcf2448650-kube-api-access-ckgtb\") pod \"payload-pre-processing-7964457696-plb8f\" (UID: \"dbed1f82-e4b6-4cdc-a68c-40dcf2448650\") " pod="openshift-ingress/payload-pre-processing-7964457696-plb8f" Jul 08 18:50:51.062149 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:51.062124 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbed1f82-e4b6-4cdc-a68c-40dcf2448650-plugins-config-volume\") pod \"payload-pre-processing-7964457696-plb8f\" (UID: \"dbed1f82-e4b6-4cdc-a68c-40dcf2448650\") " pod="openshift-ingress/payload-pre-processing-7964457696-plb8f" Jul 08 18:50:51.064096 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:51.064072 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3abe33e-9ff1-4cac-8e3b-114fb6c63761-maas-api-tls\") pod \"maas-api-658c95c5dd-svgvn\" (UID: \"d3abe33e-9ff1-4cac-8e3b-114fb6c63761\") " pod="odh-ai-gateway-infra/maas-api-658c95c5dd-svgvn" Jul 08 18:50:51.076353 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:51.076330 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckgtb\" (UniqueName: \"kubernetes.io/projected/dbed1f82-e4b6-4cdc-a68c-40dcf2448650-kube-api-access-ckgtb\") pod \"payload-pre-processing-7964457696-plb8f\" (UID: \"dbed1f82-e4b6-4cdc-a68c-40dcf2448650\") " pod="openshift-ingress/payload-pre-processing-7964457696-plb8f" Jul 08 18:50:51.162119 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:51.162075 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70b85c5d-9a96-4b41-a062-8de6a4809609-plugins-config-volume\") pod \"payload-processing-58f86c7798-qcc7s\" (UID: \"70b85c5d-9a96-4b41-a062-8de6a4809609\") " pod="openshift-ingress/payload-processing-58f86c7798-qcc7s" Jul 08 18:50:51.162119 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:51.162125 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsqss\" (UniqueName: \"kubernetes.io/projected/70b85c5d-9a96-4b41-a062-8de6a4809609-kube-api-access-tsqss\") pod \"payload-processing-58f86c7798-qcc7s\" (UID: \"70b85c5d-9a96-4b41-a062-8de6a4809609\") " pod="openshift-ingress/payload-processing-58f86c7798-qcc7s" Jul 08 18:50:51.163058 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:51.163024 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70b85c5d-9a96-4b41-a062-8de6a4809609-plugins-config-volume\") pod \"payload-processing-58f86c7798-qcc7s\" (UID: \"70b85c5d-9a96-4b41-a062-8de6a4809609\") " pod="openshift-ingress/payload-processing-58f86c7798-qcc7s" Jul 08 18:50:51.171646 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:51.171616 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsqss\" (UniqueName: \"kubernetes.io/projected/70b85c5d-9a96-4b41-a062-8de6a4809609-kube-api-access-tsqss\") pod \"payload-processing-58f86c7798-qcc7s\" (UID: \"70b85c5d-9a96-4b41-a062-8de6a4809609\") " pod="openshift-ingress/payload-processing-58f86c7798-qcc7s" Jul 08 18:50:51.246720 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:51.246633 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-plb8f" Jul 08 18:50:51.279114 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:51.279083 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-qcc7s" Jul 08 18:50:51.313115 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:51.313003 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-658c95c5dd-svgvn" Jul 08 18:50:51.407758 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:51.407290 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-plb8f"] Jul 08 18:50:51.409576 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:50:51.409543 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbed1f82_e4b6_4cdc_a68c_40dcf2448650.slice/crio-4adf2338b5f7fabf9b6f73c3d433276e855425627ba17d069d03633dbce5e482 WatchSource:0}: Error finding container 4adf2338b5f7fabf9b6f73c3d433276e855425627ba17d069d03633dbce5e482: Status 404 returned error can't find the container with id 4adf2338b5f7fabf9b6f73c3d433276e855425627ba17d069d03633dbce5e482 Jul 08 18:50:51.411633 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:51.411616 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 18:50:51.439347 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:51.439319 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-qcc7s"] Jul 08 18:50:51.441986 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:50:51.441948 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b85c5d_9a96_4b41_a062_8de6a4809609.slice/crio-070c582d78f967c41aa63d301b58f31ba536b77d5460a608cde6b5aeae44e7ee WatchSource:0}: Error finding container 070c582d78f967c41aa63d301b58f31ba536b77d5460a608cde6b5aeae44e7ee: Status 404 returned error can't find the container with id 070c582d78f967c41aa63d301b58f31ba536b77d5460a608cde6b5aeae44e7ee Jul 08 18:50:51.478317 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:51.478281 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-658c95c5dd-svgvn"] Jul 08 18:50:51.479618 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:50:51.479593 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3abe33e_9ff1_4cac_8e3b_114fb6c63761.slice/crio-f4a8744cbae5c9530de9c2b2ea96c42983410733c53d44042be50d99d5c72d68 WatchSource:0}: Error finding container f4a8744cbae5c9530de9c2b2ea96c42983410733c53d44042be50d99d5c72d68: Status 404 returned error can't find the container with id f4a8744cbae5c9530de9c2b2ea96c42983410733c53d44042be50d99d5c72d68 Jul 08 18:50:51.564741 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:51.564651 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-plb8f" event={"ID":"dbed1f82-e4b6-4cdc-a68c-40dcf2448650","Type":"ContainerStarted","Data":"4adf2338b5f7fabf9b6f73c3d433276e855425627ba17d069d03633dbce5e482"} Jul 08 18:50:51.565760 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:51.565736 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-qcc7s" event={"ID":"70b85c5d-9a96-4b41-a062-8de6a4809609","Type":"ContainerStarted","Data":"070c582d78f967c41aa63d301b58f31ba536b77d5460a608cde6b5aeae44e7ee"} Jul 08 18:50:51.566746 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:51.566719 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-658c95c5dd-svgvn" event={"ID":"d3abe33e-9ff1-4cac-8e3b-114fb6c63761","Type":"ContainerStarted","Data":"f4a8744cbae5c9530de9c2b2ea96c42983410733c53d44042be50d99d5c72d68"} Jul 08 18:50:54.586911 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:54.586857 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-658c95c5dd-svgvn" event={"ID":"d3abe33e-9ff1-4cac-8e3b-114fb6c63761","Type":"ContainerStarted","Data":"ad19d4e4c7bd6b02d39cff8ecdf9c988bf75fdee81819d66b50c36c59a58eb13"} Jul 08 18:50:54.587402 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:54.587050 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-658c95c5dd-svgvn" Jul 08 18:50:54.588480 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:54.588453 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-plb8f" event={"ID":"dbed1f82-e4b6-4cdc-a68c-40dcf2448650","Type":"ContainerStarted","Data":"f0ac30a18f981a3aca2852ba948fd294ab83ef9670ce5476c39f86d6a36ba052"} Jul 08 18:50:54.588606 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:54.588513 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-plb8f" Jul 08 18:50:54.589942 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:54.589923 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-qcc7s" event={"ID":"70b85c5d-9a96-4b41-a062-8de6a4809609","Type":"ContainerStarted","Data":"b7f0f3ea6d5037ec069d34c369581d01074d59d8d13b70f8046894d0a5887666"} Jul 08 18:50:54.590055 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:54.590044 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-qcc7s" Jul 08 18:50:54.606098 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:54.606056 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-658c95c5dd-svgvn" podStartSLOduration=2.489570593 podStartE2EDuration="4.606043023s" totalImagesPullingTime="2.11647243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:50:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:50:54.605184669 +0000 UTC m=+665.442512405" watchObservedRunningTime="2026-07-08 18:50:54.606043023 +0000 UTC m=+665.443370730" Jul 08 18:50:54.620141 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:54.620096 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-qcc7s" podStartSLOduration=2.464255411 podStartE2EDuration="4.620080151s" totalImagesPullingTime="2.15582474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:50:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:50:54.61958378 +0000 UTC m=+665.456911489" watchObservedRunningTime="2026-07-08 18:50:54.620080151 +0000 UTC m=+665.457407881" Jul 08 18:50:54.640832 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:54.640768 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-plb8f" podStartSLOduration=2.45500175 podStartE2EDuration="4.6407462s" totalImagesPullingTime="2.18574445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:50:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:50:54.63878735 +0000 UTC m=+665.476115088" watchObservedRunningTime="2026-07-08 18:50:54.6407462 +0000 UTC m=+665.478073911" Jul 08 18:50:58.593548 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:58.593513 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-w6zn2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": read tcp 10.134.0.2:42788->10.134.0.34:15021: read: connection reset by peer" start-of-body= Jul 08 18:50:58.593947 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:58.593592 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" podUID="926ae9db-d53a-44d9-93ec-c93c16c604b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": read tcp 10.134.0.2:42788->10.134.0.34:15021: read: connection reset by peer" Jul 08 18:50:58.610908 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:58.610882 2575 generic.go:356] "Generic (PLEG): container finished" podID="926ae9db-d53a-44d9-93ec-c93c16c604b2" containerID="9470bc15942c900ccb61a02b49df98e49f1c8e45e14b73f5a1c9b471fa44003e" exitCode=0 Jul 08 18:50:58.611013 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:58.610911 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" event={"ID":"926ae9db-d53a-44d9-93ec-c93c16c604b2","Type":"ContainerDied","Data":"9470bc15942c900ccb61a02b49df98e49f1c8e45e14b73f5a1c9b471fa44003e"} Jul 08 18:50:58.611386 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:58.611370 2575 scope.go:117] "RemoveContainer" containerID="9470bc15942c900ccb61a02b49df98e49f1c8e45e14b73f5a1c9b471fa44003e" Jul 08 18:50:58.612334 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:58.612278 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 18:50:58.612408 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:58.612382 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 18:50:58.612460 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:58.612429 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 18:50:59.617192 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:59.617158 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" event={"ID":"926ae9db-d53a-44d9-93ec-c93c16c604b2","Type":"ContainerStarted","Data":"2efeba57164f3c91b0af3d929edb9ceb0ff286cc6a5837c16693459a7f60bf5f"} Jul 08 18:50:59.728804 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:59.728764 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:50:59.729073 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:59.729048 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-w6zn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 08 18:50:59.729140 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:50:59.729098 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" podUID="926ae9db-d53a-44d9-93ec-c93c16c604b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 08 18:51:00.600034 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:00.600005 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-658c95c5dd-svgvn" Jul 08 18:51:00.728883 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:00.728846 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-w6zn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 08 18:51:00.729278 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:00.728909 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" podUID="926ae9db-d53a-44d9-93ec-c93c16c604b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 08 18:51:01.728276 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:01.728233 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-w6zn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 08 18:51:01.728495 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:01.728329 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" podUID="926ae9db-d53a-44d9-93ec-c93c16c604b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 08 18:51:02.728311 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:02.728272 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-w6zn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 08 18:51:02.728699 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:02.728346 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" podUID="926ae9db-d53a-44d9-93ec-c93c16c604b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 08 18:51:03.732036 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:03.732009 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:51:03.732504 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:03.732265 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:51:03.732930 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:03.732914 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-w6zn2" Jul 08 18:51:05.596521 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:05.596487 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-qcc7s" Jul 08 18:51:05.596980 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:05.596757 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-plb8f" Jul 08 18:51:22.678892 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:22.678859 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l"] Jul 08 18:51:22.682431 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:22.682409 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l" Jul 08 18:51:22.689179 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:22.689126 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 08 18:51:22.689179 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:22.689159 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 08 18:51:22.689439 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:22.689199 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-gwjg6\"" Jul 08 18:51:22.689439 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:22.689257 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 08 18:51:22.698610 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:22.698582 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l"] Jul 08 18:51:22.842683 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:22.842646 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/247aede6-ea5e-4f0f-9748-b37f16641220-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l\" (UID: \"247aede6-ea5e-4f0f-9748-b37f16641220\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l" Jul 08 18:51:22.842876 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:22.842694 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/247aede6-ea5e-4f0f-9748-b37f16641220-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l\" (UID: \"247aede6-ea5e-4f0f-9748-b37f16641220\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l" Jul 08 18:51:22.842876 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:22.842712 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/247aede6-ea5e-4f0f-9748-b37f16641220-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l\" (UID: \"247aede6-ea5e-4f0f-9748-b37f16641220\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l" Jul 08 18:51:22.842876 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:22.842814 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/247aede6-ea5e-4f0f-9748-b37f16641220-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l\" (UID: \"247aede6-ea5e-4f0f-9748-b37f16641220\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l" Jul 08 18:51:22.842876 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:22.842862 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn2cl\" (UniqueName: \"kubernetes.io/projected/247aede6-ea5e-4f0f-9748-b37f16641220-kube-api-access-gn2cl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l\" (UID: \"247aede6-ea5e-4f0f-9748-b37f16641220\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l" Jul 08 18:51:22.843064 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:22.842935 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/247aede6-ea5e-4f0f-9748-b37f16641220-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l\" (UID: \"247aede6-ea5e-4f0f-9748-b37f16641220\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l" Jul 08 18:51:22.943775 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:22.943690 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/247aede6-ea5e-4f0f-9748-b37f16641220-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l\" (UID: \"247aede6-ea5e-4f0f-9748-b37f16641220\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l" Jul 08 18:51:22.943775 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:22.943764 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/247aede6-ea5e-4f0f-9748-b37f16641220-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l\" (UID: \"247aede6-ea5e-4f0f-9748-b37f16641220\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l" Jul 08 18:51:22.943940 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:22.943811 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/247aede6-ea5e-4f0f-9748-b37f16641220-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l\" (UID: \"247aede6-ea5e-4f0f-9748-b37f16641220\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l" Jul 08 18:51:22.943992 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:22.943940 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/247aede6-ea5e-4f0f-9748-b37f16641220-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l\" (UID: \"247aede6-ea5e-4f0f-9748-b37f16641220\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l" Jul 08 18:51:22.944086 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:22.944006 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/247aede6-ea5e-4f0f-9748-b37f16641220-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l\" (UID: \"247aede6-ea5e-4f0f-9748-b37f16641220\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l" Jul 08 18:51:22.944086 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:22.944048 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn2cl\" (UniqueName: \"kubernetes.io/projected/247aede6-ea5e-4f0f-9748-b37f16641220-kube-api-access-gn2cl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l\" (UID: \"247aede6-ea5e-4f0f-9748-b37f16641220\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l" Jul 08 18:51:22.944196 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:22.944152 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/247aede6-ea5e-4f0f-9748-b37f16641220-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l\" (UID: \"247aede6-ea5e-4f0f-9748-b37f16641220\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l" Jul 08 18:51:22.944307 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:22.944270 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/247aede6-ea5e-4f0f-9748-b37f16641220-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l\" (UID: \"247aede6-ea5e-4f0f-9748-b37f16641220\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l" Jul 08 18:51:22.944402 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:22.944379 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/247aede6-ea5e-4f0f-9748-b37f16641220-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l\" (UID: \"247aede6-ea5e-4f0f-9748-b37f16641220\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l" Jul 08 18:51:22.945815 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:22.945796 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/247aede6-ea5e-4f0f-9748-b37f16641220-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l\" (UID: \"247aede6-ea5e-4f0f-9748-b37f16641220\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l" Jul 08 18:51:22.946056 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:22.946038 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/247aede6-ea5e-4f0f-9748-b37f16641220-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l\" (UID: \"247aede6-ea5e-4f0f-9748-b37f16641220\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l" Jul 08 18:51:22.953646 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:22.953624 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn2cl\" (UniqueName: \"kubernetes.io/projected/247aede6-ea5e-4f0f-9748-b37f16641220-kube-api-access-gn2cl\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l\" (UID: \"247aede6-ea5e-4f0f-9748-b37f16641220\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l" Jul 08 18:51:22.994674 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:22.994645 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l" Jul 08 18:51:23.136332 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:23.136286 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l"] Jul 08 18:51:23.136769 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:51:23.136740 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247aede6_ea5e_4f0f_9748_b37f16641220.slice/crio-74df4c479050daccd1f1575cc3e330720087630b884d719d1558bf8ad798dd6d WatchSource:0}: Error finding container 74df4c479050daccd1f1575cc3e330720087630b884d719d1558bf8ad798dd6d: Status 404 returned error can't find the container with id 74df4c479050daccd1f1575cc3e330720087630b884d719d1558bf8ad798dd6d Jul 08 18:51:23.737649 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:23.737609 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l" event={"ID":"247aede6-ea5e-4f0f-9748-b37f16641220","Type":"ContainerStarted","Data":"74df4c479050daccd1f1575cc3e330720087630b884d719d1558bf8ad798dd6d"} Jul 08 18:51:24.744030 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:24.743941 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l" event={"ID":"247aede6-ea5e-4f0f-9748-b37f16641220","Type":"ContainerStarted","Data":"8d945a656d9d38691ec1997203019f43c862a782b74f043536a767aa269ca3a4"} Jul 08 18:51:24.763487 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:24.763440 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l" podStartSLOduration=1.520554115 podStartE2EDuration="2.763424415s" totalImagesPullingTime="1.2428703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:51:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:51:24.762195046 +0000 UTC m=+695.599522767" watchObservedRunningTime="2026-07-08 18:51:24.763424415 +0000 UTC m=+695.600752123" Jul 08 18:51:32.995555 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:32.995519 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l" Jul 08 18:51:32.996062 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:32.995604 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l" Jul 08 18:51:33.007949 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:33.007926 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l" Jul 08 18:51:33.798462 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:33.798428 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l" Jul 08 18:51:47.191306 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:47.190798 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn"] Jul 08 18:51:47.197248 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:47.197216 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn" Jul 08 18:51:47.200628 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:47.200601 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 08 18:51:47.205678 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:47.205655 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn"] Jul 08 18:51:47.245882 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:47.245846 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d677253-f719-46ba-8f87-65708ec5b045-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6lspn\" (UID: \"0d677253-f719-46ba-8f87-65708ec5b045\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn" Jul 08 18:51:47.246051 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:47.245900 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d677253-f719-46ba-8f87-65708ec5b045-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6lspn\" (UID: \"0d677253-f719-46ba-8f87-65708ec5b045\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn" Jul 08 18:51:47.246051 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:47.245976 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d677253-f719-46ba-8f87-65708ec5b045-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6lspn\" (UID: \"0d677253-f719-46ba-8f87-65708ec5b045\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn" Jul 08 18:51:47.246051 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:47.246047 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d677253-f719-46ba-8f87-65708ec5b045-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6lspn\" (UID: \"0d677253-f719-46ba-8f87-65708ec5b045\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn" Jul 08 18:51:47.246199 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:47.246080 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d677253-f719-46ba-8f87-65708ec5b045-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6lspn\" (UID: \"0d677253-f719-46ba-8f87-65708ec5b045\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn" Jul 08 18:51:47.246199 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:47.246106 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2p9w\" (UniqueName: \"kubernetes.io/projected/0d677253-f719-46ba-8f87-65708ec5b045-kube-api-access-n2p9w\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6lspn\" (UID: \"0d677253-f719-46ba-8f87-65708ec5b045\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn" Jul 08 18:51:47.347219 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:47.347189 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d677253-f719-46ba-8f87-65708ec5b045-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6lspn\" (UID: \"0d677253-f719-46ba-8f87-65708ec5b045\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn" Jul 08 18:51:47.347418 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:47.347246 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d677253-f719-46ba-8f87-65708ec5b045-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6lspn\" (UID: \"0d677253-f719-46ba-8f87-65708ec5b045\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn" Jul 08 18:51:47.347418 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:47.347363 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d677253-f719-46ba-8f87-65708ec5b045-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6lspn\" (UID: \"0d677253-f719-46ba-8f87-65708ec5b045\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn" Jul 08 18:51:47.347418 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:47.347414 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d677253-f719-46ba-8f87-65708ec5b045-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6lspn\" (UID: \"0d677253-f719-46ba-8f87-65708ec5b045\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn" Jul 08 18:51:47.347559 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:47.347448 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d677253-f719-46ba-8f87-65708ec5b045-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6lspn\" (UID: \"0d677253-f719-46ba-8f87-65708ec5b045\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn" Jul 08 18:51:47.347559 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:47.347473 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2p9w\" (UniqueName: \"kubernetes.io/projected/0d677253-f719-46ba-8f87-65708ec5b045-kube-api-access-n2p9w\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6lspn\" (UID: \"0d677253-f719-46ba-8f87-65708ec5b045\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn" Jul 08 18:51:47.347719 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:47.347696 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d677253-f719-46ba-8f87-65708ec5b045-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6lspn\" (UID: \"0d677253-f719-46ba-8f87-65708ec5b045\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn" Jul 08 18:51:47.347874 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:47.347768 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d677253-f719-46ba-8f87-65708ec5b045-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6lspn\" (UID: \"0d677253-f719-46ba-8f87-65708ec5b045\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn" Jul 08 18:51:47.347939 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:47.347809 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d677253-f719-46ba-8f87-65708ec5b045-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6lspn\" (UID: \"0d677253-f719-46ba-8f87-65708ec5b045\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn" Jul 08 18:51:47.349899 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:47.349879 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d677253-f719-46ba-8f87-65708ec5b045-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6lspn\" (UID: \"0d677253-f719-46ba-8f87-65708ec5b045\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn" Jul 08 18:51:47.349969 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:47.349904 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d677253-f719-46ba-8f87-65708ec5b045-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6lspn\" (UID: \"0d677253-f719-46ba-8f87-65708ec5b045\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn" Jul 08 18:51:47.358023 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:47.358003 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2p9w\" (UniqueName: \"kubernetes.io/projected/0d677253-f719-46ba-8f87-65708ec5b045-kube-api-access-n2p9w\") pod \"e2e-trlp-test-simulated-kserve-598466548d-6lspn\" (UID: \"0d677253-f719-46ba-8f87-65708ec5b045\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn" Jul 08 18:51:47.510573 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:47.510482 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn" Jul 08 18:51:47.647805 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:47.647758 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn"] Jul 08 18:51:47.650429 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:51:47.650403 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d677253_f719_46ba_8f87_65708ec5b045.slice/crio-417ab0059a4e59e8c9fde6c8ac7f8f2683c57ef5a9a1f02599869e03043f54bf WatchSource:0}: Error finding container 417ab0059a4e59e8c9fde6c8ac7f8f2683c57ef5a9a1f02599869e03043f54bf: Status 404 returned error can't find the container with id 417ab0059a4e59e8c9fde6c8ac7f8f2683c57ef5a9a1f02599869e03043f54bf Jul 08 18:51:47.851865 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:47.851835 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn" event={"ID":"0d677253-f719-46ba-8f87-65708ec5b045","Type":"ContainerStarted","Data":"417ab0059a4e59e8c9fde6c8ac7f8f2683c57ef5a9a1f02599869e03043f54bf"} Jul 08 18:51:48.858518 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:48.858483 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn" event={"ID":"0d677253-f719-46ba-8f87-65708ec5b045","Type":"ContainerStarted","Data":"f27e5ff846577f58823b2263196740fae063ae00f3b57624d47b92e79baec078"} Jul 08 18:51:48.884942 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:48.884883 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn" podStartSLOduration=1.7127438289999999 podStartE2EDuration="1.884867041s" totalImagesPullingTime="172.123212ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:51:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:51:48.882252543 +0000 UTC m=+719.719580277" watchObservedRunningTime="2026-07-08 18:51:48.884867041 +0000 UTC m=+719.722194748" Jul 08 18:51:57.511128 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:57.511069 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn" Jul 08 18:51:57.511554 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:57.511149 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn" Jul 08 18:51:57.524340 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:57.524317 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn" Jul 08 18:51:57.917452 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:51:57.917416 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-6lspn" Jul 08 18:52:13.188723 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:13.188682 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp"] Jul 08 18:52:13.194233 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:13.194206 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp" Jul 08 18:52:13.198019 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:13.197992 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 08 18:52:13.208155 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:13.208129 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp"] Jul 08 18:52:13.274667 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:13.274627 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c183697-cbdd-4a2c-8ee6-b791aa6d7582-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6krp\" (UID: \"4c183697-cbdd-4a2c-8ee6-b791aa6d7582\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp" Jul 08 18:52:13.274667 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:13.274671 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c183697-cbdd-4a2c-8ee6-b791aa6d7582-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6krp\" (UID: \"4c183697-cbdd-4a2c-8ee6-b791aa6d7582\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp" Jul 08 18:52:13.274876 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:13.274730 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c183697-cbdd-4a2c-8ee6-b791aa6d7582-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6krp\" (UID: \"4c183697-cbdd-4a2c-8ee6-b791aa6d7582\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp" Jul 08 18:52:13.274876 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:13.274819 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c183697-cbdd-4a2c-8ee6-b791aa6d7582-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6krp\" (UID: \"4c183697-cbdd-4a2c-8ee6-b791aa6d7582\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp" Jul 08 18:52:13.274876 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:13.274840 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mvlv\" (UniqueName: \"kubernetes.io/projected/4c183697-cbdd-4a2c-8ee6-b791aa6d7582-kube-api-access-8mvlv\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6krp\" (UID: \"4c183697-cbdd-4a2c-8ee6-b791aa6d7582\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp" Jul 08 18:52:13.274967 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:13.274894 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c183697-cbdd-4a2c-8ee6-b791aa6d7582-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6krp\" (UID: \"4c183697-cbdd-4a2c-8ee6-b791aa6d7582\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp" Jul 08 18:52:13.375567 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:13.375530 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c183697-cbdd-4a2c-8ee6-b791aa6d7582-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6krp\" (UID: \"4c183697-cbdd-4a2c-8ee6-b791aa6d7582\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp" Jul 08 18:52:13.375775 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:13.375586 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c183697-cbdd-4a2c-8ee6-b791aa6d7582-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6krp\" (UID: \"4c183697-cbdd-4a2c-8ee6-b791aa6d7582\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp" Jul 08 18:52:13.375775 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:13.375616 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c183697-cbdd-4a2c-8ee6-b791aa6d7582-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6krp\" (UID: \"4c183697-cbdd-4a2c-8ee6-b791aa6d7582\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp" Jul 08 18:52:13.375775 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:13.375645 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c183697-cbdd-4a2c-8ee6-b791aa6d7582-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6krp\" (UID: \"4c183697-cbdd-4a2c-8ee6-b791aa6d7582\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp" Jul 08 18:52:13.375775 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:13.375719 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c183697-cbdd-4a2c-8ee6-b791aa6d7582-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6krp\" (UID: \"4c183697-cbdd-4a2c-8ee6-b791aa6d7582\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp" Jul 08 18:52:13.375775 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:13.375747 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mvlv\" (UniqueName: \"kubernetes.io/projected/4c183697-cbdd-4a2c-8ee6-b791aa6d7582-kube-api-access-8mvlv\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6krp\" (UID: \"4c183697-cbdd-4a2c-8ee6-b791aa6d7582\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp" Jul 08 18:52:13.376067 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:13.376043 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c183697-cbdd-4a2c-8ee6-b791aa6d7582-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6krp\" (UID: \"4c183697-cbdd-4a2c-8ee6-b791aa6d7582\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp" Jul 08 18:52:13.376115 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:13.376084 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c183697-cbdd-4a2c-8ee6-b791aa6d7582-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6krp\" (UID: \"4c183697-cbdd-4a2c-8ee6-b791aa6d7582\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp" Jul 08 18:52:13.376151 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:13.376128 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c183697-cbdd-4a2c-8ee6-b791aa6d7582-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6krp\" (UID: \"4c183697-cbdd-4a2c-8ee6-b791aa6d7582\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp" Jul 08 18:52:13.377871 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:13.377849 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c183697-cbdd-4a2c-8ee6-b791aa6d7582-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6krp\" (UID: \"4c183697-cbdd-4a2c-8ee6-b791aa6d7582\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp" Jul 08 18:52:13.378117 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:13.378100 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c183697-cbdd-4a2c-8ee6-b791aa6d7582-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6krp\" (UID: \"4c183697-cbdd-4a2c-8ee6-b791aa6d7582\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp" Jul 08 18:52:13.387026 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:13.386996 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mvlv\" (UniqueName: \"kubernetes.io/projected/4c183697-cbdd-4a2c-8ee6-b791aa6d7582-kube-api-access-8mvlv\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-r6krp\" (UID: \"4c183697-cbdd-4a2c-8ee6-b791aa6d7582\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp" Jul 08 18:52:13.505753 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:13.505666 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp" Jul 08 18:52:13.644854 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:13.644826 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp"] Jul 08 18:52:13.647616 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:52:13.647581 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c183697_cbdd_4a2c_8ee6_b791aa6d7582.slice/crio-7b91ff7e4ba9cf75462aa9ef43bce3cc6dc732bfc3fa547ba7e4d917be41984b WatchSource:0}: Error finding container 7b91ff7e4ba9cf75462aa9ef43bce3cc6dc732bfc3fa547ba7e4d917be41984b: Status 404 returned error can't find the container with id 7b91ff7e4ba9cf75462aa9ef43bce3cc6dc732bfc3fa547ba7e4d917be41984b Jul 08 18:52:13.980825 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:13.980790 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp" event={"ID":"4c183697-cbdd-4a2c-8ee6-b791aa6d7582","Type":"ContainerStarted","Data":"309a9119e35e3d940459359bf3c32fb621dd56ad16e8930fe3e446b47fad1dcf"} Jul 08 18:52:13.980982 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:13.980845 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp" event={"ID":"4c183697-cbdd-4a2c-8ee6-b791aa6d7582","Type":"ContainerStarted","Data":"7b91ff7e4ba9cf75462aa9ef43bce3cc6dc732bfc3fa547ba7e4d917be41984b"} Jul 08 18:52:14.003091 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:14.003039 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp" podStartSLOduration=0.796198778 podStartE2EDuration="1.003025428s" totalImagesPullingTime="206.82665ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:52:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:52:14.000012129 +0000 UTC m=+744.837339856" watchObservedRunningTime="2026-07-08 18:52:14.003025428 +0000 UTC m=+744.840353135" Jul 08 18:52:23.506395 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:23.506358 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp" Jul 08 18:52:23.506872 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:23.506429 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp" Jul 08 18:52:23.518859 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:23.518833 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp" Jul 08 18:52:24.049364 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:52:24.049325 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-r6krp" Jul 08 18:54:34.920309 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:34.920255 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6449898b5-8dc9t"] Jul 08 18:54:34.920978 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:34.920504 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-6449898b5-8dc9t" podUID="ecd2a433-4cca-4d9f-96db-cca824e24e64" containerName="manager" containerID="cri-o://9f3d9f9422e875bd66fb6a16d2f0c3eef2f109879e6b5f3f319c3deca95a2e02" gracePeriod=30 Jul 08 18:54:35.174583 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:35.174511 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6449898b5-8dc9t" Jul 08 18:54:35.324892 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:35.324861 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9hqm\" (UniqueName: \"kubernetes.io/projected/ecd2a433-4cca-4d9f-96db-cca824e24e64-kube-api-access-v9hqm\") pod \"ecd2a433-4cca-4d9f-96db-cca824e24e64\" (UID: \"ecd2a433-4cca-4d9f-96db-cca824e24e64\") " Jul 08 18:54:35.325063 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:35.324917 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecd2a433-4cca-4d9f-96db-cca824e24e64-webhook-cert\") pod \"ecd2a433-4cca-4d9f-96db-cca824e24e64\" (UID: \"ecd2a433-4cca-4d9f-96db-cca824e24e64\") " Jul 08 18:54:35.326955 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:35.326923 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecd2a433-4cca-4d9f-96db-cca824e24e64-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ecd2a433-4cca-4d9f-96db-cca824e24e64" (UID: "ecd2a433-4cca-4d9f-96db-cca824e24e64"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:54:35.327063 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:35.326974 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd2a433-4cca-4d9f-96db-cca824e24e64-kube-api-access-v9hqm" (OuterVolumeSpecName: "kube-api-access-v9hqm") pod "ecd2a433-4cca-4d9f-96db-cca824e24e64" (UID: "ecd2a433-4cca-4d9f-96db-cca824e24e64"). InnerVolumeSpecName "kube-api-access-v9hqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:54:35.426552 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:35.426464 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9hqm\" (UniqueName: \"kubernetes.io/projected/ecd2a433-4cca-4d9f-96db-cca824e24e64-kube-api-access-v9hqm\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 18:54:35.426552 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:35.426492 2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ecd2a433-4cca-4d9f-96db-cca824e24e64-webhook-cert\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 18:54:35.664912 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:35.664881 2575 generic.go:356] "Generic (PLEG): container finished" podID="ecd2a433-4cca-4d9f-96db-cca824e24e64" containerID="9f3d9f9422e875bd66fb6a16d2f0c3eef2f109879e6b5f3f319c3deca95a2e02" exitCode=0 Jul 08 18:54:35.665082 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:35.664926 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6449898b5-8dc9t" Jul 08 18:54:35.665082 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:35.664930 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6449898b5-8dc9t" event={"ID":"ecd2a433-4cca-4d9f-96db-cca824e24e64","Type":"ContainerDied","Data":"9f3d9f9422e875bd66fb6a16d2f0c3eef2f109879e6b5f3f319c3deca95a2e02"} Jul 08 18:54:35.665082 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:35.665029 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6449898b5-8dc9t" event={"ID":"ecd2a433-4cca-4d9f-96db-cca824e24e64","Type":"ContainerDied","Data":"5f33b45998b742dff949a32ca3af1a7862fe33d7db73caef44a190488612da21"} Jul 08 18:54:35.665082 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:35.665049 2575 scope.go:117] "RemoveContainer" containerID="9f3d9f9422e875bd66fb6a16d2f0c3eef2f109879e6b5f3f319c3deca95a2e02" Jul 08 18:54:35.674119 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:35.674104 2575 scope.go:117] "RemoveContainer" containerID="9f3d9f9422e875bd66fb6a16d2f0c3eef2f109879e6b5f3f319c3deca95a2e02" Jul 08 18:54:35.674370 ip-10-0-131-102 kubenswrapper[2575]: E0708 18:54:35.674351 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f3d9f9422e875bd66fb6a16d2f0c3eef2f109879e6b5f3f319c3deca95a2e02\": container with ID starting with 9f3d9f9422e875bd66fb6a16d2f0c3eef2f109879e6b5f3f319c3deca95a2e02 not found: ID does not exist" containerID="9f3d9f9422e875bd66fb6a16d2f0c3eef2f109879e6b5f3f319c3deca95a2e02" Jul 08 18:54:35.674441 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:35.674378 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f3d9f9422e875bd66fb6a16d2f0c3eef2f109879e6b5f3f319c3deca95a2e02"} err="failed to get container status \"9f3d9f9422e875bd66fb6a16d2f0c3eef2f109879e6b5f3f319c3deca95a2e02\": rpc error: code = NotFound desc = could not find container \"9f3d9f9422e875bd66fb6a16d2f0c3eef2f109879e6b5f3f319c3deca95a2e02\": container with ID starting with 9f3d9f9422e875bd66fb6a16d2f0c3eef2f109879e6b5f3f319c3deca95a2e02 not found: ID does not exist" Jul 08 18:54:35.686696 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:35.686637 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6449898b5-8dc9t"] Jul 08 18:54:35.688683 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:35.688664 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6449898b5-8dc9t"] Jul 08 18:54:35.779769 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:35.779734 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecd2a433-4cca-4d9f-96db-cca824e24e64" path="/var/lib/kubelet/pods/ecd2a433-4cca-4d9f-96db-cca824e24e64/volumes" Jul 08 18:54:36.529463 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:36.529428 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6449898b5-4p9q2"] Jul 08 18:54:36.529828 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:36.529797 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecd2a433-4cca-4d9f-96db-cca824e24e64" containerName="manager" Jul 08 18:54:36.529828 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:36.529808 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecd2a433-4cca-4d9f-96db-cca824e24e64" containerName="manager" Jul 08 18:54:36.529897 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:36.529874 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecd2a433-4cca-4d9f-96db-cca824e24e64" containerName="manager" Jul 08 18:54:36.534155 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:36.534133 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6449898b5-4p9q2" Jul 08 18:54:36.538198 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:36.538176 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 08 18:54:36.538350 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:36.538176 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-7drrl\"" Jul 08 18:54:36.538350 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:36.538176 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 08 18:54:36.544211 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:36.544178 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6449898b5-4p9q2"] Jul 08 18:54:36.637221 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:36.637182 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1aa899cd-b902-47df-a8e7-cc85e05e1886-webhook-cert\") pod \"maas-controller-6449898b5-4p9q2\" (UID: \"1aa899cd-b902-47df-a8e7-cc85e05e1886\") " pod="opendatahub/maas-controller-6449898b5-4p9q2" Jul 08 18:54:36.637221 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:36.637225 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg7mt\" (UniqueName: \"kubernetes.io/projected/1aa899cd-b902-47df-a8e7-cc85e05e1886-kube-api-access-lg7mt\") pod \"maas-controller-6449898b5-4p9q2\" (UID: \"1aa899cd-b902-47df-a8e7-cc85e05e1886\") " pod="opendatahub/maas-controller-6449898b5-4p9q2" Jul 08 18:54:36.737862 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:36.737828 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1aa899cd-b902-47df-a8e7-cc85e05e1886-webhook-cert\") pod \"maas-controller-6449898b5-4p9q2\" (UID: \"1aa899cd-b902-47df-a8e7-cc85e05e1886\") " pod="opendatahub/maas-controller-6449898b5-4p9q2" Jul 08 18:54:36.737862 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:36.737866 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg7mt\" (UniqueName: \"kubernetes.io/projected/1aa899cd-b902-47df-a8e7-cc85e05e1886-kube-api-access-lg7mt\") pod \"maas-controller-6449898b5-4p9q2\" (UID: \"1aa899cd-b902-47df-a8e7-cc85e05e1886\") " pod="opendatahub/maas-controller-6449898b5-4p9q2" Jul 08 18:54:36.740380 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:36.740349 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1aa899cd-b902-47df-a8e7-cc85e05e1886-webhook-cert\") pod \"maas-controller-6449898b5-4p9q2\" (UID: \"1aa899cd-b902-47df-a8e7-cc85e05e1886\") " pod="opendatahub/maas-controller-6449898b5-4p9q2" Jul 08 18:54:36.746572 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:36.746548 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg7mt\" (UniqueName: \"kubernetes.io/projected/1aa899cd-b902-47df-a8e7-cc85e05e1886-kube-api-access-lg7mt\") pod \"maas-controller-6449898b5-4p9q2\" (UID: \"1aa899cd-b902-47df-a8e7-cc85e05e1886\") " pod="opendatahub/maas-controller-6449898b5-4p9q2" Jul 08 18:54:36.844996 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:36.844908 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6449898b5-4p9q2" Jul 08 18:54:36.975739 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:36.975709 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6449898b5-4p9q2"] Jul 08 18:54:36.977738 ip-10-0-131-102 kubenswrapper[2575]: W0708 18:54:36.977712 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa899cd_b902_47df_a8e7_cc85e05e1886.slice/crio-59bdd2e2b18c5798f95add082c87c92196cc7511b1eb81aae3ca04cf33c6b1de WatchSource:0}: Error finding container 59bdd2e2b18c5798f95add082c87c92196cc7511b1eb81aae3ca04cf33c6b1de: Status 404 returned error can't find the container with id 59bdd2e2b18c5798f95add082c87c92196cc7511b1eb81aae3ca04cf33c6b1de Jul 08 18:54:37.678013 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:37.677972 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6449898b5-4p9q2" event={"ID":"1aa899cd-b902-47df-a8e7-cc85e05e1886","Type":"ContainerStarted","Data":"59bdd2e2b18c5798f95add082c87c92196cc7511b1eb81aae3ca04cf33c6b1de"} Jul 08 18:54:38.685635 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:38.685599 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6449898b5-4p9q2" event={"ID":"1aa899cd-b902-47df-a8e7-cc85e05e1886","Type":"ContainerStarted","Data":"42fc30a8ca06feb2e979e5822b909b9af37b21b94e446979884678ddf4f1730e"} Jul 08 18:54:38.686049 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:38.685739 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6449898b5-4p9q2" Jul 08 18:54:38.710027 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:38.709974 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6449898b5-4p9q2" podStartSLOduration=1.343094 podStartE2EDuration="2.709960068s" totalImagesPullingTime="1.366866068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:54:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:54:38.708539665 +0000 UTC m=+889.545867373" watchObservedRunningTime="2026-07-08 18:54:38.709960068 +0000 UTC m=+889.547287775" Jul 08 18:54:49.697686 ip-10-0-131-102 kubenswrapper[2575]: I0708 18:54:49.697662 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6449898b5-4p9q2" Jul 08 19:06:50.750669 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.750621 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf"] Jul 08 19:06:50.754624 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.754601 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:50.757500 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.757473 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-574a6850-openshift-default-dockercfg-4htbt\"" Jul 08 19:06:50.776172 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.776138 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf"] Jul 08 19:06:50.860132 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.860071 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-workload-socket\") pod \"e2e-ait-574a6850-openshift-default-568d7b579-fdstf\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:50.860132 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.860123 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a2d2098-34fe-4627-990d-7d7abac54a81-istio-token\") pod \"e2e-ait-574a6850-openshift-default-568d7b579-fdstf\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:50.860665 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.860190 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a2d2098-34fe-4627-990d-7d7abac54a81-istiod-ca-cert\") pod \"e2e-ait-574a6850-openshift-default-568d7b579-fdstf\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:50.860665 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.860217 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-istio-envoy\") pod \"e2e-ait-574a6850-openshift-default-568d7b579-fdstf\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:50.860665 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.860287 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-istio-data\") pod \"e2e-ait-574a6850-openshift-default-568d7b579-fdstf\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:50.860665 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.860417 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-workload-certs\") pod \"e2e-ait-574a6850-openshift-default-568d7b579-fdstf\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:50.860665 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.860447 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a2d2098-34fe-4627-990d-7d7abac54a81-istio-podinfo\") pod \"e2e-ait-574a6850-openshift-default-568d7b579-fdstf\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:50.860665 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.860476 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-credential-socket\") pod \"e2e-ait-574a6850-openshift-default-568d7b579-fdstf\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:50.860665 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.860505 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm88p\" (UniqueName: \"kubernetes.io/projected/9a2d2098-34fe-4627-990d-7d7abac54a81-kube-api-access-nm88p\") pod \"e2e-ait-574a6850-openshift-default-568d7b579-fdstf\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:50.961597 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.961557 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a2d2098-34fe-4627-990d-7d7abac54a81-istiod-ca-cert\") pod \"e2e-ait-574a6850-openshift-default-568d7b579-fdstf\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:50.961597 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.961602 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-istio-envoy\") pod \"e2e-ait-574a6850-openshift-default-568d7b579-fdstf\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:50.961834 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.961663 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-istio-data\") pod \"e2e-ait-574a6850-openshift-default-568d7b579-fdstf\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:50.961834 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.961798 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-workload-certs\") pod \"e2e-ait-574a6850-openshift-default-568d7b579-fdstf\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:50.961938 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.961833 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a2d2098-34fe-4627-990d-7d7abac54a81-istio-podinfo\") pod \"e2e-ait-574a6850-openshift-default-568d7b579-fdstf\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:50.961938 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.961865 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-credential-socket\") pod \"e2e-ait-574a6850-openshift-default-568d7b579-fdstf\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:50.961938 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.961895 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm88p\" (UniqueName: \"kubernetes.io/projected/9a2d2098-34fe-4627-990d-7d7abac54a81-kube-api-access-nm88p\") pod \"e2e-ait-574a6850-openshift-default-568d7b579-fdstf\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:50.961938 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.961931 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-workload-socket\") pod \"e2e-ait-574a6850-openshift-default-568d7b579-fdstf\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:50.962135 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.961954 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a2d2098-34fe-4627-990d-7d7abac54a81-istio-token\") pod \"e2e-ait-574a6850-openshift-default-568d7b579-fdstf\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:50.962135 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.962038 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-istio-data\") pod \"e2e-ait-574a6850-openshift-default-568d7b579-fdstf\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:50.962338 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.962264 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-credential-socket\") pod \"e2e-ait-574a6850-openshift-default-568d7b579-fdstf\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:50.962338 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.962276 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-workload-certs\") pod \"e2e-ait-574a6850-openshift-default-568d7b579-fdstf\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:50.962528 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.962394 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-workload-socket\") pod \"e2e-ait-574a6850-openshift-default-568d7b579-fdstf\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:50.962528 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.962440 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a2d2098-34fe-4627-990d-7d7abac54a81-istiod-ca-cert\") pod \"e2e-ait-574a6850-openshift-default-568d7b579-fdstf\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:50.963945 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.963924 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-istio-envoy\") pod \"e2e-ait-574a6850-openshift-default-568d7b579-fdstf\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:50.964182 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.964163 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a2d2098-34fe-4627-990d-7d7abac54a81-istio-podinfo\") pod \"e2e-ait-574a6850-openshift-default-568d7b579-fdstf\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:50.970341 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.970283 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a2d2098-34fe-4627-990d-7d7abac54a81-istio-token\") pod \"e2e-ait-574a6850-openshift-default-568d7b579-fdstf\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:50.970436 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:50.970358 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm88p\" (UniqueName: \"kubernetes.io/projected/9a2d2098-34fe-4627-990d-7d7abac54a81-kube-api-access-nm88p\") pod \"e2e-ait-574a6850-openshift-default-568d7b579-fdstf\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:51.066216 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:51.066108 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:51.205162 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:51.205136 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf"] Jul 08 19:06:51.207492 ip-10-0-131-102 kubenswrapper[2575]: W0708 19:06:51.207464 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a2d2098_34fe_4627_990d_7d7abac54a81.slice/crio-bb70f654d3b777cd112eef132d1ced3449dbd3bfc8fb504227c075d8db914838 WatchSource:0}: Error finding container bb70f654d3b777cd112eef132d1ced3449dbd3bfc8fb504227c075d8db914838: Status 404 returned error can't find the container with id bb70f654d3b777cd112eef132d1ced3449dbd3bfc8fb504227c075d8db914838 Jul 08 19:06:51.209930 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:51.209892 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:06:51.210023 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:51.209966 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:06:51.210023 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:51.210017 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:06:51.236562 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:51.236531 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" event={"ID":"9a2d2098-34fe-4627-990d-7d7abac54a81","Type":"ContainerStarted","Data":"bb70f654d3b777cd112eef132d1ced3449dbd3bfc8fb504227c075d8db914838"} Jul 08 19:06:52.243455 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:52.243413 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" event={"ID":"9a2d2098-34fe-4627-990d-7d7abac54a81","Type":"ContainerStarted","Data":"d1e9ed112a209387822b7db7ed2d582f109c709b4372d4b71c23879f26617500"} Jul 08 19:06:52.272106 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:52.272046 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" podStartSLOduration=2.272028429 podStartE2EDuration="2.272028429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:06:52.267906791 +0000 UTC m=+1623.105234501" watchObservedRunningTime="2026-07-08 19:06:52.272028429 +0000 UTC m=+1623.109356136" Jul 08 19:06:53.066970 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:53.066930 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:53.071895 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:53.071866 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:53.248988 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:53.248956 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:53.250116 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:53.250095 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:06:57.244864 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:57.244815 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf"] Jul 08 19:06:57.245382 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:06:57.245058 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" podUID="9a2d2098-34fe-4627-990d-7d7abac54a81" containerName="istio-proxy" containerID="cri-o://d1e9ed112a209387822b7db7ed2d582f109c709b4372d4b71c23879f26617500" gracePeriod=30 Jul 08 19:07:03.993064 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:03.993036 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:07:04.101244 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.101136 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-istio-data\") pod \"9a2d2098-34fe-4627-990d-7d7abac54a81\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " Jul 08 19:07:04.101244 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.101178 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-credential-socket\") pod \"9a2d2098-34fe-4627-990d-7d7abac54a81\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " Jul 08 19:07:04.101244 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.101202 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-workload-socket\") pod \"9a2d2098-34fe-4627-990d-7d7abac54a81\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " Jul 08 19:07:04.101244 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.101223 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-istio-envoy\") pod \"9a2d2098-34fe-4627-990d-7d7abac54a81\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " Jul 08 19:07:04.101244 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.101249 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-workload-certs\") pod \"9a2d2098-34fe-4627-990d-7d7abac54a81\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " Jul 08 19:07:04.101683 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.101330 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a2d2098-34fe-4627-990d-7d7abac54a81-istiod-ca-cert\") pod \"9a2d2098-34fe-4627-990d-7d7abac54a81\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " Jul 08 19:07:04.101683 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.101369 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a2d2098-34fe-4627-990d-7d7abac54a81-istio-podinfo\") pod \"9a2d2098-34fe-4627-990d-7d7abac54a81\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " Jul 08 19:07:04.101683 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.101409 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nm88p\" (UniqueName: \"kubernetes.io/projected/9a2d2098-34fe-4627-990d-7d7abac54a81-kube-api-access-nm88p\") pod \"9a2d2098-34fe-4627-990d-7d7abac54a81\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " Jul 08 19:07:04.101683 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.101469 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a2d2098-34fe-4627-990d-7d7abac54a81-istio-token\") pod \"9a2d2098-34fe-4627-990d-7d7abac54a81\" (UID: \"9a2d2098-34fe-4627-990d-7d7abac54a81\") " Jul 08 19:07:04.101683 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.101524 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9a2d2098-34fe-4627-990d-7d7abac54a81" (UID: "9a2d2098-34fe-4627-990d-7d7abac54a81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:07:04.101683 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.101577 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9a2d2098-34fe-4627-990d-7d7abac54a81" (UID: "9a2d2098-34fe-4627-990d-7d7abac54a81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:07:04.101967 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.101829 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-credential-socket\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:07:04.101967 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.101853 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-workload-socket\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:07:04.101967 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.101866 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9a2d2098-34fe-4627-990d-7d7abac54a81" (UID: "9a2d2098-34fe-4627-990d-7d7abac54a81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:07:04.101967 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.101902 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a2d2098-34fe-4627-990d-7d7abac54a81-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9a2d2098-34fe-4627-990d-7d7abac54a81" (UID: "9a2d2098-34fe-4627-990d-7d7abac54a81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:07:04.102161 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.102124 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-istio-data" (OuterVolumeSpecName: "istio-data") pod "9a2d2098-34fe-4627-990d-7d7abac54a81" (UID: "9a2d2098-34fe-4627-990d-7d7abac54a81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:07:04.104577 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.104544 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a2d2098-34fe-4627-990d-7d7abac54a81-istio-token" (OuterVolumeSpecName: "istio-token") pod "9a2d2098-34fe-4627-990d-7d7abac54a81" (UID: "9a2d2098-34fe-4627-990d-7d7abac54a81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:07:04.104577 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.104564 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9a2d2098-34fe-4627-990d-7d7abac54a81" (UID: "9a2d2098-34fe-4627-990d-7d7abac54a81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:07:04.104577 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.104564 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a2d2098-34fe-4627-990d-7d7abac54a81-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9a2d2098-34fe-4627-990d-7d7abac54a81" (UID: "9a2d2098-34fe-4627-990d-7d7abac54a81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 19:07:04.104801 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.104742 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a2d2098-34fe-4627-990d-7d7abac54a81-kube-api-access-nm88p" (OuterVolumeSpecName: "kube-api-access-nm88p") pod "9a2d2098-34fe-4627-990d-7d7abac54a81" (UID: "9a2d2098-34fe-4627-990d-7d7abac54a81"). InnerVolumeSpecName "kube-api-access-nm88p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:07:04.202365 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.202326 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-istio-data\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:07:04.202365 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.202358 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-istio-envoy\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:07:04.202365 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.202367 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a2d2098-34fe-4627-990d-7d7abac54a81-workload-certs\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:07:04.202365 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.202376 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a2d2098-34fe-4627-990d-7d7abac54a81-istiod-ca-cert\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:07:04.202641 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.202384 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a2d2098-34fe-4627-990d-7d7abac54a81-istio-podinfo\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:07:04.202641 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.202393 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm88p\" (UniqueName: \"kubernetes.io/projected/9a2d2098-34fe-4627-990d-7d7abac54a81-kube-api-access-nm88p\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:07:04.202641 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.202403 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a2d2098-34fe-4627-990d-7d7abac54a81-istio-token\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:07:04.305610 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.305571 2575 generic.go:356] "Generic (PLEG): container finished" podID="9a2d2098-34fe-4627-990d-7d7abac54a81" containerID="d1e9ed112a209387822b7db7ed2d582f109c709b4372d4b71c23879f26617500" exitCode=0 Jul 08 19:07:04.305774 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.305620 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" Jul 08 19:07:04.305774 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.305657 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" event={"ID":"9a2d2098-34fe-4627-990d-7d7abac54a81","Type":"ContainerDied","Data":"d1e9ed112a209387822b7db7ed2d582f109c709b4372d4b71c23879f26617500"} Jul 08 19:07:04.305774 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.305713 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf" event={"ID":"9a2d2098-34fe-4627-990d-7d7abac54a81","Type":"ContainerDied","Data":"bb70f654d3b777cd112eef132d1ced3449dbd3bfc8fb504227c075d8db914838"} Jul 08 19:07:04.305774 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.305736 2575 scope.go:117] "RemoveContainer" containerID="d1e9ed112a209387822b7db7ed2d582f109c709b4372d4b71c23879f26617500" Jul 08 19:07:04.315061 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.315041 2575 scope.go:117] "RemoveContainer" containerID="d1e9ed112a209387822b7db7ed2d582f109c709b4372d4b71c23879f26617500" Jul 08 19:07:04.315365 ip-10-0-131-102 kubenswrapper[2575]: E0708 19:07:04.315342 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1e9ed112a209387822b7db7ed2d582f109c709b4372d4b71c23879f26617500\": container with ID starting with d1e9ed112a209387822b7db7ed2d582f109c709b4372d4b71c23879f26617500 not found: ID does not exist" containerID="d1e9ed112a209387822b7db7ed2d582f109c709b4372d4b71c23879f26617500" Jul 08 19:07:04.315431 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.315373 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1e9ed112a209387822b7db7ed2d582f109c709b4372d4b71c23879f26617500"} err="failed to get container status \"d1e9ed112a209387822b7db7ed2d582f109c709b4372d4b71c23879f26617500\": rpc error: code = NotFound desc = could not find container \"d1e9ed112a209387822b7db7ed2d582f109c709b4372d4b71c23879f26617500\": container with ID starting with d1e9ed112a209387822b7db7ed2d582f109c709b4372d4b71c23879f26617500 not found: ID does not exist" Jul 08 19:07:04.331745 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.331714 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf"] Jul 08 19:07:04.336089 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.336058 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-574a6850-openshift-default-568d7b579-fdstf"] Jul 08 19:07:04.849465 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.849432 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2"] Jul 08 19:07:04.850036 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.850016 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a2d2098-34fe-4627-990d-7d7abac54a81" containerName="istio-proxy" Jul 08 19:07:04.850089 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.850041 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a2d2098-34fe-4627-990d-7d7abac54a81" containerName="istio-proxy" Jul 08 19:07:04.850156 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.850145 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a2d2098-34fe-4627-990d-7d7abac54a81" containerName="istio-proxy" Jul 08 19:07:04.854623 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.854603 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2" Jul 08 19:07:04.858001 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.857981 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8f5e817a-serving-cert\"" Jul 08 19:07:04.872230 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:04.872192 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2"] Jul 08 19:07:05.010943 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:05.010906 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecd644b6-fa8f-4a56-a245-21f1f37c0bbd-openshift-service-ca\") pod \"maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2\" (UID: \"ecd644b6-fa8f-4a56-a245-21f1f37c0bbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2" Jul 08 19:07:05.011367 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:05.011004 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecd644b6-fa8f-4a56-a245-21f1f37c0bbd-maas-api-tls\") pod \"maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2\" (UID: \"ecd644b6-fa8f-4a56-a245-21f1f37c0bbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2" Jul 08 19:07:05.011367 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:05.011026 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vt42\" (UniqueName: \"kubernetes.io/projected/ecd644b6-fa8f-4a56-a245-21f1f37c0bbd-kube-api-access-9vt42\") pod \"maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2\" (UID: \"ecd644b6-fa8f-4a56-a245-21f1f37c0bbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2" Jul 08 19:07:05.111697 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:05.111658 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecd644b6-fa8f-4a56-a245-21f1f37c0bbd-openshift-service-ca\") pod \"maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2\" (UID: \"ecd644b6-fa8f-4a56-a245-21f1f37c0bbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2" Jul 08 19:07:05.111871 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:05.111740 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecd644b6-fa8f-4a56-a245-21f1f37c0bbd-maas-api-tls\") pod \"maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2\" (UID: \"ecd644b6-fa8f-4a56-a245-21f1f37c0bbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2" Jul 08 19:07:05.111871 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:05.111767 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vt42\" (UniqueName: \"kubernetes.io/projected/ecd644b6-fa8f-4a56-a245-21f1f37c0bbd-kube-api-access-9vt42\") pod \"maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2\" (UID: \"ecd644b6-fa8f-4a56-a245-21f1f37c0bbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2" Jul 08 19:07:05.111945 ip-10-0-131-102 kubenswrapper[2575]: E0708 19:07:05.111894 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-serving-cert: secret "maas-api-e2e-ait-8f5e817a-serving-cert" not found Jul 08 19:07:05.111990 ip-10-0-131-102 kubenswrapper[2575]: E0708 19:07:05.111979 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecd644b6-fa8f-4a56-a245-21f1f37c0bbd-maas-api-tls podName:ecd644b6-fa8f-4a56-a245-21f1f37c0bbd nodeName:}" failed. No retries permitted until 2026-07-08 19:07:05.611959985 +0000 UTC m=+1636.449287674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ecd644b6-fa8f-4a56-a245-21f1f37c0bbd-maas-api-tls") pod "maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2" (UID: "ecd644b6-fa8f-4a56-a245-21f1f37c0bbd") : secret "maas-api-e2e-ait-8f5e817a-serving-cert" not found Jul 08 19:07:05.112478 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:05.112461 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecd644b6-fa8f-4a56-a245-21f1f37c0bbd-openshift-service-ca\") pod \"maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2\" (UID: \"ecd644b6-fa8f-4a56-a245-21f1f37c0bbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2" Jul 08 19:07:05.124624 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:05.124597 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vt42\" (UniqueName: \"kubernetes.io/projected/ecd644b6-fa8f-4a56-a245-21f1f37c0bbd-kube-api-access-9vt42\") pod \"maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2\" (UID: \"ecd644b6-fa8f-4a56-a245-21f1f37c0bbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2" Jul 08 19:07:05.616599 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:05.616565 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecd644b6-fa8f-4a56-a245-21f1f37c0bbd-maas-api-tls\") pod \"maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2\" (UID: \"ecd644b6-fa8f-4a56-a245-21f1f37c0bbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2" Jul 08 19:07:05.618968 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:05.618934 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecd644b6-fa8f-4a56-a245-21f1f37c0bbd-maas-api-tls\") pod \"maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2\" (UID: \"ecd644b6-fa8f-4a56-a245-21f1f37c0bbd\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2" Jul 08 19:07:05.765213 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:05.765172 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2" Jul 08 19:07:05.781886 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:05.781852 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a2d2098-34fe-4627-990d-7d7abac54a81" path="/var/lib/kubelet/pods/9a2d2098-34fe-4627-990d-7d7abac54a81/volumes" Jul 08 19:07:05.906109 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:05.906081 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2"] Jul 08 19:07:05.908592 ip-10-0-131-102 kubenswrapper[2575]: W0708 19:07:05.908558 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd644b6_fa8f_4a56_a245_21f1f37c0bbd.slice/crio-f490bd86b19603cf257cc61241874325aefec191e5de9de019ac8d3918f0604a WatchSource:0}: Error finding container f490bd86b19603cf257cc61241874325aefec191e5de9de019ac8d3918f0604a: Status 404 returned error can't find the container with id f490bd86b19603cf257cc61241874325aefec191e5de9de019ac8d3918f0604a Jul 08 19:07:05.910032 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:05.910006 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 19:07:06.318542 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:06.318447 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2" event={"ID":"ecd644b6-fa8f-4a56-a245-21f1f37c0bbd","Type":"ContainerStarted","Data":"f490bd86b19603cf257cc61241874325aefec191e5de9de019ac8d3918f0604a"} Jul 08 19:07:07.324660 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:07.324618 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2" event={"ID":"ecd644b6-fa8f-4a56-a245-21f1f37c0bbd","Type":"ContainerStarted","Data":"f99768b01de6b29d1fd687c8d3b5a275ab6942b52ee9bf16584e80a2ad20bef5"} Jul 08 19:07:07.325098 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:07.324682 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2" Jul 08 19:07:07.344180 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:07.344127 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2" podStartSLOduration=2.6628485790000003 podStartE2EDuration="3.344110256s" totalImagesPullingTime="681.261677ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:07:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:07:07.343190511 +0000 UTC m=+1638.180518221" watchObservedRunningTime="2026-07-08 19:07:07.344110256 +0000 UTC m=+1638.181437967" Jul 08 19:07:10.328557 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:10.328514 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2"] Jul 08 19:07:10.329035 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:10.328752 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2" podUID="ecd644b6-fa8f-4a56-a245-21f1f37c0bbd" containerName="maas-api" containerID="cri-o://f99768b01de6b29d1fd687c8d3b5a275ab6942b52ee9bf16584e80a2ad20bef5" gracePeriod=30 Jul 08 19:07:10.582575 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:10.582501 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2" Jul 08 19:07:10.661979 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:10.661942 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecd644b6-fa8f-4a56-a245-21f1f37c0bbd-maas-api-tls\") pod \"ecd644b6-fa8f-4a56-a245-21f1f37c0bbd\" (UID: \"ecd644b6-fa8f-4a56-a245-21f1f37c0bbd\") " Jul 08 19:07:10.662174 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:10.661990 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vt42\" (UniqueName: \"kubernetes.io/projected/ecd644b6-fa8f-4a56-a245-21f1f37c0bbd-kube-api-access-9vt42\") pod \"ecd644b6-fa8f-4a56-a245-21f1f37c0bbd\" (UID: \"ecd644b6-fa8f-4a56-a245-21f1f37c0bbd\") " Jul 08 19:07:10.662174 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:10.662018 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecd644b6-fa8f-4a56-a245-21f1f37c0bbd-openshift-service-ca\") pod \"ecd644b6-fa8f-4a56-a245-21f1f37c0bbd\" (UID: \"ecd644b6-fa8f-4a56-a245-21f1f37c0bbd\") " Jul 08 19:07:10.662465 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:10.662437 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd644b6-fa8f-4a56-a245-21f1f37c0bbd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ecd644b6-fa8f-4a56-a245-21f1f37c0bbd" (UID: "ecd644b6-fa8f-4a56-a245-21f1f37c0bbd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:07:10.664220 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:10.664186 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecd644b6-fa8f-4a56-a245-21f1f37c0bbd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ecd644b6-fa8f-4a56-a245-21f1f37c0bbd" (UID: "ecd644b6-fa8f-4a56-a245-21f1f37c0bbd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:07:10.664356 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:10.664219 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd644b6-fa8f-4a56-a245-21f1f37c0bbd-kube-api-access-9vt42" (OuterVolumeSpecName: "kube-api-access-9vt42") pod "ecd644b6-fa8f-4a56-a245-21f1f37c0bbd" (UID: "ecd644b6-fa8f-4a56-a245-21f1f37c0bbd"). InnerVolumeSpecName "kube-api-access-9vt42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:07:10.763115 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:10.763076 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vt42\" (UniqueName: \"kubernetes.io/projected/ecd644b6-fa8f-4a56-a245-21f1f37c0bbd-kube-api-access-9vt42\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:07:10.763115 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:10.763107 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecd644b6-fa8f-4a56-a245-21f1f37c0bbd-openshift-service-ca\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:07:10.763115 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:10.763116 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecd644b6-fa8f-4a56-a245-21f1f37c0bbd-maas-api-tls\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:07:11.349485 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:11.349445 2575 generic.go:356] "Generic (PLEG): container finished" podID="ecd644b6-fa8f-4a56-a245-21f1f37c0bbd" containerID="f99768b01de6b29d1fd687c8d3b5a275ab6942b52ee9bf16584e80a2ad20bef5" exitCode=0 Jul 08 19:07:11.349907 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:11.349496 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2" Jul 08 19:07:11.349907 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:11.349519 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2" event={"ID":"ecd644b6-fa8f-4a56-a245-21f1f37c0bbd","Type":"ContainerDied","Data":"f99768b01de6b29d1fd687c8d3b5a275ab6942b52ee9bf16584e80a2ad20bef5"} Jul 08 19:07:11.349907 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:11.349570 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2" event={"ID":"ecd644b6-fa8f-4a56-a245-21f1f37c0bbd","Type":"ContainerDied","Data":"f490bd86b19603cf257cc61241874325aefec191e5de9de019ac8d3918f0604a"} Jul 08 19:07:11.349907 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:11.349591 2575 scope.go:117] "RemoveContainer" containerID="f99768b01de6b29d1fd687c8d3b5a275ab6942b52ee9bf16584e80a2ad20bef5" Jul 08 19:07:11.359387 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:11.359364 2575 scope.go:117] "RemoveContainer" containerID="f99768b01de6b29d1fd687c8d3b5a275ab6942b52ee9bf16584e80a2ad20bef5" Jul 08 19:07:11.359627 ip-10-0-131-102 kubenswrapper[2575]: E0708 19:07:11.359610 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f99768b01de6b29d1fd687c8d3b5a275ab6942b52ee9bf16584e80a2ad20bef5\": container with ID starting with f99768b01de6b29d1fd687c8d3b5a275ab6942b52ee9bf16584e80a2ad20bef5 not found: ID does not exist" containerID="f99768b01de6b29d1fd687c8d3b5a275ab6942b52ee9bf16584e80a2ad20bef5" Jul 08 19:07:11.359679 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:11.359637 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f99768b01de6b29d1fd687c8d3b5a275ab6942b52ee9bf16584e80a2ad20bef5"} err="failed to get container status \"f99768b01de6b29d1fd687c8d3b5a275ab6942b52ee9bf16584e80a2ad20bef5\": rpc error: code = NotFound desc = could not find container \"f99768b01de6b29d1fd687c8d3b5a275ab6942b52ee9bf16584e80a2ad20bef5\": container with ID starting with f99768b01de6b29d1fd687c8d3b5a275ab6942b52ee9bf16584e80a2ad20bef5 not found: ID does not exist" Jul 08 19:07:11.372187 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:11.372163 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2"] Jul 08 19:07:11.376138 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:11.376110 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8f5e817a-78b9df9c48-gbkx2"] Jul 08 19:07:11.780306 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:11.780257 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecd644b6-fa8f-4a56-a245-21f1f37c0bbd" path="/var/lib/kubelet/pods/ecd644b6-fa8f-4a56-a245-21f1f37c0bbd/volumes" Jul 08 19:07:22.872560 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:22.872524 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9"] Jul 08 19:07:22.873132 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:22.873092 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecd644b6-fa8f-4a56-a245-21f1f37c0bbd" containerName="maas-api" Jul 08 19:07:22.873132 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:22.873116 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecd644b6-fa8f-4a56-a245-21f1f37c0bbd" containerName="maas-api" Jul 08 19:07:22.873254 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:22.873244 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecd644b6-fa8f-4a56-a245-21f1f37c0bbd" containerName="maas-api" Jul 08 19:07:22.876304 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:22.876270 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:22.881089 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:22.881063 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-d50245fc-openshift-default-dockercfg-hqxvn\"" Jul 08 19:07:22.900421 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:22.900390 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9"] Jul 08 19:07:22.971821 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:22.971777 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-workload-certs\") pod \"e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:22.971821 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:22.971823 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-istio-data\") pod \"e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:22.972084 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:22.972002 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d0b705c-c068-40c5-8771-434be0f26d79-istio-token\") pod \"e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:22.972084 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:22.972057 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jn24\" (UniqueName: \"kubernetes.io/projected/8d0b705c-c068-40c5-8771-434be0f26d79-kube-api-access-6jn24\") pod \"e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:22.972211 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:22.972096 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d0b705c-c068-40c5-8771-434be0f26d79-istiod-ca-cert\") pod \"e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:22.972211 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:22.972127 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-istio-envoy\") pod \"e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:22.972318 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:22.972224 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-credential-socket\") pod \"e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:22.972318 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:22.972254 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d0b705c-c068-40c5-8771-434be0f26d79-istio-podinfo\") pod \"e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:22.972318 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:22.972276 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-workload-socket\") pod \"e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:23.072744 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:23.072700 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d0b705c-c068-40c5-8771-434be0f26d79-istiod-ca-cert\") pod \"e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:23.072744 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:23.072748 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-istio-envoy\") pod \"e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:23.073009 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:23.072783 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-credential-socket\") pod \"e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:23.073009 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:23.072802 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d0b705c-c068-40c5-8771-434be0f26d79-istio-podinfo\") pod \"e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:23.073009 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:23.072823 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-workload-socket\") pod \"e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:23.073009 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:23.072858 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-workload-certs\") pod \"e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:23.073381 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:23.073272 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-istio-data\") pod \"e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:23.073381 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:23.073287 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-credential-socket\") pod \"e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:23.073526 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:23.073381 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d0b705c-c068-40c5-8771-434be0f26d79-istio-token\") pod \"e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:23.073526 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:23.073436 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jn24\" (UniqueName: \"kubernetes.io/projected/8d0b705c-c068-40c5-8771-434be0f26d79-kube-api-access-6jn24\") pod \"e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:23.073526 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:23.073462 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-workload-socket\") pod \"e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:23.073678 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:23.073560 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d0b705c-c068-40c5-8771-434be0f26d79-istiod-ca-cert\") pod \"e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:23.073872 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:23.073848 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-workload-certs\") pod \"e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:23.073997 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:23.073975 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-istio-data\") pod \"e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:23.075191 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:23.075170 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-istio-envoy\") pod \"e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:23.075419 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:23.075399 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d0b705c-c068-40c5-8771-434be0f26d79-istio-podinfo\") pod \"e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:23.086631 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:23.086608 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jn24\" (UniqueName: \"kubernetes.io/projected/8d0b705c-c068-40c5-8771-434be0f26d79-kube-api-access-6jn24\") pod \"e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:23.100564 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:23.100540 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d0b705c-c068-40c5-8771-434be0f26d79-istio-token\") pod \"e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:23.187616 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:23.187514 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:23.331829 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:23.331796 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9"] Jul 08 19:07:23.333034 ip-10-0-131-102 kubenswrapper[2575]: W0708 19:07:23.333011 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d0b705c_c068_40c5_8771_434be0f26d79.slice/crio-ec065a0b58d40c1c5867959dde03530649a8202ede3fab5240b172260fe978a8 WatchSource:0}: Error finding container ec065a0b58d40c1c5867959dde03530649a8202ede3fab5240b172260fe978a8: Status 404 returned error can't find the container with id ec065a0b58d40c1c5867959dde03530649a8202ede3fab5240b172260fe978a8 Jul 08 19:07:23.335122 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:23.335091 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:07:23.335185 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:23.335161 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:07:23.335217 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:23.335187 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:07:23.408427 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:23.408399 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" event={"ID":"8d0b705c-c068-40c5-8771-434be0f26d79","Type":"ContainerStarted","Data":"e6c4a8b27db8e38c0909d063a065decda9f6a958bd33f97aeb3d3c2ff1db2be6"} Jul 08 19:07:23.408558 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:23.408451 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" event={"ID":"8d0b705c-c068-40c5-8771-434be0f26d79","Type":"ContainerStarted","Data":"ec065a0b58d40c1c5867959dde03530649a8202ede3fab5240b172260fe978a8"} Jul 08 19:07:23.435955 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:23.435887 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" podStartSLOduration=1.4358668909999999 podStartE2EDuration="1.435866891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:07:23.434632364 +0000 UTC m=+1654.271960073" watchObservedRunningTime="2026-07-08 19:07:23.435866891 +0000 UTC m=+1654.273194601" Jul 08 19:07:24.188739 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:24.188709 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:24.193264 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:24.193241 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:24.413773 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:24.413740 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:24.414751 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:24.414722 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:28.747819 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:28.747765 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9"] Jul 08 19:07:28.748333 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:28.747993 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" podUID="8d0b705c-c068-40c5-8771-434be0f26d79" containerName="istio-proxy" containerID="cri-o://e6c4a8b27db8e38c0909d063a065decda9f6a958bd33f97aeb3d3c2ff1db2be6" gracePeriod=30 Jul 08 19:07:35.002786 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.002756 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:35.094101 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.094003 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-workload-certs\") pod \"8d0b705c-c068-40c5-8771-434be0f26d79\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " Jul 08 19:07:35.094101 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.094051 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-workload-socket\") pod \"8d0b705c-c068-40c5-8771-434be0f26d79\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " Jul 08 19:07:35.094101 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.094099 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-istio-envoy\") pod \"8d0b705c-c068-40c5-8771-434be0f26d79\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " Jul 08 19:07:35.094405 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.094123 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-credential-socket\") pod \"8d0b705c-c068-40c5-8771-434be0f26d79\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " Jul 08 19:07:35.094405 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.094169 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-istio-data\") pod \"8d0b705c-c068-40c5-8771-434be0f26d79\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " Jul 08 19:07:35.094405 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.094212 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d0b705c-c068-40c5-8771-434be0f26d79-istiod-ca-cert\") pod \"8d0b705c-c068-40c5-8771-434be0f26d79\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " Jul 08 19:07:35.094405 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.094242 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d0b705c-c068-40c5-8771-434be0f26d79-istio-podinfo\") pod \"8d0b705c-c068-40c5-8771-434be0f26d79\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " Jul 08 19:07:35.094405 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.094286 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d0b705c-c068-40c5-8771-434be0f26d79-istio-token\") pod \"8d0b705c-c068-40c5-8771-434be0f26d79\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " Jul 08 19:07:35.094405 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.094382 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8d0b705c-c068-40c5-8771-434be0f26d79" (UID: "8d0b705c-c068-40c5-8771-434be0f26d79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:07:35.094712 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.094461 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jn24\" (UniqueName: \"kubernetes.io/projected/8d0b705c-c068-40c5-8771-434be0f26d79-kube-api-access-6jn24\") pod \"8d0b705c-c068-40c5-8771-434be0f26d79\" (UID: \"8d0b705c-c068-40c5-8771-434be0f26d79\") " Jul 08 19:07:35.094712 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.094495 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8d0b705c-c068-40c5-8771-434be0f26d79" (UID: "8d0b705c-c068-40c5-8771-434be0f26d79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:07:35.094712 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.094514 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8d0b705c-c068-40c5-8771-434be0f26d79" (UID: "8d0b705c-c068-40c5-8771-434be0f26d79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:07:35.094883 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.094812 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-workload-certs\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:07:35.094883 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.094812 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d0b705c-c068-40c5-8771-434be0f26d79-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8d0b705c-c068-40c5-8771-434be0f26d79" (UID: "8d0b705c-c068-40c5-8771-434be0f26d79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:07:35.094883 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.094830 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-workload-socket\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:07:35.094883 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.094845 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-credential-socket\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:07:35.095271 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.095246 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-istio-data" (OuterVolumeSpecName: "istio-data") pod "8d0b705c-c068-40c5-8771-434be0f26d79" (UID: "8d0b705c-c068-40c5-8771-434be0f26d79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:07:35.096530 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.096495 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d0b705c-c068-40c5-8771-434be0f26d79-istio-token" (OuterVolumeSpecName: "istio-token") pod "8d0b705c-c068-40c5-8771-434be0f26d79" (UID: "8d0b705c-c068-40c5-8771-434be0f26d79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:07:35.096530 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.096522 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d0b705c-c068-40c5-8771-434be0f26d79-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8d0b705c-c068-40c5-8771-434be0f26d79" (UID: "8d0b705c-c068-40c5-8771-434be0f26d79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 19:07:35.096780 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.096764 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8d0b705c-c068-40c5-8771-434be0f26d79" (UID: "8d0b705c-c068-40c5-8771-434be0f26d79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:07:35.096822 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.096801 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d0b705c-c068-40c5-8771-434be0f26d79-kube-api-access-6jn24" (OuterVolumeSpecName: "kube-api-access-6jn24") pod "8d0b705c-c068-40c5-8771-434be0f26d79" (UID: "8d0b705c-c068-40c5-8771-434be0f26d79"). InnerVolumeSpecName "kube-api-access-6jn24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:07:35.195495 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.195456 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-istio-envoy\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:07:35.195495 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.195486 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d0b705c-c068-40c5-8771-434be0f26d79-istio-data\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:07:35.195495 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.195498 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d0b705c-c068-40c5-8771-434be0f26d79-istiod-ca-cert\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:07:35.195738 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.195510 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d0b705c-c068-40c5-8771-434be0f26d79-istio-podinfo\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:07:35.195738 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.195521 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d0b705c-c068-40c5-8771-434be0f26d79-istio-token\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:07:35.195738 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.195534 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jn24\" (UniqueName: \"kubernetes.io/projected/8d0b705c-c068-40c5-8771-434be0f26d79-kube-api-access-6jn24\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:07:35.489619 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.489581 2575 generic.go:356] "Generic (PLEG): container finished" podID="8d0b705c-c068-40c5-8771-434be0f26d79" containerID="e6c4a8b27db8e38c0909d063a065decda9f6a958bd33f97aeb3d3c2ff1db2be6" exitCode=0 Jul 08 19:07:35.489791 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.489629 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" Jul 08 19:07:35.489791 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.489663 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" event={"ID":"8d0b705c-c068-40c5-8771-434be0f26d79","Type":"ContainerDied","Data":"e6c4a8b27db8e38c0909d063a065decda9f6a958bd33f97aeb3d3c2ff1db2be6"} Jul 08 19:07:35.489791 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.489719 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9" event={"ID":"8d0b705c-c068-40c5-8771-434be0f26d79","Type":"ContainerDied","Data":"ec065a0b58d40c1c5867959dde03530649a8202ede3fab5240b172260fe978a8"} Jul 08 19:07:35.489791 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.489741 2575 scope.go:117] "RemoveContainer" containerID="e6c4a8b27db8e38c0909d063a065decda9f6a958bd33f97aeb3d3c2ff1db2be6" Jul 08 19:07:35.499289 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.499270 2575 scope.go:117] "RemoveContainer" containerID="e6c4a8b27db8e38c0909d063a065decda9f6a958bd33f97aeb3d3c2ff1db2be6" Jul 08 19:07:35.499591 ip-10-0-131-102 kubenswrapper[2575]: E0708 19:07:35.499574 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6c4a8b27db8e38c0909d063a065decda9f6a958bd33f97aeb3d3c2ff1db2be6\": container with ID starting with e6c4a8b27db8e38c0909d063a065decda9f6a958bd33f97aeb3d3c2ff1db2be6 not found: ID does not exist" containerID="e6c4a8b27db8e38c0909d063a065decda9f6a958bd33f97aeb3d3c2ff1db2be6" Jul 08 19:07:35.499633 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.499598 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6c4a8b27db8e38c0909d063a065decda9f6a958bd33f97aeb3d3c2ff1db2be6"} err="failed to get container status \"e6c4a8b27db8e38c0909d063a065decda9f6a958bd33f97aeb3d3c2ff1db2be6\": rpc error: code = NotFound desc = could not find container \"e6c4a8b27db8e38c0909d063a065decda9f6a958bd33f97aeb3d3c2ff1db2be6\": container with ID starting with e6c4a8b27db8e38c0909d063a065decda9f6a958bd33f97aeb3d3c2ff1db2be6 not found: ID does not exist" Jul 08 19:07:35.512582 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.512553 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9"] Jul 08 19:07:35.516307 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.516272 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-d50245fc-openshift-default-57567b6d7-4qzm9"] Jul 08 19:07:35.780448 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:07:35.780369 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d0b705c-c068-40c5-8771-434be0f26d79" path="/var/lib/kubelet/pods/8d0b705c-c068-40c5-8771-434be0f26d79/volumes" Jul 08 19:09:16.748891 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:16.748851 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j"] Jul 08 19:09:16.749444 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:16.749427 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d0b705c-c068-40c5-8771-434be0f26d79" containerName="istio-proxy" Jul 08 19:09:16.749491 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:16.749447 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d0b705c-c068-40c5-8771-434be0f26d79" containerName="istio-proxy" Jul 08 19:09:16.749601 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:16.749590 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d0b705c-c068-40c5-8771-434be0f26d79" containerName="istio-proxy" Jul 08 19:09:16.753011 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:16.752990 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j" Jul 08 19:09:16.755933 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:16.755913 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5071b272-serving-cert\"" Jul 08 19:09:16.766841 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:16.766819 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j"] Jul 08 19:09:16.821613 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:16.821575 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5-maas-api-tls\") pod \"maas-api-e2e-mt-5071b272-655cbf6558-php6j\" (UID: \"a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j" Jul 08 19:09:16.821793 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:16.821626 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5-openshift-service-ca\") pod \"maas-api-e2e-mt-5071b272-655cbf6558-php6j\" (UID: \"a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j" Jul 08 19:09:16.821793 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:16.821657 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnrv9\" (UniqueName: \"kubernetes.io/projected/a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5-kube-api-access-jnrv9\") pod \"maas-api-e2e-mt-5071b272-655cbf6558-php6j\" (UID: \"a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j" Jul 08 19:09:16.922610 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:16.922570 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5-maas-api-tls\") pod \"maas-api-e2e-mt-5071b272-655cbf6558-php6j\" (UID: \"a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j" Jul 08 19:09:16.922828 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:16.922634 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5-openshift-service-ca\") pod \"maas-api-e2e-mt-5071b272-655cbf6558-php6j\" (UID: \"a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j" Jul 08 19:09:16.922828 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:16.922664 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnrv9\" (UniqueName: \"kubernetes.io/projected/a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5-kube-api-access-jnrv9\") pod \"maas-api-e2e-mt-5071b272-655cbf6558-php6j\" (UID: \"a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j" Jul 08 19:09:16.922828 ip-10-0-131-102 kubenswrapper[2575]: E0708 19:09:16.922662 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-serving-cert: secret "maas-api-e2e-mt-5071b272-serving-cert" not found Jul 08 19:09:16.922828 ip-10-0-131-102 kubenswrapper[2575]: E0708 19:09:16.922788 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5-maas-api-tls podName:a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5 nodeName:}" failed. No retries permitted until 2026-07-08 19:09:17.422764931 +0000 UTC m=+1768.260092640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5-maas-api-tls") pod "maas-api-e2e-mt-5071b272-655cbf6558-php6j" (UID: "a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5") : secret "maas-api-e2e-mt-5071b272-serving-cert" not found Jul 08 19:09:16.923242 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:16.923221 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5-openshift-service-ca\") pod \"maas-api-e2e-mt-5071b272-655cbf6558-php6j\" (UID: \"a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j" Jul 08 19:09:16.931517 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:16.931493 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnrv9\" (UniqueName: \"kubernetes.io/projected/a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5-kube-api-access-jnrv9\") pod \"maas-api-e2e-mt-5071b272-655cbf6558-php6j\" (UID: \"a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j" Jul 08 19:09:17.426975 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:17.426939 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5-maas-api-tls\") pod \"maas-api-e2e-mt-5071b272-655cbf6558-php6j\" (UID: \"a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j" Jul 08 19:09:17.429419 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:17.429396 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5-maas-api-tls\") pod \"maas-api-e2e-mt-5071b272-655cbf6558-php6j\" (UID: \"a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j" Jul 08 19:09:17.666541 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:17.666499 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j" Jul 08 19:09:17.806780 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:17.806749 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j"] Jul 08 19:09:17.808373 ip-10-0-131-102 kubenswrapper[2575]: W0708 19:09:17.808347 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7bf98df_d3ca_4102_a3a9_a7b8bf5061d5.slice/crio-630e0ce9f004ce339727219651ee2ba1d03f9dff9507635a36766183e0f80924 WatchSource:0}: Error finding container 630e0ce9f004ce339727219651ee2ba1d03f9dff9507635a36766183e0f80924: Status 404 returned error can't find the container with id 630e0ce9f004ce339727219651ee2ba1d03f9dff9507635a36766183e0f80924 Jul 08 19:09:17.996176 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:17.996085 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j" event={"ID":"a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5","Type":"ContainerStarted","Data":"630e0ce9f004ce339727219651ee2ba1d03f9dff9507635a36766183e0f80924"} Jul 08 19:09:19.002188 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:19.002148 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j" event={"ID":"a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5","Type":"ContainerStarted","Data":"7b50ee41ce2340c8c013e25687b6acd786ea08d1ec88e46c1459fbd854b70729"} Jul 08 19:09:19.002600 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:19.002218 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j" Jul 08 19:09:19.020953 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:19.020900 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j" podStartSLOduration=2.601835146 podStartE2EDuration="3.020885389s" totalImagesPullingTime="419.050243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:09:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:09:19.019720471 +0000 UTC m=+1769.857048178" watchObservedRunningTime="2026-07-08 19:09:19.020885389 +0000 UTC m=+1769.858213097" Jul 08 19:09:25.014751 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:25.014718 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j" Jul 08 19:09:29.003445 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:29.003411 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j"] Jul 08 19:09:29.003919 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:29.003659 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j" podUID="a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5" containerName="maas-api" containerID="cri-o://7b50ee41ce2340c8c013e25687b6acd786ea08d1ec88e46c1459fbd854b70729" gracePeriod=30 Jul 08 19:09:29.260809 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:29.260748 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j" Jul 08 19:09:29.431777 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:29.431747 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5-maas-api-tls\") pod \"a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5\" (UID: \"a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5\") " Jul 08 19:09:29.431777 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:29.431783 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5-openshift-service-ca\") pod \"a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5\" (UID: \"a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5\") " Jul 08 19:09:29.432037 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:29.431802 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnrv9\" (UniqueName: \"kubernetes.io/projected/a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5-kube-api-access-jnrv9\") pod \"a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5\" (UID: \"a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5\") " Jul 08 19:09:29.432215 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:29.432187 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5" (UID: "a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:09:29.433972 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:29.433938 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5" (UID: "a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:09:29.433972 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:29.433952 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5-kube-api-access-jnrv9" (OuterVolumeSpecName: "kube-api-access-jnrv9") pod "a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5" (UID: "a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5"). InnerVolumeSpecName "kube-api-access-jnrv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:09:29.533341 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:29.533234 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5-maas-api-tls\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:09:29.533341 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:29.533265 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5-openshift-service-ca\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:09:29.533341 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:29.533275 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnrv9\" (UniqueName: \"kubernetes.io/projected/a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5-kube-api-access-jnrv9\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:09:30.057806 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:30.057716 2575 generic.go:356] "Generic (PLEG): container finished" podID="a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5" containerID="7b50ee41ce2340c8c013e25687b6acd786ea08d1ec88e46c1459fbd854b70729" exitCode=0 Jul 08 19:09:30.057806 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:30.057760 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j" Jul 08 19:09:30.058234 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:30.057810 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j" event={"ID":"a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5","Type":"ContainerDied","Data":"7b50ee41ce2340c8c013e25687b6acd786ea08d1ec88e46c1459fbd854b70729"} Jul 08 19:09:30.058234 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:30.057864 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j" event={"ID":"a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5","Type":"ContainerDied","Data":"630e0ce9f004ce339727219651ee2ba1d03f9dff9507635a36766183e0f80924"} Jul 08 19:09:30.058234 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:30.057888 2575 scope.go:117] "RemoveContainer" containerID="7b50ee41ce2340c8c013e25687b6acd786ea08d1ec88e46c1459fbd854b70729" Jul 08 19:09:30.066253 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:30.066234 2575 scope.go:117] "RemoveContainer" containerID="7b50ee41ce2340c8c013e25687b6acd786ea08d1ec88e46c1459fbd854b70729" Jul 08 19:09:30.066545 ip-10-0-131-102 kubenswrapper[2575]: E0708 19:09:30.066518 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b50ee41ce2340c8c013e25687b6acd786ea08d1ec88e46c1459fbd854b70729\": container with ID starting with 7b50ee41ce2340c8c013e25687b6acd786ea08d1ec88e46c1459fbd854b70729 not found: ID does not exist" containerID="7b50ee41ce2340c8c013e25687b6acd786ea08d1ec88e46c1459fbd854b70729" Jul 08 19:09:30.066610 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:30.066552 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b50ee41ce2340c8c013e25687b6acd786ea08d1ec88e46c1459fbd854b70729"} err="failed to get container status \"7b50ee41ce2340c8c013e25687b6acd786ea08d1ec88e46c1459fbd854b70729\": rpc error: code = NotFound desc = could not find container \"7b50ee41ce2340c8c013e25687b6acd786ea08d1ec88e46c1459fbd854b70729\": container with ID starting with 7b50ee41ce2340c8c013e25687b6acd786ea08d1ec88e46c1459fbd854b70729 not found: ID does not exist" Jul 08 19:09:30.076271 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:30.076244 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j"] Jul 08 19:09:30.078079 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:30.078059 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5071b272-655cbf6558-php6j"] Jul 08 19:09:31.780607 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:31.780575 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5" path="/var/lib/kubelet/pods/a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5/volumes" Jul 08 19:09:59.128241 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:59.128202 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr"] Jul 08 19:09:59.128756 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:59.128646 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5" containerName="maas-api" Jul 08 19:09:59.128756 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:59.128661 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5" containerName="maas-api" Jul 08 19:09:59.128756 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:59.128747 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7bf98df-d3ca-4102-a3a9-a7b8bf5061d5" containerName="maas-api" Jul 08 19:09:59.131795 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:59.131775 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr" Jul 08 19:09:59.134993 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:59.134974 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2ab8abd0-serving-cert\"" Jul 08 19:09:59.148685 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:59.148660 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr"] Jul 08 19:09:59.198010 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:59.197974 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33903f98-67bc-4f80-905c-b1e5f1a7b225-maas-api-tls\") pod \"maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr\" (UID: \"33903f98-67bc-4f80-905c-b1e5f1a7b225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr" Jul 08 19:09:59.198010 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:59.198010 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33903f98-67bc-4f80-905c-b1e5f1a7b225-openshift-service-ca\") pod \"maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr\" (UID: \"33903f98-67bc-4f80-905c-b1e5f1a7b225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr" Jul 08 19:09:59.198227 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:59.198153 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwx7g\" (UniqueName: \"kubernetes.io/projected/33903f98-67bc-4f80-905c-b1e5f1a7b225-kube-api-access-nwx7g\") pod \"maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr\" (UID: \"33903f98-67bc-4f80-905c-b1e5f1a7b225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr" Jul 08 19:09:59.299619 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:59.299579 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwx7g\" (UniqueName: \"kubernetes.io/projected/33903f98-67bc-4f80-905c-b1e5f1a7b225-kube-api-access-nwx7g\") pod \"maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr\" (UID: \"33903f98-67bc-4f80-905c-b1e5f1a7b225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr" Jul 08 19:09:59.299812 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:59.299645 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33903f98-67bc-4f80-905c-b1e5f1a7b225-maas-api-tls\") pod \"maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr\" (UID: \"33903f98-67bc-4f80-905c-b1e5f1a7b225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr" Jul 08 19:09:59.299812 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:59.299749 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33903f98-67bc-4f80-905c-b1e5f1a7b225-openshift-service-ca\") pod \"maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr\" (UID: \"33903f98-67bc-4f80-905c-b1e5f1a7b225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr" Jul 08 19:09:59.300454 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:59.300433 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33903f98-67bc-4f80-905c-b1e5f1a7b225-openshift-service-ca\") pod \"maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr\" (UID: \"33903f98-67bc-4f80-905c-b1e5f1a7b225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr" Jul 08 19:09:59.302067 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:59.302036 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33903f98-67bc-4f80-905c-b1e5f1a7b225-maas-api-tls\") pod \"maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr\" (UID: \"33903f98-67bc-4f80-905c-b1e5f1a7b225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr" Jul 08 19:09:59.309981 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:59.309948 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwx7g\" (UniqueName: \"kubernetes.io/projected/33903f98-67bc-4f80-905c-b1e5f1a7b225-kube-api-access-nwx7g\") pod \"maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr\" (UID: \"33903f98-67bc-4f80-905c-b1e5f1a7b225\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr" Jul 08 19:09:59.442536 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:59.442432 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr" Jul 08 19:09:59.592448 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:09:59.592414 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr"] Jul 08 19:09:59.593259 ip-10-0-131-102 kubenswrapper[2575]: W0708 19:09:59.593232 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33903f98_67bc_4f80_905c_b1e5f1a7b225.slice/crio-5717f94e12f357815425daa4e82cc9f24d9972fd6141888740f3f0f3ce64dc9b WatchSource:0}: Error finding container 5717f94e12f357815425daa4e82cc9f24d9972fd6141888740f3f0f3ce64dc9b: Status 404 returned error can't find the container with id 5717f94e12f357815425daa4e82cc9f24d9972fd6141888740f3f0f3ce64dc9b Jul 08 19:10:00.205191 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:10:00.205091 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr" event={"ID":"33903f98-67bc-4f80-905c-b1e5f1a7b225","Type":"ContainerStarted","Data":"ead2984a2a6bd7a11dbebc6d1340b4f7b4f4ef546485223b59eeaaa4814a9d91"} Jul 08 19:10:00.205191 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:10:00.205160 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr" Jul 08 19:10:00.205191 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:10:00.205186 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr" event={"ID":"33903f98-67bc-4f80-905c-b1e5f1a7b225","Type":"ContainerStarted","Data":"5717f94e12f357815425daa4e82cc9f24d9972fd6141888740f3f0f3ce64dc9b"} Jul 08 19:10:00.226547 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:10:00.226490 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr" podStartSLOduration=0.869553625 podStartE2EDuration="1.226473258s" totalImagesPullingTime="356.919633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:09:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:10:00.225379512 +0000 UTC m=+1811.062707247" watchObservedRunningTime="2026-07-08 19:10:00.226473258 +0000 UTC m=+1811.063800971" Jul 08 19:10:06.217165 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:10:06.217133 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr" Jul 08 19:11:02.058573 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:11:02.058535 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr"] Jul 08 19:11:02.059199 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:11:02.058758 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr" podUID="33903f98-67bc-4f80-905c-b1e5f1a7b225" containerName="maas-api" containerID="cri-o://ead2984a2a6bd7a11dbebc6d1340b4f7b4f4ef546485223b59eeaaa4814a9d91" gracePeriod=30 Jul 08 19:11:02.305046 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:11:02.305020 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr" Jul 08 19:11:02.357334 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:11:02.357276 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33903f98-67bc-4f80-905c-b1e5f1a7b225-openshift-service-ca\") pod \"33903f98-67bc-4f80-905c-b1e5f1a7b225\" (UID: \"33903f98-67bc-4f80-905c-b1e5f1a7b225\") " Jul 08 19:11:02.357502 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:11:02.357342 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwx7g\" (UniqueName: \"kubernetes.io/projected/33903f98-67bc-4f80-905c-b1e5f1a7b225-kube-api-access-nwx7g\") pod \"33903f98-67bc-4f80-905c-b1e5f1a7b225\" (UID: \"33903f98-67bc-4f80-905c-b1e5f1a7b225\") " Jul 08 19:11:02.357502 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:11:02.357429 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33903f98-67bc-4f80-905c-b1e5f1a7b225-maas-api-tls\") pod \"33903f98-67bc-4f80-905c-b1e5f1a7b225\" (UID: \"33903f98-67bc-4f80-905c-b1e5f1a7b225\") " Jul 08 19:11:02.357725 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:11:02.357697 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33903f98-67bc-4f80-905c-b1e5f1a7b225-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "33903f98-67bc-4f80-905c-b1e5f1a7b225" (UID: "33903f98-67bc-4f80-905c-b1e5f1a7b225"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:11:02.359485 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:11:02.359463 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33903f98-67bc-4f80-905c-b1e5f1a7b225-kube-api-access-nwx7g" (OuterVolumeSpecName: "kube-api-access-nwx7g") pod "33903f98-67bc-4f80-905c-b1e5f1a7b225" (UID: "33903f98-67bc-4f80-905c-b1e5f1a7b225"). InnerVolumeSpecName "kube-api-access-nwx7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:11:02.359594 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:11:02.359499 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33903f98-67bc-4f80-905c-b1e5f1a7b225-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "33903f98-67bc-4f80-905c-b1e5f1a7b225" (UID: "33903f98-67bc-4f80-905c-b1e5f1a7b225"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:11:02.458394 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:11:02.458358 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33903f98-67bc-4f80-905c-b1e5f1a7b225-maas-api-tls\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:11:02.458394 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:11:02.458388 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33903f98-67bc-4f80-905c-b1e5f1a7b225-openshift-service-ca\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:11:02.458394 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:11:02.458398 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwx7g\" (UniqueName: \"kubernetes.io/projected/33903f98-67bc-4f80-905c-b1e5f1a7b225-kube-api-access-nwx7g\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:11:02.505668 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:11:02.505633 2575 generic.go:356] "Generic (PLEG): container finished" podID="33903f98-67bc-4f80-905c-b1e5f1a7b225" containerID="ead2984a2a6bd7a11dbebc6d1340b4f7b4f4ef546485223b59eeaaa4814a9d91" exitCode=0 Jul 08 19:11:02.505854 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:11:02.505674 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr" event={"ID":"33903f98-67bc-4f80-905c-b1e5f1a7b225","Type":"ContainerDied","Data":"ead2984a2a6bd7a11dbebc6d1340b4f7b4f4ef546485223b59eeaaa4814a9d91"} Jul 08 19:11:02.505854 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:11:02.505685 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr" Jul 08 19:11:02.505854 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:11:02.505727 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr" event={"ID":"33903f98-67bc-4f80-905c-b1e5f1a7b225","Type":"ContainerDied","Data":"5717f94e12f357815425daa4e82cc9f24d9972fd6141888740f3f0f3ce64dc9b"} Jul 08 19:11:02.505854 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:11:02.505755 2575 scope.go:117] "RemoveContainer" containerID="ead2984a2a6bd7a11dbebc6d1340b4f7b4f4ef546485223b59eeaaa4814a9d91" Jul 08 19:11:02.515400 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:11:02.515378 2575 scope.go:117] "RemoveContainer" containerID="ead2984a2a6bd7a11dbebc6d1340b4f7b4f4ef546485223b59eeaaa4814a9d91" Jul 08 19:11:02.515668 ip-10-0-131-102 kubenswrapper[2575]: E0708 19:11:02.515648 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ead2984a2a6bd7a11dbebc6d1340b4f7b4f4ef546485223b59eeaaa4814a9d91\": container with ID starting with ead2984a2a6bd7a11dbebc6d1340b4f7b4f4ef546485223b59eeaaa4814a9d91 not found: ID does not exist" containerID="ead2984a2a6bd7a11dbebc6d1340b4f7b4f4ef546485223b59eeaaa4814a9d91" Jul 08 19:11:02.515718 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:11:02.515676 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ead2984a2a6bd7a11dbebc6d1340b4f7b4f4ef546485223b59eeaaa4814a9d91"} err="failed to get container status \"ead2984a2a6bd7a11dbebc6d1340b4f7b4f4ef546485223b59eeaaa4814a9d91\": rpc error: code = NotFound desc = could not find container \"ead2984a2a6bd7a11dbebc6d1340b4f7b4f4ef546485223b59eeaaa4814a9d91\": container with ID starting with ead2984a2a6bd7a11dbebc6d1340b4f7b4f4ef546485223b59eeaaa4814a9d91 not found: ID does not exist" Jul 08 19:11:02.529355 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:11:02.529330 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr"] Jul 08 19:11:02.533742 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:11:02.533714 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2ab8abd0-596b5cbb86-kdzhr"] Jul 08 19:11:03.780469 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:11:03.780439 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33903f98-67bc-4f80-905c-b1e5f1a7b225" path="/var/lib/kubelet/pods/33903f98-67bc-4f80-905c-b1e5f1a7b225/volumes" Jul 08 19:12:54.694018 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:54.693964 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf"] Jul 08 19:12:54.696629 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:54.694512 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33903f98-67bc-4f80-905c-b1e5f1a7b225" containerName="maas-api" Jul 08 19:12:54.696629 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:54.694538 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="33903f98-67bc-4f80-905c-b1e5f1a7b225" containerName="maas-api" Jul 08 19:12:54.696629 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:54.694620 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="33903f98-67bc-4f80-905c-b1e5f1a7b225" containerName="maas-api" Jul 08 19:12:54.697588 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:54.697565 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf" Jul 08 19:12:54.700576 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:54.700554 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-abcbfa-serving-cert\"" Jul 08 19:12:54.709160 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:54.709135 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf"] Jul 08 19:12:54.867288 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:54.867249 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27f81dc7-2757-43af-bf4c-2fd111318802-openshift-service-ca\") pod \"maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf\" (UID: \"27f81dc7-2757-43af-bf4c-2fd111318802\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf" Jul 08 19:12:54.867288 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:54.867307 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27f81dc7-2757-43af-bf4c-2fd111318802-maas-api-tls\") pod \"maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf\" (UID: \"27f81dc7-2757-43af-bf4c-2fd111318802\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf" Jul 08 19:12:54.867524 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:54.867371 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gz2r\" (UniqueName: \"kubernetes.io/projected/27f81dc7-2757-43af-bf4c-2fd111318802-kube-api-access-7gz2r\") pod \"maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf\" (UID: \"27f81dc7-2757-43af-bf4c-2fd111318802\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf" Jul 08 19:12:54.967975 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:54.967874 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27f81dc7-2757-43af-bf4c-2fd111318802-openshift-service-ca\") pod \"maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf\" (UID: \"27f81dc7-2757-43af-bf4c-2fd111318802\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf" Jul 08 19:12:54.967975 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:54.967919 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27f81dc7-2757-43af-bf4c-2fd111318802-maas-api-tls\") pod \"maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf\" (UID: \"27f81dc7-2757-43af-bf4c-2fd111318802\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf" Jul 08 19:12:54.967975 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:54.967950 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gz2r\" (UniqueName: \"kubernetes.io/projected/27f81dc7-2757-43af-bf4c-2fd111318802-kube-api-access-7gz2r\") pod \"maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf\" (UID: \"27f81dc7-2757-43af-bf4c-2fd111318802\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf" Jul 08 19:12:54.968661 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:54.968632 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27f81dc7-2757-43af-bf4c-2fd111318802-openshift-service-ca\") pod \"maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf\" (UID: \"27f81dc7-2757-43af-bf4c-2fd111318802\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf" Jul 08 19:12:54.970420 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:54.970400 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27f81dc7-2757-43af-bf4c-2fd111318802-maas-api-tls\") pod \"maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf\" (UID: \"27f81dc7-2757-43af-bf4c-2fd111318802\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf" Jul 08 19:12:54.976521 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:54.976497 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gz2r\" (UniqueName: \"kubernetes.io/projected/27f81dc7-2757-43af-bf4c-2fd111318802-kube-api-access-7gz2r\") pod \"maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf\" (UID: \"27f81dc7-2757-43af-bf4c-2fd111318802\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf" Jul 08 19:12:55.010129 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:55.010095 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf" Jul 08 19:12:55.153227 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:55.153173 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf"] Jul 08 19:12:55.155660 ip-10-0-131-102 kubenswrapper[2575]: W0708 19:12:55.155631 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f81dc7_2757_43af_bf4c_2fd111318802.slice/crio-0ff0eaed9feb86cfc335bf8db7e406abed691ee062814054e6c3dd90954758db WatchSource:0}: Error finding container 0ff0eaed9feb86cfc335bf8db7e406abed691ee062814054e6c3dd90954758db: Status 404 returned error can't find the container with id 0ff0eaed9feb86cfc335bf8db7e406abed691ee062814054e6c3dd90954758db Jul 08 19:12:55.157002 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:55.156981 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 19:12:56.073961 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:56.073924 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf" event={"ID":"27f81dc7-2757-43af-bf4c-2fd111318802","Type":"ContainerStarted","Data":"1aec7c8d43379b4772377004153988f4532d0aebf137ed9877dbb8ceec11ba3f"} Jul 08 19:12:56.074418 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:56.073977 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf" event={"ID":"27f81dc7-2757-43af-bf4c-2fd111318802","Type":"ContainerStarted","Data":"0ff0eaed9feb86cfc335bf8db7e406abed691ee062814054e6c3dd90954758db"} Jul 08 19:12:56.074418 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:56.074017 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf" Jul 08 19:12:56.097060 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:56.097007 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf" podStartSLOduration=1.363365327 podStartE2EDuration="2.096994188s" totalImagesPullingTime="733.628861ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:12:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:12:56.096141549 +0000 UTC m=+1986.933469284" watchObservedRunningTime="2026-07-08 19:12:56.096994188 +0000 UTC m=+1986.934321895" Jul 08 19:12:59.753778 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:59.753745 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q"] Jul 08 19:12:59.757545 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:59.757522 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:12:59.760730 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:59.760706 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-521fd5-openshift-default-dockercfg-2r6cg\"" Jul 08 19:12:59.771719 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:59.771687 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q"] Jul 08 19:12:59.913522 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:59.913484 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-workload-certs\") pod \"e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:12:59.913722 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:59.913530 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-workload-socket\") pod \"e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:12:59.913722 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:59.913568 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57dmm\" (UniqueName: \"kubernetes.io/projected/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-kube-api-access-57dmm\") pod \"e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:12:59.913722 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:59.913600 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istio-token\") pod \"e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:12:59.913722 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:59.913629 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-credential-socket\") pod \"e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:12:59.913722 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:59.913691 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istio-envoy\") pod \"e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:12:59.913896 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:59.913757 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istio-podinfo\") pod \"e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:12:59.913896 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:59.913788 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istio-data\") pod \"e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:12:59.913896 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:12:59.913846 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istiod-ca-cert\") pod \"e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:13:00.014988 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:00.014891 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-workload-socket\") pod \"e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:13:00.014988 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:00.014928 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57dmm\" (UniqueName: \"kubernetes.io/projected/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-kube-api-access-57dmm\") pod \"e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:13:00.014988 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:00.014954 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istio-token\") pod \"e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:13:00.015322 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:00.015077 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-credential-socket\") pod \"e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:13:00.015322 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:00.015119 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istio-envoy\") pod \"e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:13:00.015322 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:00.015173 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istio-podinfo\") pod \"e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:13:00.015322 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:00.015203 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istio-data\") pod \"e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:13:00.015322 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:00.015269 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istiod-ca-cert\") pod \"e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:13:00.015665 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:00.015412 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-workload-certs\") pod \"e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:13:00.015665 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:00.015456 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-credential-socket\") pod \"e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:13:00.015665 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:00.015409 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-workload-socket\") pod \"e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:13:00.015665 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:00.015574 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istio-data\") pod \"e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:13:00.015828 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:00.015666 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-workload-certs\") pod \"e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:13:00.016023 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:00.016003 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istiod-ca-cert\") pod \"e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:13:00.017469 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:00.017445 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istio-envoy\") pod \"e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:13:00.017600 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:00.017581 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istio-podinfo\") pod \"e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:13:00.023730 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:00.023705 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istio-token\") pod \"e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:13:00.024017 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:00.023999 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57dmm\" (UniqueName: \"kubernetes.io/projected/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-kube-api-access-57dmm\") pod \"e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:13:00.069136 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:00.069102 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:13:00.219444 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:00.219414 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q"] Jul 08 19:13:00.221556 ip-10-0-131-102 kubenswrapper[2575]: W0708 19:13:00.221534 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a4a75e_f272_4fbf_97a6_b04cc2cfd970.slice/crio-b28a78243e3255b30705cf1fe37f3b153b7db17e563c2953688e1654d80b9aff WatchSource:0}: Error finding container b28a78243e3255b30705cf1fe37f3b153b7db17e563c2953688e1654d80b9aff: Status 404 returned error can't find the container with id b28a78243e3255b30705cf1fe37f3b153b7db17e563c2953688e1654d80b9aff Jul 08 19:13:00.224260 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:00.224230 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:13:00.224372 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:00.224330 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:13:00.224372 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:00.224364 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:13:01.109211 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:01.109167 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" event={"ID":"06a4a75e-f272-4fbf-97a6-b04cc2cfd970","Type":"ContainerStarted","Data":"6fd5bbcd1049fb165a7a9c84e2ae884810ba1200f6a90551757606b7b7932950"} Jul 08 19:13:01.109746 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:01.109260 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" event={"ID":"06a4a75e-f272-4fbf-97a6-b04cc2cfd970","Type":"ContainerStarted","Data":"b28a78243e3255b30705cf1fe37f3b153b7db17e563c2953688e1654d80b9aff"} Jul 08 19:13:01.135889 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:01.135833 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" podStartSLOduration=2.135816922 podStartE2EDuration="2.135816922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:13:01.134019759 +0000 UTC m=+1991.971347496" watchObservedRunningTime="2026-07-08 19:13:01.135816922 +0000 UTC m=+1991.973144687" Jul 08 19:13:02.069338 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:02.069279 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:13:02.074474 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:02.074445 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:13:02.084617 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:02.084596 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf" Jul 08 19:13:02.114039 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:02.114006 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:13:02.115098 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:02.115074 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:13:14.185043 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:14.184939 2575 generic.go:356] "Generic (PLEG): container finished" podID="06a4a75e-f272-4fbf-97a6-b04cc2cfd970" containerID="6fd5bbcd1049fb165a7a9c84e2ae884810ba1200f6a90551757606b7b7932950" exitCode=0 Jul 08 19:13:14.185580 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:14.185101 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" event={"ID":"06a4a75e-f272-4fbf-97a6-b04cc2cfd970","Type":"ContainerDied","Data":"6fd5bbcd1049fb165a7a9c84e2ae884810ba1200f6a90551757606b7b7932950"} Jul 08 19:13:14.185919 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:14.185888 2575 scope.go:117] "RemoveContainer" containerID="6fd5bbcd1049fb165a7a9c84e2ae884810ba1200f6a90551757606b7b7932950" Jul 08 19:13:14.189470 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:14.189420 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:13:14.189602 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:14.189579 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:13:14.189707 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:14.189685 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:13:15.192071 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:15.192036 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" event={"ID":"06a4a75e-f272-4fbf-97a6-b04cc2cfd970","Type":"ContainerStarted","Data":"b420c81fa6cf9dce142e487cd29bea5809620692300b0eaaeca7678e4ce77137"} Jul 08 19:13:16.069857 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:16.069822 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:13:16.071141 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:16.071102 2575 patch_prober.go:28] interesting pod/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 08 19:13:16.071317 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:16.071154 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" podUID="06a4a75e-f272-4fbf-97a6-b04cc2cfd970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 08 19:13:17.069841 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:17.069807 2575 patch_prober.go:28] interesting pod/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 08 19:13:17.070265 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:17.069861 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" podUID="06a4a75e-f272-4fbf-97a6-b04cc2cfd970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 08 19:13:17.114721 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:17.114685 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:13:18.070353 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:18.070284 2575 patch_prober.go:28] interesting pod/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 08 19:13:18.070753 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:18.070382 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" podUID="06a4a75e-f272-4fbf-97a6-b04cc2cfd970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 08 19:13:19.074000 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:19.073965 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:13:19.075021 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:13:19.075000 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:14:08.330850 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:14:08.330811 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf"] Jul 08 19:14:08.331489 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:14:08.331118 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf" podUID="27f81dc7-2757-43af-bf4c-2fd111318802" containerName="maas-api" containerID="cri-o://1aec7c8d43379b4772377004153988f4532d0aebf137ed9877dbb8ceec11ba3f" gracePeriod=30 Jul 08 19:14:08.785060 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:14:08.785023 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf" Jul 08 19:14:08.829274 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:14:08.829238 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27f81dc7-2757-43af-bf4c-2fd111318802-openshift-service-ca\") pod \"27f81dc7-2757-43af-bf4c-2fd111318802\" (UID: \"27f81dc7-2757-43af-bf4c-2fd111318802\") " Jul 08 19:14:08.829490 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:14:08.829321 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27f81dc7-2757-43af-bf4c-2fd111318802-maas-api-tls\") pod \"27f81dc7-2757-43af-bf4c-2fd111318802\" (UID: \"27f81dc7-2757-43af-bf4c-2fd111318802\") " Jul 08 19:14:08.829490 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:14:08.829357 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gz2r\" (UniqueName: \"kubernetes.io/projected/27f81dc7-2757-43af-bf4c-2fd111318802-kube-api-access-7gz2r\") pod \"27f81dc7-2757-43af-bf4c-2fd111318802\" (UID: \"27f81dc7-2757-43af-bf4c-2fd111318802\") " Jul 08 19:14:08.829690 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:14:08.829663 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f81dc7-2757-43af-bf4c-2fd111318802-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "27f81dc7-2757-43af-bf4c-2fd111318802" (UID: "27f81dc7-2757-43af-bf4c-2fd111318802"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:14:08.831543 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:14:08.831519 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27f81dc7-2757-43af-bf4c-2fd111318802-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "27f81dc7-2757-43af-bf4c-2fd111318802" (UID: "27f81dc7-2757-43af-bf4c-2fd111318802"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:14:08.831613 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:14:08.831532 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f81dc7-2757-43af-bf4c-2fd111318802-kube-api-access-7gz2r" (OuterVolumeSpecName: "kube-api-access-7gz2r") pod "27f81dc7-2757-43af-bf4c-2fd111318802" (UID: "27f81dc7-2757-43af-bf4c-2fd111318802"). InnerVolumeSpecName "kube-api-access-7gz2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:14:08.930321 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:14:08.930239 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27f81dc7-2757-43af-bf4c-2fd111318802-openshift-service-ca\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:14:08.930321 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:14:08.930289 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27f81dc7-2757-43af-bf4c-2fd111318802-maas-api-tls\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:14:08.930321 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:14:08.930334 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gz2r\" (UniqueName: \"kubernetes.io/projected/27f81dc7-2757-43af-bf4c-2fd111318802-kube-api-access-7gz2r\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:14:09.462864 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:14:09.462826 2575 generic.go:356] "Generic (PLEG): container finished" podID="27f81dc7-2757-43af-bf4c-2fd111318802" containerID="1aec7c8d43379b4772377004153988f4532d0aebf137ed9877dbb8ceec11ba3f" exitCode=0 Jul 08 19:14:09.463359 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:14:09.462869 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf" Jul 08 19:14:09.463359 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:14:09.462917 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf" event={"ID":"27f81dc7-2757-43af-bf4c-2fd111318802","Type":"ContainerDied","Data":"1aec7c8d43379b4772377004153988f4532d0aebf137ed9877dbb8ceec11ba3f"} Jul 08 19:14:09.463359 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:14:09.462968 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf" event={"ID":"27f81dc7-2757-43af-bf4c-2fd111318802","Type":"ContainerDied","Data":"0ff0eaed9feb86cfc335bf8db7e406abed691ee062814054e6c3dd90954758db"} Jul 08 19:14:09.463359 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:14:09.462989 2575 scope.go:117] "RemoveContainer" containerID="1aec7c8d43379b4772377004153988f4532d0aebf137ed9877dbb8ceec11ba3f" Jul 08 19:14:09.473387 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:14:09.473362 2575 scope.go:117] "RemoveContainer" containerID="1aec7c8d43379b4772377004153988f4532d0aebf137ed9877dbb8ceec11ba3f" Jul 08 19:14:09.473679 ip-10-0-131-102 kubenswrapper[2575]: E0708 19:14:09.473656 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aec7c8d43379b4772377004153988f4532d0aebf137ed9877dbb8ceec11ba3f\": container with ID starting with 1aec7c8d43379b4772377004153988f4532d0aebf137ed9877dbb8ceec11ba3f not found: ID does not exist" containerID="1aec7c8d43379b4772377004153988f4532d0aebf137ed9877dbb8ceec11ba3f" Jul 08 19:14:09.473754 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:14:09.473688 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aec7c8d43379b4772377004153988f4532d0aebf137ed9877dbb8ceec11ba3f"} err="failed to get container status \"1aec7c8d43379b4772377004153988f4532d0aebf137ed9877dbb8ceec11ba3f\": rpc error: code = NotFound desc = could not find container \"1aec7c8d43379b4772377004153988f4532d0aebf137ed9877dbb8ceec11ba3f\": container with ID starting with 1aec7c8d43379b4772377004153988f4532d0aebf137ed9877dbb8ceec11ba3f not found: ID does not exist" Jul 08 19:14:09.488950 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:14:09.488917 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf"] Jul 08 19:14:09.498278 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:14:09.498248 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-abcbfa-77b4778d79-jfhmf"] Jul 08 19:14:09.780886 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:14:09.780795 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27f81dc7-2757-43af-bf4c-2fd111318802" path="/var/lib/kubelet/pods/27f81dc7-2757-43af-bf4c-2fd111318802/volumes" Jul 08 19:14:57.905011 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:14:57.904974 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q"] Jul 08 19:14:57.905580 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:14:57.905192 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" podUID="06a4a75e-f272-4fbf-97a6-b04cc2cfd970" containerName="istio-proxy" containerID="cri-o://b420c81fa6cf9dce142e487cd29bea5809620692300b0eaaeca7678e4ce77137" gracePeriod=30 Jul 08 19:15:03.168669 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.168545 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:15:03.214059 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.214021 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istio-data\") pod \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " Jul 08 19:15:03.214236 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.214120 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57dmm\" (UniqueName: \"kubernetes.io/projected/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-kube-api-access-57dmm\") pod \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " Jul 08 19:15:03.214236 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.214143 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istio-podinfo\") pod \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " Jul 08 19:15:03.214236 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.214168 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istiod-ca-cert\") pod \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " Jul 08 19:15:03.214236 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.214213 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-workload-certs\") pod \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " Jul 08 19:15:03.214579 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.214237 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-workload-socket\") pod \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " Jul 08 19:15:03.214579 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.214276 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-credential-socket\") pod \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " Jul 08 19:15:03.214579 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.214344 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istio-token\") pod \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " Jul 08 19:15:03.214579 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.214372 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istio-envoy\") pod \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\" (UID: \"06a4a75e-f272-4fbf-97a6-b04cc2cfd970\") " Jul 08 19:15:03.214579 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.214559 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "06a4a75e-f272-4fbf-97a6-b04cc2cfd970" (UID: "06a4a75e-f272-4fbf-97a6-b04cc2cfd970"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:15:03.214844 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.214670 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "06a4a75e-f272-4fbf-97a6-b04cc2cfd970" (UID: "06a4a75e-f272-4fbf-97a6-b04cc2cfd970"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:15:03.214844 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.214743 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-workload-certs\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:15:03.214844 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.214775 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "06a4a75e-f272-4fbf-97a6-b04cc2cfd970" (UID: "06a4a75e-f272-4fbf-97a6-b04cc2cfd970"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:15:03.215005 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.214944 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "06a4a75e-f272-4fbf-97a6-b04cc2cfd970" (UID: "06a4a75e-f272-4fbf-97a6-b04cc2cfd970"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:15:03.215005 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.214982 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istio-data" (OuterVolumeSpecName: "istio-data") pod "06a4a75e-f272-4fbf-97a6-b04cc2cfd970" (UID: "06a4a75e-f272-4fbf-97a6-b04cc2cfd970"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:15:03.216723 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.216694 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istio-token" (OuterVolumeSpecName: "istio-token") pod "06a4a75e-f272-4fbf-97a6-b04cc2cfd970" (UID: "06a4a75e-f272-4fbf-97a6-b04cc2cfd970"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:15:03.217125 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.217090 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "06a4a75e-f272-4fbf-97a6-b04cc2cfd970" (UID: "06a4a75e-f272-4fbf-97a6-b04cc2cfd970"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:15:03.217272 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.217252 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-kube-api-access-57dmm" (OuterVolumeSpecName: "kube-api-access-57dmm") pod "06a4a75e-f272-4fbf-97a6-b04cc2cfd970" (UID: "06a4a75e-f272-4fbf-97a6-b04cc2cfd970"). InnerVolumeSpecName "kube-api-access-57dmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:15:03.217691 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.217668 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "06a4a75e-f272-4fbf-97a6-b04cc2cfd970" (UID: "06a4a75e-f272-4fbf-97a6-b04cc2cfd970"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 19:15:03.315618 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.315581 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istio-token\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:15:03.315618 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.315612 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istio-envoy\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:15:03.315618 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.315624 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istio-data\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:15:03.315870 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.315638 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57dmm\" (UniqueName: \"kubernetes.io/projected/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-kube-api-access-57dmm\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:15:03.315870 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.315650 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istio-podinfo\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:15:03.315870 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.315658 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-istiod-ca-cert\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:15:03.315870 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.315666 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-workload-socket\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:15:03.315870 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.315674 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06a4a75e-f272-4fbf-97a6-b04cc2cfd970-credential-socket\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:15:03.739991 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.739956 2575 generic.go:356] "Generic (PLEG): container finished" podID="06a4a75e-f272-4fbf-97a6-b04cc2cfd970" containerID="b420c81fa6cf9dce142e487cd29bea5809620692300b0eaaeca7678e4ce77137" exitCode=0 Jul 08 19:15:03.740286 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.740022 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" Jul 08 19:15:03.740286 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.740028 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" event={"ID":"06a4a75e-f272-4fbf-97a6-b04cc2cfd970","Type":"ContainerDied","Data":"b420c81fa6cf9dce142e487cd29bea5809620692300b0eaaeca7678e4ce77137"} Jul 08 19:15:03.740286 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.740107 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q" event={"ID":"06a4a75e-f272-4fbf-97a6-b04cc2cfd970","Type":"ContainerDied","Data":"b28a78243e3255b30705cf1fe37f3b153b7db17e563c2953688e1654d80b9aff"} Jul 08 19:15:03.740286 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.740129 2575 scope.go:117] "RemoveContainer" containerID="b420c81fa6cf9dce142e487cd29bea5809620692300b0eaaeca7678e4ce77137" Jul 08 19:15:03.750362 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.750343 2575 scope.go:117] "RemoveContainer" containerID="6fd5bbcd1049fb165a7a9c84e2ae884810ba1200f6a90551757606b7b7932950" Jul 08 19:15:03.760706 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.760670 2575 scope.go:117] "RemoveContainer" containerID="b420c81fa6cf9dce142e487cd29bea5809620692300b0eaaeca7678e4ce77137" Jul 08 19:15:03.761038 ip-10-0-131-102 kubenswrapper[2575]: E0708 19:15:03.761018 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b420c81fa6cf9dce142e487cd29bea5809620692300b0eaaeca7678e4ce77137\": container with ID starting with b420c81fa6cf9dce142e487cd29bea5809620692300b0eaaeca7678e4ce77137 not found: ID does not exist" containerID="b420c81fa6cf9dce142e487cd29bea5809620692300b0eaaeca7678e4ce77137" Jul 08 19:15:03.761146 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.761051 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b420c81fa6cf9dce142e487cd29bea5809620692300b0eaaeca7678e4ce77137"} err="failed to get container status \"b420c81fa6cf9dce142e487cd29bea5809620692300b0eaaeca7678e4ce77137\": rpc error: code = NotFound desc = could not find container \"b420c81fa6cf9dce142e487cd29bea5809620692300b0eaaeca7678e4ce77137\": container with ID starting with b420c81fa6cf9dce142e487cd29bea5809620692300b0eaaeca7678e4ce77137 not found: ID does not exist" Jul 08 19:15:03.761146 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.761075 2575 scope.go:117] "RemoveContainer" containerID="6fd5bbcd1049fb165a7a9c84e2ae884810ba1200f6a90551757606b7b7932950" Jul 08 19:15:03.761402 ip-10-0-131-102 kubenswrapper[2575]: E0708 19:15:03.761377 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fd5bbcd1049fb165a7a9c84e2ae884810ba1200f6a90551757606b7b7932950\": container with ID starting with 6fd5bbcd1049fb165a7a9c84e2ae884810ba1200f6a90551757606b7b7932950 not found: ID does not exist" containerID="6fd5bbcd1049fb165a7a9c84e2ae884810ba1200f6a90551757606b7b7932950" Jul 08 19:15:03.761522 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.761406 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fd5bbcd1049fb165a7a9c84e2ae884810ba1200f6a90551757606b7b7932950"} err="failed to get container status \"6fd5bbcd1049fb165a7a9c84e2ae884810ba1200f6a90551757606b7b7932950\": rpc error: code = NotFound desc = could not find container \"6fd5bbcd1049fb165a7a9c84e2ae884810ba1200f6a90551757606b7b7932950\": container with ID starting with 6fd5bbcd1049fb165a7a9c84e2ae884810ba1200f6a90551757606b7b7932950 not found: ID does not exist" Jul 08 19:15:03.766590 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.766568 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q"] Jul 08 19:15:03.773197 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.773169 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-521fd5-openshift-default-7ddb957f5c-8849q"] Jul 08 19:15:03.781044 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:03.781016 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06a4a75e-f272-4fbf-97a6-b04cc2cfd970" path="/var/lib/kubelet/pods/06a4a75e-f272-4fbf-97a6-b04cc2cfd970/volumes" Jul 08 19:15:07.535121 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.535086 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m"] Jul 08 19:15:07.535721 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.535696 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06a4a75e-f272-4fbf-97a6-b04cc2cfd970" containerName="istio-proxy" Jul 08 19:15:07.535809 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.535724 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="06a4a75e-f272-4fbf-97a6-b04cc2cfd970" containerName="istio-proxy" Jul 08 19:15:07.535809 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.535744 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06a4a75e-f272-4fbf-97a6-b04cc2cfd970" containerName="istio-proxy" Jul 08 19:15:07.535809 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.535753 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="06a4a75e-f272-4fbf-97a6-b04cc2cfd970" containerName="istio-proxy" Jul 08 19:15:07.535809 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.535789 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27f81dc7-2757-43af-bf4c-2fd111318802" containerName="maas-api" Jul 08 19:15:07.535809 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.535798 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="27f81dc7-2757-43af-bf4c-2fd111318802" containerName="maas-api" Jul 08 19:15:07.536067 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.535906 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="06a4a75e-f272-4fbf-97a6-b04cc2cfd970" containerName="istio-proxy" Jul 08 19:15:07.536067 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.535919 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="27f81dc7-2757-43af-bf4c-2fd111318802" containerName="maas-api" Jul 08 19:15:07.540573 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.540549 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.544646 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.544620 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b742e8-openshift-default-dockercfg-7lk7l\"" Jul 08 19:15:07.553112 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.553086 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m"] Jul 08 19:15:07.652819 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.652786 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-credential-socket\") pod \"e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.652819 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.652821 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/785fe3c4-936e-43a3-a89e-5216265806fa-istiod-ca-cert\") pod \"e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.653057 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.652840 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsj8v\" (UniqueName: \"kubernetes.io/projected/785fe3c4-936e-43a3-a89e-5216265806fa-kube-api-access-zsj8v\") pod \"e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.653057 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.652873 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-workload-socket\") pod \"e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.653057 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.652939 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-workload-certs\") pod \"e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.653057 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.652957 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/785fe3c4-936e-43a3-a89e-5216265806fa-istio-podinfo\") pod \"e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.653057 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.652976 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-istio-envoy\") pod \"e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.653057 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.653001 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-istio-data\") pod \"e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.653057 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.653018 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/785fe3c4-936e-43a3-a89e-5216265806fa-istio-token\") pod \"e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.753487 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.753448 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-istio-data\") pod \"e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.753487 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.753495 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/785fe3c4-936e-43a3-a89e-5216265806fa-istio-token\") pod \"e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.753760 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.753560 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-credential-socket\") pod \"e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.753760 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.753618 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/785fe3c4-936e-43a3-a89e-5216265806fa-istiod-ca-cert\") pod \"e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.753760 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.753640 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsj8v\" (UniqueName: \"kubernetes.io/projected/785fe3c4-936e-43a3-a89e-5216265806fa-kube-api-access-zsj8v\") pod \"e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.753760 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.753681 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-workload-socket\") pod \"e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.753975 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.753770 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-workload-certs\") pod \"e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.753975 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.753800 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/785fe3c4-936e-43a3-a89e-5216265806fa-istio-podinfo\") pod \"e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.753975 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.753832 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-istio-envoy\") pod \"e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.753975 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.753938 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-istio-data\") pod \"e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.754148 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.754000 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-credential-socket\") pod \"e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.754572 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.754537 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-workload-certs\") pod \"e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.754731 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.754117 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-workload-socket\") pod \"e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.755003 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.754963 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/785fe3c4-936e-43a3-a89e-5216265806fa-istiod-ca-cert\") pod \"e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.762095 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.757610 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/785fe3c4-936e-43a3-a89e-5216265806fa-istio-podinfo\") pod \"e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.762330 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.762280 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-istio-envoy\") pod \"e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.764102 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.764075 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/785fe3c4-936e-43a3-a89e-5216265806fa-istio-token\") pod \"e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.764499 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.764478 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsj8v\" (UniqueName: \"kubernetes.io/projected/785fe3c4-936e-43a3-a89e-5216265806fa-kube-api-access-zsj8v\") pod \"e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:07.852350 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:07.852223 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:08.008026 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:08.007995 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m"] Jul 08 19:15:08.009682 ip-10-0-131-102 kubenswrapper[2575]: W0708 19:15:08.009653 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod785fe3c4_936e_43a3_a89e_5216265806fa.slice/crio-d663db6bb3174f8f7e14835881e20cb9156bd56d2bd71d4606ef12959cdafa87 WatchSource:0}: Error finding container d663db6bb3174f8f7e14835881e20cb9156bd56d2bd71d4606ef12959cdafa87: Status 404 returned error can't find the container with id d663db6bb3174f8f7e14835881e20cb9156bd56d2bd71d4606ef12959cdafa87 Jul 08 19:15:08.011895 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:08.011854 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:15:08.011964 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:08.011931 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:15:08.012001 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:08.011966 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:15:08.773785 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:08.773750 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" event={"ID":"785fe3c4-936e-43a3-a89e-5216265806fa","Type":"ContainerStarted","Data":"58eb1343bd7ad50717177d0c5923c0a801c1e5b8caa4d523cf127396aed5961f"} Jul 08 19:15:08.774199 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:08.773810 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" event={"ID":"785fe3c4-936e-43a3-a89e-5216265806fa","Type":"ContainerStarted","Data":"d663db6bb3174f8f7e14835881e20cb9156bd56d2bd71d4606ef12959cdafa87"} Jul 08 19:15:08.799868 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:08.799812 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" podStartSLOduration=1.799795431 podStartE2EDuration="1.799795431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:15:08.799155759 +0000 UTC m=+2119.636483493" watchObservedRunningTime="2026-07-08 19:15:08.799795431 +0000 UTC m=+2119.637123140" Jul 08 19:15:08.853165 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:08.853131 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:09.858029 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:09.857997 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:10.783500 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:10.783470 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:10.784380 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:10.784358 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:10.993973 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:10.993939 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd"] Jul 08 19:15:10.994545 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:10.994518 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="06a4a75e-f272-4fbf-97a6-b04cc2cfd970" containerName="istio-proxy" Jul 08 19:15:10.997836 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:10.997815 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd" Jul 08 19:15:11.001352 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:11.001329 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b742e8-serving-cert\"" Jul 08 19:15:11.008887 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:11.008787 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd"] Jul 08 19:15:11.082141 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:11.082044 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da33b491-4efa-40a7-8864-7615b562423d-maas-api-tls\") pod \"maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd\" (UID: \"da33b491-4efa-40a7-8864-7615b562423d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd" Jul 08 19:15:11.082141 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:11.082091 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da33b491-4efa-40a7-8864-7615b562423d-openshift-service-ca\") pod \"maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd\" (UID: \"da33b491-4efa-40a7-8864-7615b562423d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd" Jul 08 19:15:11.082375 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:11.082203 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8kb6\" (UniqueName: \"kubernetes.io/projected/da33b491-4efa-40a7-8864-7615b562423d-kube-api-access-q8kb6\") pod \"maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd\" (UID: \"da33b491-4efa-40a7-8864-7615b562423d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd" Jul 08 19:15:11.183289 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:11.183251 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da33b491-4efa-40a7-8864-7615b562423d-maas-api-tls\") pod \"maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd\" (UID: \"da33b491-4efa-40a7-8864-7615b562423d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd" Jul 08 19:15:11.183523 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:11.183327 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da33b491-4efa-40a7-8864-7615b562423d-openshift-service-ca\") pod \"maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd\" (UID: \"da33b491-4efa-40a7-8864-7615b562423d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd" Jul 08 19:15:11.183523 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:11.183439 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8kb6\" (UniqueName: \"kubernetes.io/projected/da33b491-4efa-40a7-8864-7615b562423d-kube-api-access-q8kb6\") pod \"maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd\" (UID: \"da33b491-4efa-40a7-8864-7615b562423d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd" Jul 08 19:15:11.184010 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:11.183990 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da33b491-4efa-40a7-8864-7615b562423d-openshift-service-ca\") pod \"maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd\" (UID: \"da33b491-4efa-40a7-8864-7615b562423d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd" Jul 08 19:15:11.185880 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:11.185859 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da33b491-4efa-40a7-8864-7615b562423d-maas-api-tls\") pod \"maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd\" (UID: \"da33b491-4efa-40a7-8864-7615b562423d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd" Jul 08 19:15:11.196320 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:11.196278 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8kb6\" (UniqueName: \"kubernetes.io/projected/da33b491-4efa-40a7-8864-7615b562423d-kube-api-access-q8kb6\") pod \"maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd\" (UID: \"da33b491-4efa-40a7-8864-7615b562423d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd" Jul 08 19:15:11.311174 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:11.311128 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd" Jul 08 19:15:11.457672 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:11.457643 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd"] Jul 08 19:15:11.459616 ip-10-0-131-102 kubenswrapper[2575]: W0708 19:15:11.459588 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda33b491_4efa_40a7_8864_7615b562423d.slice/crio-a1025bf0a5cfe03da72f1c8b7b36bc158b0022357e2889e99d1a00e46f5bdcb7 WatchSource:0}: Error finding container a1025bf0a5cfe03da72f1c8b7b36bc158b0022357e2889e99d1a00e46f5bdcb7: Status 404 returned error can't find the container with id a1025bf0a5cfe03da72f1c8b7b36bc158b0022357e2889e99d1a00e46f5bdcb7 Jul 08 19:15:11.789047 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:11.788952 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd" event={"ID":"da33b491-4efa-40a7-8864-7615b562423d","Type":"ContainerStarted","Data":"a1025bf0a5cfe03da72f1c8b7b36bc158b0022357e2889e99d1a00e46f5bdcb7"} Jul 08 19:15:12.795039 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:12.794998 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd" event={"ID":"da33b491-4efa-40a7-8864-7615b562423d","Type":"ContainerStarted","Data":"92dca145050b2a97e173518770c47737ba6d80ab8428034a2455eb33bd26cc26"} Jul 08 19:15:12.846266 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:12.846204 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd" podStartSLOduration=2.508805935 podStartE2EDuration="2.846188344s" totalImagesPullingTime="337.382409ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:15:12.845112973 +0000 UTC m=+2123.682440695" watchObservedRunningTime="2026-07-08 19:15:12.846188344 +0000 UTC m=+2123.683516051" Jul 08 19:15:13.800158 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:13.800125 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd" Jul 08 19:15:19.811570 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:19.811532 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd" Jul 08 19:15:26.799526 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:26.799473 2575 patch_prober.go:28] interesting pod/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 08 19:15:26.799966 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:26.799537 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" podUID="785fe3c4-936e-43a3-a89e-5216265806fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 08 19:15:26.874236 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:26.874201 2575 generic.go:356] "Generic (PLEG): container finished" podID="785fe3c4-936e-43a3-a89e-5216265806fa" containerID="58eb1343bd7ad50717177d0c5923c0a801c1e5b8caa4d523cf127396aed5961f" exitCode=0 Jul 08 19:15:26.874425 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:26.874251 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" event={"ID":"785fe3c4-936e-43a3-a89e-5216265806fa","Type":"ContainerDied","Data":"58eb1343bd7ad50717177d0c5923c0a801c1e5b8caa4d523cf127396aed5961f"} Jul 08 19:15:26.874670 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:26.874651 2575 scope.go:117] "RemoveContainer" containerID="58eb1343bd7ad50717177d0c5923c0a801c1e5b8caa4d523cf127396aed5961f" Jul 08 19:15:26.875702 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:26.875670 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:15:26.875760 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:26.875748 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:15:26.875801 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:26.875782 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 19:15:27.881162 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:27.881124 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" event={"ID":"785fe3c4-936e-43a3-a89e-5216265806fa","Type":"ContainerStarted","Data":"29aeedd36ca2a834ff84d7ad705b2c91d508f4e77db6d3f538ad79d0bbc89d67"} Jul 08 19:15:28.853012 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:28.852967 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:28.853463 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:28.853360 2575 patch_prober.go:28] interesting pod/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 08 19:15:28.853658 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:28.853633 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" podUID="785fe3c4-936e-43a3-a89e-5216265806fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 08 19:15:29.853148 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:29.853095 2575 patch_prober.go:28] interesting pod/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 08 19:15:29.853692 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:29.853171 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" podUID="785fe3c4-936e-43a3-a89e-5216265806fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 08 19:15:30.852797 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:30.852758 2575 patch_prober.go:28] interesting pod/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 08 19:15:30.853114 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:30.853081 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" podUID="785fe3c4-936e-43a3-a89e-5216265806fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 08 19:15:31.813757 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:31.813703 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd"] Jul 08 19:15:31.814220 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:31.813976 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd" podUID="da33b491-4efa-40a7-8864-7615b562423d" containerName="maas-api" containerID="cri-o://92dca145050b2a97e173518770c47737ba6d80ab8428034a2455eb33bd26cc26" gracePeriod=30 Jul 08 19:15:31.852794 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:31.852751 2575 patch_prober.go:28] interesting pod/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 08 19:15:31.852992 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:31.852815 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" podUID="785fe3c4-936e-43a3-a89e-5216265806fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 08 19:15:32.069266 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:32.069204 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd" Jul 08 19:15:32.193147 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:32.193111 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da33b491-4efa-40a7-8864-7615b562423d-openshift-service-ca\") pod \"da33b491-4efa-40a7-8864-7615b562423d\" (UID: \"da33b491-4efa-40a7-8864-7615b562423d\") " Jul 08 19:15:32.193381 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:32.193202 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da33b491-4efa-40a7-8864-7615b562423d-maas-api-tls\") pod \"da33b491-4efa-40a7-8864-7615b562423d\" (UID: \"da33b491-4efa-40a7-8864-7615b562423d\") " Jul 08 19:15:32.193381 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:32.193335 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8kb6\" (UniqueName: \"kubernetes.io/projected/da33b491-4efa-40a7-8864-7615b562423d-kube-api-access-q8kb6\") pod \"da33b491-4efa-40a7-8864-7615b562423d\" (UID: \"da33b491-4efa-40a7-8864-7615b562423d\") " Jul 08 19:15:32.193520 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:32.193502 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da33b491-4efa-40a7-8864-7615b562423d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "da33b491-4efa-40a7-8864-7615b562423d" (UID: "da33b491-4efa-40a7-8864-7615b562423d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:15:32.193699 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:32.193680 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da33b491-4efa-40a7-8864-7615b562423d-openshift-service-ca\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:15:32.195399 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:32.195377 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da33b491-4efa-40a7-8864-7615b562423d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "da33b491-4efa-40a7-8864-7615b562423d" (UID: "da33b491-4efa-40a7-8864-7615b562423d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:15:32.195464 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:32.195403 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da33b491-4efa-40a7-8864-7615b562423d-kube-api-access-q8kb6" (OuterVolumeSpecName: "kube-api-access-q8kb6") pod "da33b491-4efa-40a7-8864-7615b562423d" (UID: "da33b491-4efa-40a7-8864-7615b562423d"). InnerVolumeSpecName "kube-api-access-q8kb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:15:32.294830 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:32.294792 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da33b491-4efa-40a7-8864-7615b562423d-maas-api-tls\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:15:32.294830 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:32.294823 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8kb6\" (UniqueName: \"kubernetes.io/projected/da33b491-4efa-40a7-8864-7615b562423d-kube-api-access-q8kb6\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:15:32.855867 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:32.855836 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:32.856288 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:32.856075 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:32.856772 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:32.856754 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:32.906910 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:32.906873 2575 generic.go:356] "Generic (PLEG): container finished" podID="da33b491-4efa-40a7-8864-7615b562423d" containerID="92dca145050b2a97e173518770c47737ba6d80ab8428034a2455eb33bd26cc26" exitCode=0 Jul 08 19:15:32.907078 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:32.906940 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd" Jul 08 19:15:32.907078 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:32.906975 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd" event={"ID":"da33b491-4efa-40a7-8864-7615b562423d","Type":"ContainerDied","Data":"92dca145050b2a97e173518770c47737ba6d80ab8428034a2455eb33bd26cc26"} Jul 08 19:15:32.907078 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:32.907044 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd" event={"ID":"da33b491-4efa-40a7-8864-7615b562423d","Type":"ContainerDied","Data":"a1025bf0a5cfe03da72f1c8b7b36bc158b0022357e2889e99d1a00e46f5bdcb7"} Jul 08 19:15:32.907078 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:32.907075 2575 scope.go:117] "RemoveContainer" containerID="92dca145050b2a97e173518770c47737ba6d80ab8428034a2455eb33bd26cc26" Jul 08 19:15:32.920080 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:32.920061 2575 scope.go:117] "RemoveContainer" containerID="92dca145050b2a97e173518770c47737ba6d80ab8428034a2455eb33bd26cc26" Jul 08 19:15:32.920433 ip-10-0-131-102 kubenswrapper[2575]: E0708 19:15:32.920406 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92dca145050b2a97e173518770c47737ba6d80ab8428034a2455eb33bd26cc26\": container with ID starting with 92dca145050b2a97e173518770c47737ba6d80ab8428034a2455eb33bd26cc26 not found: ID does not exist" containerID="92dca145050b2a97e173518770c47737ba6d80ab8428034a2455eb33bd26cc26" Jul 08 19:15:32.920540 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:32.920448 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92dca145050b2a97e173518770c47737ba6d80ab8428034a2455eb33bd26cc26"} err="failed to get container status \"92dca145050b2a97e173518770c47737ba6d80ab8428034a2455eb33bd26cc26\": rpc error: code = NotFound desc = could not find container \"92dca145050b2a97e173518770c47737ba6d80ab8428034a2455eb33bd26cc26\": container with ID starting with 92dca145050b2a97e173518770c47737ba6d80ab8428034a2455eb33bd26cc26 not found: ID does not exist" Jul 08 19:15:32.936178 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:32.936150 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd"] Jul 08 19:15:32.941600 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:32.941575 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b742e8-568f57cf5b-n68fd"] Jul 08 19:15:33.780057 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:33.780019 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da33b491-4efa-40a7-8864-7615b562423d" path="/var/lib/kubelet/pods/da33b491-4efa-40a7-8864-7615b562423d/volumes" Jul 08 19:15:37.571179 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:37.571145 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m"] Jul 08 19:15:37.571799 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:37.571386 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" podUID="785fe3c4-936e-43a3-a89e-5216265806fa" containerName="istio-proxy" containerID="cri-o://29aeedd36ca2a834ff84d7ad705b2c91d508f4e77db6d3f538ad79d0bbc89d67" gracePeriod=30 Jul 08 19:15:41.404833 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:41.404621 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1f446e-6d479557f4-82chh"] Jul 08 19:15:41.406288 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:41.406256 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da33b491-4efa-40a7-8864-7615b562423d" containerName="maas-api" Jul 08 19:15:41.406471 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:41.406457 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="da33b491-4efa-40a7-8864-7615b562423d" containerName="maas-api" Jul 08 19:15:41.406805 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:41.406786 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="da33b491-4efa-40a7-8864-7615b562423d" containerName="maas-api" Jul 08 19:15:41.413610 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:41.413580 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1f446e-6d479557f4-82chh" Jul 08 19:15:41.415660 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:41.415637 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1f446e-6d479557f4-82chh"] Jul 08 19:15:41.417476 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:41.417450 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-1f446e-serving-cert\"" Jul 08 19:15:41.479367 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:41.479331 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eefada08-f12d-4e93-88a1-3e710e9bc583-maas-api-tls\") pod \"maas-api-e2e-shared-a-1f446e-6d479557f4-82chh\" (UID: \"eefada08-f12d-4e93-88a1-3e710e9bc583\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1f446e-6d479557f4-82chh" Jul 08 19:15:41.479578 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:41.479486 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eefada08-f12d-4e93-88a1-3e710e9bc583-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1f446e-6d479557f4-82chh\" (UID: \"eefada08-f12d-4e93-88a1-3e710e9bc583\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1f446e-6d479557f4-82chh" Jul 08 19:15:41.479578 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:41.479536 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsflm\" (UniqueName: \"kubernetes.io/projected/eefada08-f12d-4e93-88a1-3e710e9bc583-kube-api-access-zsflm\") pod \"maas-api-e2e-shared-a-1f446e-6d479557f4-82chh\" (UID: \"eefada08-f12d-4e93-88a1-3e710e9bc583\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1f446e-6d479557f4-82chh" Jul 08 19:15:41.580847 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:41.580796 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eefada08-f12d-4e93-88a1-3e710e9bc583-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1f446e-6d479557f4-82chh\" (UID: \"eefada08-f12d-4e93-88a1-3e710e9bc583\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1f446e-6d479557f4-82chh" Jul 08 19:15:41.581045 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:41.580875 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsflm\" (UniqueName: \"kubernetes.io/projected/eefada08-f12d-4e93-88a1-3e710e9bc583-kube-api-access-zsflm\") pod \"maas-api-e2e-shared-a-1f446e-6d479557f4-82chh\" (UID: \"eefada08-f12d-4e93-88a1-3e710e9bc583\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1f446e-6d479557f4-82chh" Jul 08 19:15:41.581045 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:41.580966 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eefada08-f12d-4e93-88a1-3e710e9bc583-maas-api-tls\") pod \"maas-api-e2e-shared-a-1f446e-6d479557f4-82chh\" (UID: \"eefada08-f12d-4e93-88a1-3e710e9bc583\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1f446e-6d479557f4-82chh" Jul 08 19:15:41.581552 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:41.581525 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eefada08-f12d-4e93-88a1-3e710e9bc583-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1f446e-6d479557f4-82chh\" (UID: \"eefada08-f12d-4e93-88a1-3e710e9bc583\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1f446e-6d479557f4-82chh" Jul 08 19:15:41.583551 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:41.583520 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eefada08-f12d-4e93-88a1-3e710e9bc583-maas-api-tls\") pod \"maas-api-e2e-shared-a-1f446e-6d479557f4-82chh\" (UID: \"eefada08-f12d-4e93-88a1-3e710e9bc583\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1f446e-6d479557f4-82chh" Jul 08 19:15:41.589539 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:41.589510 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsflm\" (UniqueName: \"kubernetes.io/projected/eefada08-f12d-4e93-88a1-3e710e9bc583-kube-api-access-zsflm\") pod \"maas-api-e2e-shared-a-1f446e-6d479557f4-82chh\" (UID: \"eefada08-f12d-4e93-88a1-3e710e9bc583\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1f446e-6d479557f4-82chh" Jul 08 19:15:41.726311 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:41.726200 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1f446e-6d479557f4-82chh" Jul 08 19:15:41.870261 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:41.870171 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1f446e-6d479557f4-82chh"] Jul 08 19:15:41.875251 ip-10-0-131-102 kubenswrapper[2575]: W0708 19:15:41.875210 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeefada08_f12d_4e93_88a1_3e710e9bc583.slice/crio-17aec5b646822e12095e4c446e77d8e1033be4e6927c7f7fee9bab01f392166d WatchSource:0}: Error finding container 17aec5b646822e12095e4c446e77d8e1033be4e6927c7f7fee9bab01f392166d: Status 404 returned error can't find the container with id 17aec5b646822e12095e4c446e77d8e1033be4e6927c7f7fee9bab01f392166d Jul 08 19:15:41.952596 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:41.952561 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1f446e-6d479557f4-82chh" event={"ID":"eefada08-f12d-4e93-88a1-3e710e9bc583","Type":"ContainerStarted","Data":"17aec5b646822e12095e4c446e77d8e1033be4e6927c7f7fee9bab01f392166d"} Jul 08 19:15:42.831781 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.831717 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:42.894109 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.894072 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-credential-socket\") pod \"785fe3c4-936e-43a3-a89e-5216265806fa\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " Jul 08 19:15:42.894109 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.894113 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/785fe3c4-936e-43a3-a89e-5216265806fa-istiod-ca-cert\") pod \"785fe3c4-936e-43a3-a89e-5216265806fa\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " Jul 08 19:15:42.894362 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.894135 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsj8v\" (UniqueName: \"kubernetes.io/projected/785fe3c4-936e-43a3-a89e-5216265806fa-kube-api-access-zsj8v\") pod \"785fe3c4-936e-43a3-a89e-5216265806fa\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " Jul 08 19:15:42.894362 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.894165 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/785fe3c4-936e-43a3-a89e-5216265806fa-istio-token\") pod \"785fe3c4-936e-43a3-a89e-5216265806fa\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " Jul 08 19:15:42.894362 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.894217 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-istio-data\") pod \"785fe3c4-936e-43a3-a89e-5216265806fa\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " Jul 08 19:15:42.894362 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.894236 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-workload-socket\") pod \"785fe3c4-936e-43a3-a89e-5216265806fa\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " Jul 08 19:15:42.894362 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.894321 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/785fe3c4-936e-43a3-a89e-5216265806fa-istio-podinfo\") pod \"785fe3c4-936e-43a3-a89e-5216265806fa\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " Jul 08 19:15:42.894362 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.894347 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-istio-envoy\") pod \"785fe3c4-936e-43a3-a89e-5216265806fa\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " Jul 08 19:15:42.894683 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.894375 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-workload-certs\") pod \"785fe3c4-936e-43a3-a89e-5216265806fa\" (UID: \"785fe3c4-936e-43a3-a89e-5216265806fa\") " Jul 08 19:15:42.894683 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.894379 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "785fe3c4-936e-43a3-a89e-5216265806fa" (UID: "785fe3c4-936e-43a3-a89e-5216265806fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:15:42.894683 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.894555 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/785fe3c4-936e-43a3-a89e-5216265806fa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "785fe3c4-936e-43a3-a89e-5216265806fa" (UID: "785fe3c4-936e-43a3-a89e-5216265806fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:15:42.894817 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.894695 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-credential-socket\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:15:42.894817 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.894713 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/785fe3c4-936e-43a3-a89e-5216265806fa-istiod-ca-cert\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:15:42.894817 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.894747 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "785fe3c4-936e-43a3-a89e-5216265806fa" (UID: "785fe3c4-936e-43a3-a89e-5216265806fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:15:42.894817 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.894760 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "785fe3c4-936e-43a3-a89e-5216265806fa" (UID: "785fe3c4-936e-43a3-a89e-5216265806fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:15:42.895312 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.895274 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-istio-data" (OuterVolumeSpecName: "istio-data") pod "785fe3c4-936e-43a3-a89e-5216265806fa" (UID: "785fe3c4-936e-43a3-a89e-5216265806fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:15:42.896642 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.896612 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/785fe3c4-936e-43a3-a89e-5216265806fa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "785fe3c4-936e-43a3-a89e-5216265806fa" (UID: "785fe3c4-936e-43a3-a89e-5216265806fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 19:15:42.896642 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.896630 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/785fe3c4-936e-43a3-a89e-5216265806fa-istio-token" (OuterVolumeSpecName: "istio-token") pod "785fe3c4-936e-43a3-a89e-5216265806fa" (UID: "785fe3c4-936e-43a3-a89e-5216265806fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:15:42.896931 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.896899 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "785fe3c4-936e-43a3-a89e-5216265806fa" (UID: "785fe3c4-936e-43a3-a89e-5216265806fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 19:15:42.896931 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.896910 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/785fe3c4-936e-43a3-a89e-5216265806fa-kube-api-access-zsj8v" (OuterVolumeSpecName: "kube-api-access-zsj8v") pod "785fe3c4-936e-43a3-a89e-5216265806fa" (UID: "785fe3c4-936e-43a3-a89e-5216265806fa"). InnerVolumeSpecName "kube-api-access-zsj8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:15:42.958961 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.958921 2575 generic.go:356] "Generic (PLEG): container finished" podID="785fe3c4-936e-43a3-a89e-5216265806fa" containerID="29aeedd36ca2a834ff84d7ad705b2c91d508f4e77db6d3f538ad79d0bbc89d67" exitCode=0 Jul 08 19:15:42.959151 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.958972 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" Jul 08 19:15:42.959151 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.959006 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" event={"ID":"785fe3c4-936e-43a3-a89e-5216265806fa","Type":"ContainerDied","Data":"29aeedd36ca2a834ff84d7ad705b2c91d508f4e77db6d3f538ad79d0bbc89d67"} Jul 08 19:15:42.959151 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.959062 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m" event={"ID":"785fe3c4-936e-43a3-a89e-5216265806fa","Type":"ContainerDied","Data":"d663db6bb3174f8f7e14835881e20cb9156bd56d2bd71d4606ef12959cdafa87"} Jul 08 19:15:42.959151 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.959094 2575 scope.go:117] "RemoveContainer" containerID="29aeedd36ca2a834ff84d7ad705b2c91d508f4e77db6d3f538ad79d0bbc89d67" Jul 08 19:15:42.960667 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.960638 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1f446e-6d479557f4-82chh" event={"ID":"eefada08-f12d-4e93-88a1-3e710e9bc583","Type":"ContainerStarted","Data":"27c64f75c111661fa64fca7966d71929df70c89daaf2deb31c99f644e80c76e2"} Jul 08 19:15:42.960796 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.960688 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1f446e-6d479557f4-82chh" Jul 08 19:15:42.968533 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.968518 2575 scope.go:117] "RemoveContainer" containerID="58eb1343bd7ad50717177d0c5923c0a801c1e5b8caa4d523cf127396aed5961f" Jul 08 19:15:42.976366 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.976351 2575 scope.go:117] "RemoveContainer" containerID="29aeedd36ca2a834ff84d7ad705b2c91d508f4e77db6d3f538ad79d0bbc89d67" Jul 08 19:15:42.976604 ip-10-0-131-102 kubenswrapper[2575]: E0708 19:15:42.976586 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29aeedd36ca2a834ff84d7ad705b2c91d508f4e77db6d3f538ad79d0bbc89d67\": container with ID starting with 29aeedd36ca2a834ff84d7ad705b2c91d508f4e77db6d3f538ad79d0bbc89d67 not found: ID does not exist" containerID="29aeedd36ca2a834ff84d7ad705b2c91d508f4e77db6d3f538ad79d0bbc89d67" Jul 08 19:15:42.976670 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.976611 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29aeedd36ca2a834ff84d7ad705b2c91d508f4e77db6d3f538ad79d0bbc89d67"} err="failed to get container status \"29aeedd36ca2a834ff84d7ad705b2c91d508f4e77db6d3f538ad79d0bbc89d67\": rpc error: code = NotFound desc = could not find container \"29aeedd36ca2a834ff84d7ad705b2c91d508f4e77db6d3f538ad79d0bbc89d67\": container with ID starting with 29aeedd36ca2a834ff84d7ad705b2c91d508f4e77db6d3f538ad79d0bbc89d67 not found: ID does not exist" Jul 08 19:15:42.976670 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.976627 2575 scope.go:117] "RemoveContainer" containerID="58eb1343bd7ad50717177d0c5923c0a801c1e5b8caa4d523cf127396aed5961f" Jul 08 19:15:42.976838 ip-10-0-131-102 kubenswrapper[2575]: E0708 19:15:42.976819 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58eb1343bd7ad50717177d0c5923c0a801c1e5b8caa4d523cf127396aed5961f\": container with ID starting with 58eb1343bd7ad50717177d0c5923c0a801c1e5b8caa4d523cf127396aed5961f not found: ID does not exist" containerID="58eb1343bd7ad50717177d0c5923c0a801c1e5b8caa4d523cf127396aed5961f" Jul 08 19:15:42.976877 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.976842 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58eb1343bd7ad50717177d0c5923c0a801c1e5b8caa4d523cf127396aed5961f"} err="failed to get container status \"58eb1343bd7ad50717177d0c5923c0a801c1e5b8caa4d523cf127396aed5961f\": rpc error: code = NotFound desc = could not find container \"58eb1343bd7ad50717177d0c5923c0a801c1e5b8caa4d523cf127396aed5961f\": container with ID starting with 58eb1343bd7ad50717177d0c5923c0a801c1e5b8caa4d523cf127396aed5961f not found: ID does not exist" Jul 08 19:15:42.989214 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.989188 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m"] Jul 08 19:15:42.994159 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.994136 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b742e8-openshift-default-5f48f5db65-br52m"] Jul 08 19:15:42.995612 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.995589 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-istio-data\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:15:42.995751 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.995616 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-workload-socket\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:15:42.995751 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.995630 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/785fe3c4-936e-43a3-a89e-5216265806fa-istio-podinfo\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:15:42.995751 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.995643 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-istio-envoy\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:15:42.995751 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.995655 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/785fe3c4-936e-43a3-a89e-5216265806fa-workload-certs\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:15:42.995751 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.995668 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsj8v\" (UniqueName: \"kubernetes.io/projected/785fe3c4-936e-43a3-a89e-5216265806fa-kube-api-access-zsj8v\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:15:42.995751 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:42.995681 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/785fe3c4-936e-43a3-a89e-5216265806fa-istio-token\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:15:43.031272 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:43.031218 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1f446e-6d479557f4-82chh" podStartSLOduration=1.5956956839999998 podStartE2EDuration="2.031200518s" totalImagesPullingTime="435.504834ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:15:43.029259543 +0000 UTC m=+2153.866587252" watchObservedRunningTime="2026-07-08 19:15:43.031200518 +0000 UTC m=+2153.868528242" Jul 08 19:15:43.780372 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:43.780332 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="785fe3c4-936e-43a3-a89e-5216265806fa" path="/var/lib/kubelet/pods/785fe3c4-936e-43a3-a89e-5216265806fa/volumes" Jul 08 19:15:48.972595 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:48.972565 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1f446e-6d479557f4-82chh" Jul 08 19:15:49.411393 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:49.411354 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp"] Jul 08 19:15:49.411972 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:49.411950 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="785fe3c4-936e-43a3-a89e-5216265806fa" containerName="istio-proxy" Jul 08 19:15:49.412035 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:49.411976 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="785fe3c4-936e-43a3-a89e-5216265806fa" containerName="istio-proxy" Jul 08 19:15:49.412035 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:49.412006 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="785fe3c4-936e-43a3-a89e-5216265806fa" containerName="istio-proxy" Jul 08 19:15:49.412035 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:49.412015 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="785fe3c4-936e-43a3-a89e-5216265806fa" containerName="istio-proxy" Jul 08 19:15:49.412181 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:49.412117 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="785fe3c4-936e-43a3-a89e-5216265806fa" containerName="istio-proxy" Jul 08 19:15:49.412181 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:49.412131 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="785fe3c4-936e-43a3-a89e-5216265806fa" containerName="istio-proxy" Jul 08 19:15:49.417015 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:49.416987 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp" Jul 08 19:15:49.419867 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:49.419841 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-824e89-serving-cert\"" Jul 08 19:15:49.423866 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:49.423839 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp"] Jul 08 19:15:49.459596 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:49.459555 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb8810d-7e99-4f63-a771-48f5146fb854-openshift-service-ca\") pod \"maas-api-e2e-shared-b-824e89-5556675655-ht7qp\" (UID: \"0bb8810d-7e99-4f63-a771-48f5146fb854\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp" Jul 08 19:15:49.459596 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:49.459601 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgw55\" (UniqueName: \"kubernetes.io/projected/0bb8810d-7e99-4f63-a771-48f5146fb854-kube-api-access-bgw55\") pod \"maas-api-e2e-shared-b-824e89-5556675655-ht7qp\" (UID: \"0bb8810d-7e99-4f63-a771-48f5146fb854\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp" Jul 08 19:15:49.459828 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:49.459686 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bb8810d-7e99-4f63-a771-48f5146fb854-maas-api-tls\") pod \"maas-api-e2e-shared-b-824e89-5556675655-ht7qp\" (UID: \"0bb8810d-7e99-4f63-a771-48f5146fb854\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp" Jul 08 19:15:49.560854 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:49.560824 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bb8810d-7e99-4f63-a771-48f5146fb854-maas-api-tls\") pod \"maas-api-e2e-shared-b-824e89-5556675655-ht7qp\" (UID: \"0bb8810d-7e99-4f63-a771-48f5146fb854\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp" Jul 08 19:15:49.561022 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:49.560891 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb8810d-7e99-4f63-a771-48f5146fb854-openshift-service-ca\") pod \"maas-api-e2e-shared-b-824e89-5556675655-ht7qp\" (UID: \"0bb8810d-7e99-4f63-a771-48f5146fb854\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp" Jul 08 19:15:49.561022 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:49.560919 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgw55\" (UniqueName: \"kubernetes.io/projected/0bb8810d-7e99-4f63-a771-48f5146fb854-kube-api-access-bgw55\") pod \"maas-api-e2e-shared-b-824e89-5556675655-ht7qp\" (UID: \"0bb8810d-7e99-4f63-a771-48f5146fb854\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp" Jul 08 19:15:49.561687 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:49.561663 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb8810d-7e99-4f63-a771-48f5146fb854-openshift-service-ca\") pod \"maas-api-e2e-shared-b-824e89-5556675655-ht7qp\" (UID: \"0bb8810d-7e99-4f63-a771-48f5146fb854\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp" Jul 08 19:15:49.563675 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:49.563654 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-824e89-serving-cert\"" Jul 08 19:15:49.571160 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:49.571136 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgw55\" (UniqueName: \"kubernetes.io/projected/0bb8810d-7e99-4f63-a771-48f5146fb854-kube-api-access-bgw55\") pod \"maas-api-e2e-shared-b-824e89-5556675655-ht7qp\" (UID: \"0bb8810d-7e99-4f63-a771-48f5146fb854\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp" Jul 08 19:15:49.571251 ip-10-0-131-102 kubenswrapper[2575]: E0708 19:15:49.571196 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-serving-cert: secret "maas-api-e2e-shared-b-824e89-serving-cert" not found Jul 08 19:15:49.571321 ip-10-0-131-102 kubenswrapper[2575]: E0708 19:15:49.571262 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bb8810d-7e99-4f63-a771-48f5146fb854-maas-api-tls podName:0bb8810d-7e99-4f63-a771-48f5146fb854 nodeName:}" failed. No retries permitted until 2026-07-08 19:15:50.071243716 +0000 UTC m=+2160.908571408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0bb8810d-7e99-4f63-a771-48f5146fb854-maas-api-tls") pod "maas-api-e2e-shared-b-824e89-5556675655-ht7qp" (UID: "0bb8810d-7e99-4f63-a771-48f5146fb854") : secret "maas-api-e2e-shared-b-824e89-serving-cert" not found Jul 08 19:15:50.167745 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:50.167649 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bb8810d-7e99-4f63-a771-48f5146fb854-maas-api-tls\") pod \"maas-api-e2e-shared-b-824e89-5556675655-ht7qp\" (UID: \"0bb8810d-7e99-4f63-a771-48f5146fb854\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp" Jul 08 19:15:50.170100 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:50.170076 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bb8810d-7e99-4f63-a771-48f5146fb854-maas-api-tls\") pod \"maas-api-e2e-shared-b-824e89-5556675655-ht7qp\" (UID: \"0bb8810d-7e99-4f63-a771-48f5146fb854\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp" Jul 08 19:15:50.331645 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:50.331607 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp" Jul 08 19:15:50.473875 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:50.473847 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp"] Jul 08 19:15:50.475883 ip-10-0-131-102 kubenswrapper[2575]: W0708 19:15:50.475852 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb8810d_7e99_4f63_a771_48f5146fb854.slice/crio-522fa7e93ca49f4cc2f6e12580c01c35680892998d46d615bc87d2fc8f3ca6fd WatchSource:0}: Error finding container 522fa7e93ca49f4cc2f6e12580c01c35680892998d46d615bc87d2fc8f3ca6fd: Status 404 returned error can't find the container with id 522fa7e93ca49f4cc2f6e12580c01c35680892998d46d615bc87d2fc8f3ca6fd Jul 08 19:15:51.009011 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:51.008973 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp" event={"ID":"0bb8810d-7e99-4f63-a771-48f5146fb854","Type":"ContainerStarted","Data":"614373dc4e2ccc74415eabbf2d152c81f4c6aea20bc005b07411dc00930965d6"} Jul 08 19:15:51.009207 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:51.009049 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp" event={"ID":"0bb8810d-7e99-4f63-a771-48f5146fb854","Type":"ContainerStarted","Data":"522fa7e93ca49f4cc2f6e12580c01c35680892998d46d615bc87d2fc8f3ca6fd"} Jul 08 19:15:51.009207 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:51.009117 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp" Jul 08 19:15:51.029311 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:51.029243 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp" podStartSLOduration=1.649646728 podStartE2EDuration="2.029226544s" totalImagesPullingTime="379.579816ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 19:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:15:51.027894588 +0000 UTC m=+2161.865222297" watchObservedRunningTime="2026-07-08 19:15:51.029226544 +0000 UTC m=+2161.866554253" Jul 08 19:15:57.020043 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:15:57.020013 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp" Jul 08 19:18:54.954781 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:18:54.954746 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1f446e-6d479557f4-82chh"] Jul 08 19:18:54.955621 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:18:54.954968 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1f446e-6d479557f4-82chh" podUID="eefada08-f12d-4e93-88a1-3e710e9bc583" containerName="maas-api" containerID="cri-o://27c64f75c111661fa64fca7966d71929df70c89daaf2deb31c99f644e80c76e2" gracePeriod=30 Jul 08 19:18:55.207235 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:18:55.207169 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1f446e-6d479557f4-82chh" Jul 08 19:18:55.353409 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:18:55.353375 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsflm\" (UniqueName: \"kubernetes.io/projected/eefada08-f12d-4e93-88a1-3e710e9bc583-kube-api-access-zsflm\") pod \"eefada08-f12d-4e93-88a1-3e710e9bc583\" (UID: \"eefada08-f12d-4e93-88a1-3e710e9bc583\") " Jul 08 19:18:55.353561 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:18:55.353422 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eefada08-f12d-4e93-88a1-3e710e9bc583-openshift-service-ca\") pod \"eefada08-f12d-4e93-88a1-3e710e9bc583\" (UID: \"eefada08-f12d-4e93-88a1-3e710e9bc583\") " Jul 08 19:18:55.353561 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:18:55.353461 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eefada08-f12d-4e93-88a1-3e710e9bc583-maas-api-tls\") pod \"eefada08-f12d-4e93-88a1-3e710e9bc583\" (UID: \"eefada08-f12d-4e93-88a1-3e710e9bc583\") " Jul 08 19:18:55.353870 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:18:55.353841 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eefada08-f12d-4e93-88a1-3e710e9bc583-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eefada08-f12d-4e93-88a1-3e710e9bc583" (UID: "eefada08-f12d-4e93-88a1-3e710e9bc583"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:18:55.355614 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:18:55.355590 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eefada08-f12d-4e93-88a1-3e710e9bc583-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eefada08-f12d-4e93-88a1-3e710e9bc583" (UID: "eefada08-f12d-4e93-88a1-3e710e9bc583"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:18:55.355690 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:18:55.355658 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eefada08-f12d-4e93-88a1-3e710e9bc583-kube-api-access-zsflm" (OuterVolumeSpecName: "kube-api-access-zsflm") pod "eefada08-f12d-4e93-88a1-3e710e9bc583" (UID: "eefada08-f12d-4e93-88a1-3e710e9bc583"). InnerVolumeSpecName "kube-api-access-zsflm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:18:55.454723 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:18:55.454688 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eefada08-f12d-4e93-88a1-3e710e9bc583-maas-api-tls\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:18:55.454723 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:18:55.454718 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsflm\" (UniqueName: \"kubernetes.io/projected/eefada08-f12d-4e93-88a1-3e710e9bc583-kube-api-access-zsflm\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:18:55.454723 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:18:55.454729 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eefada08-f12d-4e93-88a1-3e710e9bc583-openshift-service-ca\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:18:55.914410 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:18:55.914375 2575 generic.go:356] "Generic (PLEG): container finished" podID="eefada08-f12d-4e93-88a1-3e710e9bc583" containerID="27c64f75c111661fa64fca7966d71929df70c89daaf2deb31c99f644e80c76e2" exitCode=0 Jul 08 19:18:55.914598 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:18:55.914418 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1f446e-6d479557f4-82chh" Jul 08 19:18:55.914598 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:18:55.914451 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1f446e-6d479557f4-82chh" event={"ID":"eefada08-f12d-4e93-88a1-3e710e9bc583","Type":"ContainerDied","Data":"27c64f75c111661fa64fca7966d71929df70c89daaf2deb31c99f644e80c76e2"} Jul 08 19:18:55.914598 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:18:55.914505 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1f446e-6d479557f4-82chh" event={"ID":"eefada08-f12d-4e93-88a1-3e710e9bc583","Type":"ContainerDied","Data":"17aec5b646822e12095e4c446e77d8e1033be4e6927c7f7fee9bab01f392166d"} Jul 08 19:18:55.914598 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:18:55.914526 2575 scope.go:117] "RemoveContainer" containerID="27c64f75c111661fa64fca7966d71929df70c89daaf2deb31c99f644e80c76e2" Jul 08 19:18:55.923168 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:18:55.923151 2575 scope.go:117] "RemoveContainer" containerID="27c64f75c111661fa64fca7966d71929df70c89daaf2deb31c99f644e80c76e2" Jul 08 19:18:55.923467 ip-10-0-131-102 kubenswrapper[2575]: E0708 19:18:55.923445 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c64f75c111661fa64fca7966d71929df70c89daaf2deb31c99f644e80c76e2\": container with ID starting with 27c64f75c111661fa64fca7966d71929df70c89daaf2deb31c99f644e80c76e2 not found: ID does not exist" containerID="27c64f75c111661fa64fca7966d71929df70c89daaf2deb31c99f644e80c76e2" Jul 08 19:18:55.923533 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:18:55.923475 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c64f75c111661fa64fca7966d71929df70c89daaf2deb31c99f644e80c76e2"} err="failed to get container status \"27c64f75c111661fa64fca7966d71929df70c89daaf2deb31c99f644e80c76e2\": rpc error: code = NotFound desc = could not find container \"27c64f75c111661fa64fca7966d71929df70c89daaf2deb31c99f644e80c76e2\": container with ID starting with 27c64f75c111661fa64fca7966d71929df70c89daaf2deb31c99f644e80c76e2 not found: ID does not exist" Jul 08 19:18:55.932804 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:18:55.932782 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1f446e-6d479557f4-82chh"] Jul 08 19:18:55.936732 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:18:55.936712 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1f446e-6d479557f4-82chh"] Jul 08 19:18:57.781026 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:18:57.780988 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eefada08-f12d-4e93-88a1-3e710e9bc583" path="/var/lib/kubelet/pods/eefada08-f12d-4e93-88a1-3e710e9bc583/volumes" Jul 08 19:19:01.977043 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:01.977012 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp"] Jul 08 19:19:01.977499 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:01.977234 2575 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp" podUID="0bb8810d-7e99-4f63-a771-48f5146fb854" containerName="maas-api" containerID="cri-o://614373dc4e2ccc74415eabbf2d152c81f4c6aea20bc005b07411dc00930965d6" gracePeriod=30 Jul 08 19:19:02.014972 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:02.014932 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp" podUID="0bb8810d-7e99-4f63-a771-48f5146fb854" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.55:8443/health\": dial tcp 10.134.0.55:8443: connect: connection refused" Jul 08 19:19:02.225520 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:02.225494 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp" Jul 08 19:19:02.316905 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:02.316813 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgw55\" (UniqueName: \"kubernetes.io/projected/0bb8810d-7e99-4f63-a771-48f5146fb854-kube-api-access-bgw55\") pod \"0bb8810d-7e99-4f63-a771-48f5146fb854\" (UID: \"0bb8810d-7e99-4f63-a771-48f5146fb854\") " Jul 08 19:19:02.317089 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:02.316919 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb8810d-7e99-4f63-a771-48f5146fb854-openshift-service-ca\") pod \"0bb8810d-7e99-4f63-a771-48f5146fb854\" (UID: \"0bb8810d-7e99-4f63-a771-48f5146fb854\") " Jul 08 19:19:02.317089 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:02.316949 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bb8810d-7e99-4f63-a771-48f5146fb854-maas-api-tls\") pod \"0bb8810d-7e99-4f63-a771-48f5146fb854\" (UID: \"0bb8810d-7e99-4f63-a771-48f5146fb854\") " Jul 08 19:19:02.317276 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:02.317245 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb8810d-7e99-4f63-a771-48f5146fb854-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0bb8810d-7e99-4f63-a771-48f5146fb854" (UID: "0bb8810d-7e99-4f63-a771-48f5146fb854"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 19:19:02.319016 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:02.318986 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb8810d-7e99-4f63-a771-48f5146fb854-kube-api-access-bgw55" (OuterVolumeSpecName: "kube-api-access-bgw55") pod "0bb8810d-7e99-4f63-a771-48f5146fb854" (UID: "0bb8810d-7e99-4f63-a771-48f5146fb854"). InnerVolumeSpecName "kube-api-access-bgw55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 19:19:02.319124 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:02.319030 2575 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bb8810d-7e99-4f63-a771-48f5146fb854-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0bb8810d-7e99-4f63-a771-48f5146fb854" (UID: "0bb8810d-7e99-4f63-a771-48f5146fb854"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 19:19:02.418650 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:02.418599 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bb8810d-7e99-4f63-a771-48f5146fb854-openshift-service-ca\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:19:02.418650 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:02.418642 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0bb8810d-7e99-4f63-a771-48f5146fb854-maas-api-tls\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:19:02.418650 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:02.418652 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgw55\" (UniqueName: \"kubernetes.io/projected/0bb8810d-7e99-4f63-a771-48f5146fb854-kube-api-access-bgw55\") on node \"ip-10-0-131-102.ec2.internal\" DevicePath \"\"" Jul 08 19:19:02.955421 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:02.955389 2575 generic.go:356] "Generic (PLEG): container finished" podID="0bb8810d-7e99-4f63-a771-48f5146fb854" containerID="614373dc4e2ccc74415eabbf2d152c81f4c6aea20bc005b07411dc00930965d6" exitCode=0 Jul 08 19:19:02.955589 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:02.955435 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp" Jul 08 19:19:02.955589 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:02.955453 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp" event={"ID":"0bb8810d-7e99-4f63-a771-48f5146fb854","Type":"ContainerDied","Data":"614373dc4e2ccc74415eabbf2d152c81f4c6aea20bc005b07411dc00930965d6"} Jul 08 19:19:02.955589 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:02.955497 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp" event={"ID":"0bb8810d-7e99-4f63-a771-48f5146fb854","Type":"ContainerDied","Data":"522fa7e93ca49f4cc2f6e12580c01c35680892998d46d615bc87d2fc8f3ca6fd"} Jul 08 19:19:02.955589 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:02.955517 2575 scope.go:117] "RemoveContainer" containerID="614373dc4e2ccc74415eabbf2d152c81f4c6aea20bc005b07411dc00930965d6" Jul 08 19:19:02.964168 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:02.964148 2575 scope.go:117] "RemoveContainer" containerID="614373dc4e2ccc74415eabbf2d152c81f4c6aea20bc005b07411dc00930965d6" Jul 08 19:19:02.964448 ip-10-0-131-102 kubenswrapper[2575]: E0708 19:19:02.964429 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"614373dc4e2ccc74415eabbf2d152c81f4c6aea20bc005b07411dc00930965d6\": container with ID starting with 614373dc4e2ccc74415eabbf2d152c81f4c6aea20bc005b07411dc00930965d6 not found: ID does not exist" containerID="614373dc4e2ccc74415eabbf2d152c81f4c6aea20bc005b07411dc00930965d6" Jul 08 19:19:02.964517 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:02.964456 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"614373dc4e2ccc74415eabbf2d152c81f4c6aea20bc005b07411dc00930965d6"} err="failed to get container status \"614373dc4e2ccc74415eabbf2d152c81f4c6aea20bc005b07411dc00930965d6\": rpc error: code = NotFound desc = could not find container \"614373dc4e2ccc74415eabbf2d152c81f4c6aea20bc005b07411dc00930965d6\": container with ID starting with 614373dc4e2ccc74415eabbf2d152c81f4c6aea20bc005b07411dc00930965d6 not found: ID does not exist" Jul 08 19:19:02.977182 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:02.977160 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp"] Jul 08 19:19:02.980525 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:02.980501 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-824e89-5556675655-ht7qp"] Jul 08 19:19:03.781242 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:03.781207 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bb8810d-7e99-4f63-a771-48f5146fb854" path="/var/lib/kubelet/pods/0bb8810d-7e99-4f63-a771-48f5146fb854/volumes" Jul 08 19:19:13.863450 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:13.863418 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6449898b5-4p9q2_1aa899cd-b902-47df-a8e7-cc85e05e1886/manager/0.log" Jul 08 19:19:14.129320 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:14.129217 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8vw4w_eeae891b-9bad-4f61-915c-f0f4cac1bc18/manager/0.log" Jul 08 19:19:14.529601 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:14.529501 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c5cf545b9-r59jn_34dd0db6-dd56-49aa-8302-1d90fd9c31cf/manager/0.log" Jul 08 19:19:14.535223 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:14.535204 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c5cf545b9-r59jn_34dd0db6-dd56-49aa-8302-1d90fd9c31cf/copy-manifests/0.log" Jul 08 19:19:16.147319 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:16.147233 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-2zjk4_66b826e7-33cf-4a39-a0b9-d9615dd3f360/manager/0.log" Jul 08 19:19:16.298671 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:16.298643 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-f4ps5_5238549b-03a9-48be-844a-9c9e0fc9d722/manager/0.log" Jul 08 19:19:16.584903 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:16.584823 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-9b9mq_2b0b7393-a76e-44cd-82aa-3a19ed4ce0bc/registry-server/0.log" Jul 08 19:19:16.980165 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:16.980137 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-m8gzx_ec105bbe-ad50-426a-bf2d-c73942258586/manager/0.log" Jul 08 19:19:17.512898 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:17.512863 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-w9rg6_362ce324-f216-489b-89c5-ddd7244f8fbd/discovery/0.log" Jul 08 19:19:17.763631 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:17.763545 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8b75dd56d-wk2ph_dc05552e-bd5b-46e6-95df-b70171c218f5/kube-auth-proxy/0.log" Jul 08 19:19:17.904952 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:17.904921 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-w6zn2_926ae9db-d53a-44d9-93ec-c93c16c604b2/istio-proxy/1.log" Jul 08 19:19:18.041272 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:18.041195 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-plb8f_dbed1f82-e4b6-4cdc-a68c-40dcf2448650/payload-pre-processing/0.log" Jul 08 19:19:18.167964 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:18.167937 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-qcc7s_70b85c5d-9a96-4b41-a062-8de6a4809609/payload-processing/0.log" Jul 08 19:19:18.697613 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:18.697585 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-msj5l_247aede6-ea5e-4f0f-9748-b37f16641220/main/0.log" Jul 08 19:19:18.943724 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:18.943697 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-6lspn_0d677253-f719-46ba-8f87-65708ec5b045/main/0.log" Jul 08 19:19:19.210033 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:19.210004 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-r6krp_4c183697-cbdd-4a2c-8ee6-b791aa6d7582/main/0.log" Jul 08 19:19:26.540495 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:26.540470 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-szlwf_610063ae-dbb1-4a58-bacd-06bf90518168/global-pull-secret-syncer/0.log" Jul 08 19:19:26.643864 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:26.643836 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v7qvx_d61025be-679c-46f7-9a55-5049757d2d53/konnectivity-agent/0.log" Jul 08 19:19:26.685796 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:26.685772 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-102.ec2.internal_a4bb240418508e65be90c2a28bf12044/haproxy/0.log" Jul 08 19:19:31.323598 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:31.323569 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-2zjk4_66b826e7-33cf-4a39-a0b9-d9615dd3f360/manager/0.log" Jul 08 19:19:31.364220 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:31.364183 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-f4ps5_5238549b-03a9-48be-844a-9c9e0fc9d722/manager/0.log" Jul 08 19:19:31.426876 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:31.426843 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-9b9mq_2b0b7393-a76e-44cd-82aa-3a19ed4ce0bc/registry-server/0.log" Jul 08 19:19:31.621827 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:31.621793 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-m8gzx_ec105bbe-ad50-426a-bf2d-c73942258586/manager/0.log" Jul 08 19:19:33.393639 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:33.393610 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-7drlq_3cdaf3eb-a0ab-4001-a1d6-489fad86e0a7/cluster-monitoring-operator/0.log" Jul 08 19:19:33.603315 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:33.603279 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-hckvj_dc93f7a0-7dd6-4c0e-b213-68624306f061/monitoring-plugin/0.log" Jul 08 19:19:33.684700 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:33.684630 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gx7nm_6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286/node-exporter/0.log" Jul 08 19:19:33.723440 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:33.723410 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gx7nm_6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286/kube-rbac-proxy/0.log" Jul 08 19:19:33.748342 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:33.748314 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gx7nm_6e28f6a7-ea12-4dfb-a1f1-fb7a1e2d4286/init-textfile/0.log" Jul 08 19:19:35.856163 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:35.856129 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-qcqpm_f18a01b1-7095-480e-ac45-e3a3cce3cfe5/networking-console-plugin/0.log" Jul 08 19:19:36.713208 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:36.713174 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-prwc8/perf-node-gather-daemonset-m9zcl"] Jul 08 19:19:36.713565 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:36.713551 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bb8810d-7e99-4f63-a771-48f5146fb854" containerName="maas-api" Jul 08 19:19:36.713619 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:36.713566 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bb8810d-7e99-4f63-a771-48f5146fb854" containerName="maas-api" Jul 08 19:19:36.713619 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:36.713578 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eefada08-f12d-4e93-88a1-3e710e9bc583" containerName="maas-api" Jul 08 19:19:36.713619 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:36.713584 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="eefada08-f12d-4e93-88a1-3e710e9bc583" containerName="maas-api" Jul 08 19:19:36.713713 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:36.713645 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="eefada08-f12d-4e93-88a1-3e710e9bc583" containerName="maas-api" Jul 08 19:19:36.713713 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:36.713655 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bb8810d-7e99-4f63-a771-48f5146fb854" containerName="maas-api" Jul 08 19:19:36.716322 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:36.716289 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-prwc8/perf-node-gather-daemonset-m9zcl" Jul 08 19:19:36.719197 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:36.719173 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-prwc8\"/\"openshift-service-ca.crt\"" Jul 08 19:19:36.720488 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:36.720467 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-prwc8\"/\"default-dockercfg-q5xj4\"" Jul 08 19:19:36.720596 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:36.720477 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-prwc8\"/\"kube-root-ca.crt\"" Jul 08 19:19:36.727080 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:36.727058 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-prwc8/perf-node-gather-daemonset-m9zcl"] Jul 08 19:19:36.819681 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:36.819635 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1c5c29c2-3215-4800-a45e-f0cf69601df3-proc\") pod \"perf-node-gather-daemonset-m9zcl\" (UID: \"1c5c29c2-3215-4800-a45e-f0cf69601df3\") " pod="openshift-must-gather-prwc8/perf-node-gather-daemonset-m9zcl" Jul 08 19:19:36.819897 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:36.819687 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c5c29c2-3215-4800-a45e-f0cf69601df3-lib-modules\") pod \"perf-node-gather-daemonset-m9zcl\" (UID: \"1c5c29c2-3215-4800-a45e-f0cf69601df3\") " pod="openshift-must-gather-prwc8/perf-node-gather-daemonset-m9zcl" Jul 08 19:19:36.819897 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:36.819766 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdntx\" (UniqueName: \"kubernetes.io/projected/1c5c29c2-3215-4800-a45e-f0cf69601df3-kube-api-access-cdntx\") pod \"perf-node-gather-daemonset-m9zcl\" (UID: \"1c5c29c2-3215-4800-a45e-f0cf69601df3\") " pod="openshift-must-gather-prwc8/perf-node-gather-daemonset-m9zcl" Jul 08 19:19:36.819897 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:36.819792 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c5c29c2-3215-4800-a45e-f0cf69601df3-sys\") pod \"perf-node-gather-daemonset-m9zcl\" (UID: \"1c5c29c2-3215-4800-a45e-f0cf69601df3\") " pod="openshift-must-gather-prwc8/perf-node-gather-daemonset-m9zcl" Jul 08 19:19:36.819897 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:36.819841 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1c5c29c2-3215-4800-a45e-f0cf69601df3-podres\") pod \"perf-node-gather-daemonset-m9zcl\" (UID: \"1c5c29c2-3215-4800-a45e-f0cf69601df3\") " pod="openshift-must-gather-prwc8/perf-node-gather-daemonset-m9zcl" Jul 08 19:19:36.921250 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:36.921206 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdntx\" (UniqueName: \"kubernetes.io/projected/1c5c29c2-3215-4800-a45e-f0cf69601df3-kube-api-access-cdntx\") pod \"perf-node-gather-daemonset-m9zcl\" (UID: \"1c5c29c2-3215-4800-a45e-f0cf69601df3\") " pod="openshift-must-gather-prwc8/perf-node-gather-daemonset-m9zcl" Jul 08 19:19:36.921714 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:36.921255 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c5c29c2-3215-4800-a45e-f0cf69601df3-sys\") pod \"perf-node-gather-daemonset-m9zcl\" (UID: \"1c5c29c2-3215-4800-a45e-f0cf69601df3\") " pod="openshift-must-gather-prwc8/perf-node-gather-daemonset-m9zcl" Jul 08 19:19:36.921714 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:36.921339 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1c5c29c2-3215-4800-a45e-f0cf69601df3-podres\") pod \"perf-node-gather-daemonset-m9zcl\" (UID: \"1c5c29c2-3215-4800-a45e-f0cf69601df3\") " pod="openshift-must-gather-prwc8/perf-node-gather-daemonset-m9zcl" Jul 08 19:19:36.921714 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:36.921413 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1c5c29c2-3215-4800-a45e-f0cf69601df3-proc\") pod \"perf-node-gather-daemonset-m9zcl\" (UID: \"1c5c29c2-3215-4800-a45e-f0cf69601df3\") " pod="openshift-must-gather-prwc8/perf-node-gather-daemonset-m9zcl" Jul 08 19:19:36.921714 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:36.921422 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1c5c29c2-3215-4800-a45e-f0cf69601df3-sys\") pod \"perf-node-gather-daemonset-m9zcl\" (UID: \"1c5c29c2-3215-4800-a45e-f0cf69601df3\") " pod="openshift-must-gather-prwc8/perf-node-gather-daemonset-m9zcl" Jul 08 19:19:36.921714 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:36.921451 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c5c29c2-3215-4800-a45e-f0cf69601df3-lib-modules\") pod \"perf-node-gather-daemonset-m9zcl\" (UID: \"1c5c29c2-3215-4800-a45e-f0cf69601df3\") " pod="openshift-must-gather-prwc8/perf-node-gather-daemonset-m9zcl" Jul 08 19:19:36.921714 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:36.921508 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1c5c29c2-3215-4800-a45e-f0cf69601df3-proc\") pod \"perf-node-gather-daemonset-m9zcl\" (UID: \"1c5c29c2-3215-4800-a45e-f0cf69601df3\") " pod="openshift-must-gather-prwc8/perf-node-gather-daemonset-m9zcl" Jul 08 19:19:36.921714 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:36.921566 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1c5c29c2-3215-4800-a45e-f0cf69601df3-podres\") pod \"perf-node-gather-daemonset-m9zcl\" (UID: \"1c5c29c2-3215-4800-a45e-f0cf69601df3\") " pod="openshift-must-gather-prwc8/perf-node-gather-daemonset-m9zcl" Jul 08 19:19:36.921714 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:36.921588 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1c5c29c2-3215-4800-a45e-f0cf69601df3-lib-modules\") pod \"perf-node-gather-daemonset-m9zcl\" (UID: \"1c5c29c2-3215-4800-a45e-f0cf69601df3\") " pod="openshift-must-gather-prwc8/perf-node-gather-daemonset-m9zcl" Jul 08 19:19:36.929145 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:36.929128 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdntx\" (UniqueName: \"kubernetes.io/projected/1c5c29c2-3215-4800-a45e-f0cf69601df3-kube-api-access-cdntx\") pod \"perf-node-gather-daemonset-m9zcl\" (UID: \"1c5c29c2-3215-4800-a45e-f0cf69601df3\") " pod="openshift-must-gather-prwc8/perf-node-gather-daemonset-m9zcl" Jul 08 19:19:37.026798 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:37.026719 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-prwc8/perf-node-gather-daemonset-m9zcl" Jul 08 19:19:37.171649 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:37.171625 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-prwc8/perf-node-gather-daemonset-m9zcl"] Jul 08 19:19:37.173720 ip-10-0-131-102 kubenswrapper[2575]: W0708 19:19:37.173692 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1c5c29c2_3215_4800_a45e_f0cf69601df3.slice/crio-0e820b7bf1b11a7eb9a687afe6bf686a03feff78317eece735ddb06c921de0a0 WatchSource:0}: Error finding container 0e820b7bf1b11a7eb9a687afe6bf686a03feff78317eece735ddb06c921de0a0: Status 404 returned error can't find the container with id 0e820b7bf1b11a7eb9a687afe6bf686a03feff78317eece735ddb06c921de0a0 Jul 08 19:19:37.746119 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:37.746082 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-zpzc6_4baf0565-cc14-47a2-823c-c02c1d82e981/volume-data-source-validator/0.log" Jul 08 19:19:38.129733 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:38.129699 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-prwc8/perf-node-gather-daemonset-m9zcl" event={"ID":"1c5c29c2-3215-4800-a45e-f0cf69601df3","Type":"ContainerStarted","Data":"0e87807844b12fd25398103a55236b10a9450392a54d7e2e1371657d6571b194"} Jul 08 19:19:38.130135 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:38.129756 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-prwc8/perf-node-gather-daemonset-m9zcl" event={"ID":"1c5c29c2-3215-4800-a45e-f0cf69601df3","Type":"ContainerStarted","Data":"0e820b7bf1b11a7eb9a687afe6bf686a03feff78317eece735ddb06c921de0a0"} Jul 08 19:19:38.130135 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:38.129789 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-prwc8/perf-node-gather-daemonset-m9zcl" Jul 08 19:19:38.148307 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:38.148243 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-prwc8/perf-node-gather-daemonset-m9zcl" podStartSLOduration=2.148225046 podStartE2EDuration="2.148225046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 19:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 19:19:38.14580095 +0000 UTC m=+2388.983128672" watchObservedRunningTime="2026-07-08 19:19:38.148225046 +0000 UTC m=+2388.985552754" Jul 08 19:19:38.826008 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:38.825980 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ljtbq_0b2b7480-7b33-4fd9-9993-71441bbb6c78/dns/0.log" Jul 08 19:19:38.845265 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:38.845241 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ljtbq_0b2b7480-7b33-4fd9-9993-71441bbb6c78/kube-rbac-proxy/0.log" Jul 08 19:19:38.909968 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:38.909936 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7gjd9_bcc6d89d-de5e-44a5-9fe2-02785d57a8e0/dns-node-resolver/0.log" Jul 08 19:19:39.518385 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:39.518351 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5d6d887bf8-98j5w_3bf87a41-e859-4773-a975-7b8633d52799/registry/0.log" Jul 08 19:19:39.562971 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:39.562945 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jj7hl_06234e92-3ccc-4e0a-9250-bfe6bf99befd/node-ca/0.log" Jul 08 19:19:40.642827 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:40.642795 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-w9rg6_362ce324-f216-489b-89c5-ddd7244f8fbd/discovery/0.log" Jul 08 19:19:40.688131 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:40.688107 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8b75dd56d-wk2ph_dc05552e-bd5b-46e6-95df-b70171c218f5/kube-auth-proxy/0.log" Jul 08 19:19:40.716463 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:40.716429 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-w6zn2_926ae9db-d53a-44d9-93ec-c93c16c604b2/istio-proxy/0.log" Jul 08 19:19:40.743050 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:40.743020 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-w6zn2_926ae9db-d53a-44d9-93ec-c93c16c604b2/istio-proxy/1.log" Jul 08 19:19:40.780198 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:40.780171 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-plb8f_dbed1f82-e4b6-4cdc-a68c-40dcf2448650/payload-pre-processing/0.log" Jul 08 19:19:40.824452 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:40.824419 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-qcc7s_70b85c5d-9a96-4b41-a062-8de6a4809609/payload-processing/0.log" Jul 08 19:19:41.576404 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:41.576377 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-l582k_a2d88ce3-6a8c-4b89-8bcd-e5a8ead15f44/serve-healthcheck-canary/0.log" Jul 08 19:19:42.198592 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:42.198563 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lz2g7_23d40abf-d3f0-4faa-9768-57e6486a5a6c/kube-rbac-proxy/0.log" Jul 08 19:19:42.219013 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:42.218981 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lz2g7_23d40abf-d3f0-4faa-9768-57e6486a5a6c/exporter/0.log" Jul 08 19:19:42.240938 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:42.240908 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lz2g7_23d40abf-d3f0-4faa-9768-57e6486a5a6c/extractor/0.log" Jul 08 19:19:44.145714 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:44.145686 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-prwc8/perf-node-gather-daemonset-m9zcl" Jul 08 19:19:44.497652 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:44.497571 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6449898b5-4p9q2_1aa899cd-b902-47df-a8e7-cc85e05e1886/manager/0.log" Jul 08 19:19:44.587106 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:44.587071 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8vw4w_eeae891b-9bad-4f61-915c-f0f4cac1bc18/manager/0.log" Jul 08 19:19:44.743761 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:44.743730 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c5cf545b9-r59jn_34dd0db6-dd56-49aa-8302-1d90fd9c31cf/manager/0.log" Jul 08 19:19:44.763539 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:44.763465 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c5cf545b9-r59jn_34dd0db6-dd56-49aa-8302-1d90fd9c31cf/copy-manifests/0.log" Jul 08 19:19:46.159771 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:46.159740 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-4pw65_b1f22463-b8a0-4a4d-973b-21f1a8db4c30/openshift-lws-operator/0.log" Jul 08 19:19:46.869313 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:46.869251 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-w9rg6_362ce324-f216-489b-89c5-ddd7244f8fbd/discovery/0.log" Jul 08 19:19:46.894696 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:46.894660 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8b75dd56d-wk2ph_dc05552e-bd5b-46e6-95df-b70171c218f5/kube-auth-proxy/0.log" Jul 08 19:19:46.912957 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:46.912927 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-w6zn2_926ae9db-d53a-44d9-93ec-c93c16c604b2/istio-proxy/0.log" Jul 08 19:19:46.939689 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:46.939664 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-w6zn2_926ae9db-d53a-44d9-93ec-c93c16c604b2/istio-proxy/1.log" Jul 08 19:19:46.967535 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:46.967467 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-plb8f_dbed1f82-e4b6-4cdc-a68c-40dcf2448650/payload-pre-processing/0.log" Jul 08 19:19:47.001667 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:47.001641 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-qcc7s_70b85c5d-9a96-4b41-a062-8de6a4809609/payload-processing/0.log" Jul 08 19:19:48.373748 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:48.373717 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-w9rg6_362ce324-f216-489b-89c5-ddd7244f8fbd/discovery/0.log" Jul 08 19:19:48.406625 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:48.406602 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8b75dd56d-wk2ph_dc05552e-bd5b-46e6-95df-b70171c218f5/kube-auth-proxy/0.log" Jul 08 19:19:48.427284 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:48.427258 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-w6zn2_926ae9db-d53a-44d9-93ec-c93c16c604b2/istio-proxy/0.log" Jul 08 19:19:48.454986 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:48.454957 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-w6zn2_926ae9db-d53a-44d9-93ec-c93c16c604b2/istio-proxy/1.log" Jul 08 19:19:48.485125 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:48.485052 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-plb8f_dbed1f82-e4b6-4cdc-a68c-40dcf2448650/payload-pre-processing/0.log" Jul 08 19:19:48.520635 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:48.520600 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-qcc7s_70b85c5d-9a96-4b41-a062-8de6a4809609/payload-processing/0.log" Jul 08 19:19:51.794558 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:51.794522 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-2wffx_aedf6d82-9258-424b-a9f0-af1f0154058b/kube-storage-version-migrator-operator/1.log" Jul 08 19:19:51.795676 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:51.795654 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-2wffx_aedf6d82-9258-424b-a9f0-af1f0154058b/kube-storage-version-migrator-operator/0.log" Jul 08 19:19:52.841051 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:52.840974 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-grt82_1e7898c0-39bf-42c9-a65e-30409d5a5ea1/kube-multus-additional-cni-plugins/0.log" Jul 08 19:19:52.862594 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:52.862568 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-grt82_1e7898c0-39bf-42c9-a65e-30409d5a5ea1/egress-router-binary-copy/0.log" Jul 08 19:19:52.882550 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:52.882520 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-grt82_1e7898c0-39bf-42c9-a65e-30409d5a5ea1/cni-plugins/0.log" Jul 08 19:19:52.902726 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:52.902702 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-grt82_1e7898c0-39bf-42c9-a65e-30409d5a5ea1/bond-cni-plugin/0.log" Jul 08 19:19:52.922812 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:52.922785 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-grt82_1e7898c0-39bf-42c9-a65e-30409d5a5ea1/routeoverride-cni/0.log" Jul 08 19:19:52.942116 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:52.942096 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-grt82_1e7898c0-39bf-42c9-a65e-30409d5a5ea1/whereabouts-cni-bincopy/0.log" Jul 08 19:19:52.962700 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:52.962682 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-grt82_1e7898c0-39bf-42c9-a65e-30409d5a5ea1/whereabouts-cni/0.log" Jul 08 19:19:53.352341 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:53.352290 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-rsw44_713392b6-78a2-4ca4-8cf8-43f8ab24b0cb/kube-multus/0.log" Jul 08 19:19:53.375258 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:53.375232 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-252x5_ee39c730-7ca1-4260-afea-a884940c00fc/network-metrics-daemon/0.log" Jul 08 19:19:53.394188 ip-10-0-131-102 kubenswrapper[2575]: I0708 19:19:53.394163 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-252x5_ee39c730-7ca1-4260-afea-a884940c00fc/kube-rbac-proxy/0.log"